Sep 09 02:28:26.856903 ip-10-0-136-188 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 09 02:28:26.856912 ip-10-0-136-188 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 09 02:28:26.856919 ip-10-0-136-188 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 09 02:28:26.857147 ip-10-0-136-188 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 09 02:28:35.941883 ip-10-0-136-188 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 09 02:28:35.941898 ip-10-0-136-188 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot bda8a0cf64434f258a840df864fb0360 --
Sep 09 02:30:17.620301 ip-10-0-136-188 systemd[1]: Starting Kubernetes Kubelet...
Sep 09 02:30:18.068956 ip-10-0-136-188 kubenswrapper[2570]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 02:30:18.068956 ip-10-0-136-188 kubenswrapper[2570]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 09 02:30:18.068956 ip-10-0-136-188 kubenswrapper[2570]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 02:30:18.068956 ip-10-0-136-188 kubenswrapper[2570]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 09 02:30:18.068956 ip-10-0-136-188 kubenswrapper[2570]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 02:30:18.070124 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.070038    2570 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 09 02:30:18.074421 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.074402    2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 02:30:18.074421 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.074416    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 02:30:18.074421 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.074422    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 02:30:18.074620 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.074428    2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 02:30:18.074620 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.074432    2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 02:30:18.074620 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.074436    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 02:30:18.074620 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.074441    2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 02:30:18.074620 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.074459    2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 02:30:18.074620 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.074464    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 02:30:18.074620 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.074468    2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 02:30:18.074620 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.074472    2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 02:30:18.074620 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.074476    2570 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 02:30:18.074620 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.074480    2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 02:30:18.074620 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.074484    2570 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 02:30:18.074620 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.074487    2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 02:30:18.074620 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.074492    2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 02:30:18.074620 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.074496    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 02:30:18.074620 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.074500    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 02:30:18.074620 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.074504    2570 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 02:30:18.074620 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.074508    2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 02:30:18.074620 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.074511    2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 02:30:18.074620 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.074515    2570 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 02:30:18.074620 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.074519    2570 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 02:30:18.075431 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.074522    2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 02:30:18.075431 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.074526    2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 02:30:18.075431 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.074530    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 02:30:18.075431 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.074534    2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 02:30:18.075431 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.074538    2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 02:30:18.075431 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.074542    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 02:30:18.075431 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.074547    2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 02:30:18.075431 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.074554    2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 02:30:18.075431 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.074560    2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 02:30:18.075431 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.074565    2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 02:30:18.075431 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.074574    2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 02:30:18.075431 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.074580    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 02:30:18.075431 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.074584    2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 02:30:18.075431 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.074590    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 02:30:18.075431 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.074595    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 02:30:18.075431 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.074599    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 02:30:18.075431 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.074603    2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 02:30:18.075431 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.074615    2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 02:30:18.075431 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.074619    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 02:30:18.076104 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.074624    2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 02:30:18.076104 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.074628    2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 02:30:18.076104 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.074633    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 02:30:18.076104 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.074637    2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 02:30:18.076104 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.074641    2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 02:30:18.076104 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.074645    2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 02:30:18.076104 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.074650    2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 02:30:18.076104 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.074654    2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 02:30:18.076104 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.074661    2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 02:30:18.076104 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.074667    2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 02:30:18.076104 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.074672    2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 02:30:18.076104 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.074677    2570 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 02:30:18.076104 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.074681    2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 02:30:18.076104 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.074686    2570 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 02:30:18.076104 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.074690    2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 02:30:18.076104 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.074695    2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 02:30:18.076104 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.074699    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 02:30:18.076104 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.074703    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 02:30:18.076104 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.074707    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 02:30:18.076104 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.074711    2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 02:30:18.076589 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.074716    2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 02:30:18.076589 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.074720    2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 02:30:18.076589 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.074723    2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 02:30:18.076589 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.074727    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 02:30:18.076589 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.074732    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 02:30:18.076589 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.074737    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 02:30:18.076589 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.074747    2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 02:30:18.076589 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.074751    2570 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 02:30:18.076589 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.074756    2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 02:30:18.076589 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.074761    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 02:30:18.076589 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.074765    2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 02:30:18.076589 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.074770    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 02:30:18.076589 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.074774    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 02:30:18.076589 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.074779    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 02:30:18.076589 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.074783    2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 02:30:18.076589 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.074787    2570 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 02:30:18.076589 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.074792    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 02:30:18.076589 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.074796    2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 02:30:18.076589 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.074800    2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 02:30:18.076589 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.074804    2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 02:30:18.077163 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.074808    2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 02:30:18.077163 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.074812    2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 02:30:18.077163 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.074816    2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 02:30:18.077163 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.074821    2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 02:30:18.077163 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.074825    2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 02:30:18.077163 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.074829    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 02:30:18.077163 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.074834    2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 02:30:18.077163 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.074838    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 02:30:18.077163 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.074842    2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 02:30:18.077163 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.074846    2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 02:30:18.077163 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.074850    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 02:30:18.077163 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.074855    2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 02:30:18.077163 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.074859    2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 02:30:18.077163 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.074864    2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 02:30:18.077163 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.074868    2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 02:30:18.077163 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.074872    2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 02:30:18.077163 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.074876    2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 02:30:18.077163 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.076962    2570 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 02:30:18.077163 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.076991    2570 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 02:30:18.077163 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.076996    2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 02:30:18.077659 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077001    2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 02:30:18.077659 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077005    2570 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 02:30:18.077659 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077008    2570 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 02:30:18.077659 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077012    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 02:30:18.077659 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077014    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 02:30:18.077659 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077017    2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 02:30:18.077659 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077019    2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 02:30:18.077659 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077023    2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 02:30:18.077659 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077025    2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 02:30:18.077659 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077028    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 02:30:18.077659 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077031    2570 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 02:30:18.077659 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077033    2570 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 02:30:18.077659 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077036    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 02:30:18.077659 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077038    2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 02:30:18.077659 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077041    2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 02:30:18.077659 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077044    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 02:30:18.077659 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077046    2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 02:30:18.077659 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077049    2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 02:30:18.077659 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077051    2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 02:30:18.077659 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077054    2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 02:30:18.078175 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077056    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 02:30:18.078175 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077059    2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 02:30:18.078175 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077061    2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 02:30:18.078175 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077064    2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 02:30:18.078175 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077066    2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 02:30:18.078175 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077069    2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 02:30:18.078175 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077072    2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 02:30:18.078175 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077075    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 02:30:18.078175 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077078    2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 02:30:18.078175 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077080    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 02:30:18.078175 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077083    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 02:30:18.078175 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077086    2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 02:30:18.078175 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077089    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 02:30:18.078175 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077091    2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 02:30:18.078175 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077094    2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 02:30:18.078175 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077097    2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 02:30:18.078175 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077100    2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 02:30:18.078175 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077102    2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 02:30:18.078175 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077105    2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 02:30:18.078644 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077107    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 02:30:18.078644 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077110    2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 02:30:18.078644 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077112    2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 02:30:18.078644 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077115    2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 02:30:18.078644 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077117    2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 02:30:18.078644 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077120    2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 02:30:18.078644 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077122    2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 02:30:18.078644 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077124    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 02:30:18.078644 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077127    2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 02:30:18.078644 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077130    2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 02:30:18.078644 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077132    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 02:30:18.078644 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077135    2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 02:30:18.078644 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077138    2570 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 02:30:18.078644 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077141    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 02:30:18.078644 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077143    2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 02:30:18.078644 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077149    2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 02:30:18.078644 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077153    2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 02:30:18.078644 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077156    2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 02:30:18.078644 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077159    2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 02:30:18.079126 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077162    2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 02:30:18.079126 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077165    2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 02:30:18.079126 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077167    2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 02:30:18.079126 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077171    2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 02:30:18.079126 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077174    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 02:30:18.079126 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077177    2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 02:30:18.079126 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077180    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 02:30:18.079126 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077183    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 02:30:18.079126 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077186    2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 02:30:18.079126 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077188    2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 02:30:18.079126 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077192    2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 02:30:18.079126 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077195    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 02:30:18.079126 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077198    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 02:30:18.079126 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077201    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 02:30:18.079126 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077204    2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 02:30:18.079126 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077206    2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 02:30:18.079126 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077209    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 02:30:18.079126 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077211    2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 02:30:18.079126 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077214    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 02:30:18.079126 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077218    2570 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 02:30:18.079611 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077220    2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 02:30:18.079611 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077222    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 02:30:18.079611 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077225    2570 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 02:30:18.079611 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077228    2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 02:30:18.079611 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077230    2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 02:30:18.079611 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077233    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 02:30:18.079611 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077236    2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 02:30:18.079611 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077238    2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 02:30:18.079611 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077241    2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 02:30:18.079611 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077245    2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 02:30:18.079611 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077248    2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 02:30:18.079611 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077251    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 02:30:18.079611 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077253    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 02:30:18.079611 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077256    2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 02:30:18.079611 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077258    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 02:30:18.079611 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077261    2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 02:30:18.079611 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077264    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 02:30:18.079611 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077267    2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 02:30:18.079611 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077672    2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 02:30:18.080095 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077679    2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 02:30:18.080095 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077684    2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 02:30:18.080095 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077688    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 02:30:18.080095 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077692    2570 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 02:30:18.080095 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077695    2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 02:30:18.080095 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077698    2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 02:30:18.080095 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077701    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 02:30:18.080095 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077703    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 02:30:18.080095 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077706    2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 02:30:18.080095 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077708    2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 02:30:18.080095 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077711    2570 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 02:30:18.080095 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077714    2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 02:30:18.080095 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077717    2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 02:30:18.080095 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077719    2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 02:30:18.080095 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077722    2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 02:30:18.080095 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077724    2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 02:30:18.080095 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077727    2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 02:30:18.080095 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077730    2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 02:30:18.080095 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077732    2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 02:30:18.080562 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077735    2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 02:30:18.080562 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077738    2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 02:30:18.080562 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077740    2570 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 02:30:18.080562 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077745    2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 02:30:18.080562 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077748    2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 02:30:18.080562 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077751    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 02:30:18.080562 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077754    2570 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 02:30:18.080562 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077757    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 02:30:18.080562 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077760    2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 02:30:18.080562 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077763    2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 02:30:18.080562 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077765    2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 02:30:18.080562 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077768    2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 02:30:18.080562 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077771    2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 02:30:18.080562 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077773    2570 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 02:30:18.080562 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077775    2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 02:30:18.080562 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077778    2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 02:30:18.080562 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077781    2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 02:30:18.080562 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077783    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 02:30:18.080562 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077786    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 02:30:18.081059 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077788    2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 02:30:18.081059 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077791    2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 02:30:18.081059 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077793    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 02:30:18.081059 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077796    2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 02:30:18.081059 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077798    2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 02:30:18.081059 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077800    2570 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 02:30:18.081059 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077803    2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 02:30:18.081059 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077805    2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 02:30:18.081059 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077808    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 02:30:18.081059 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077810    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 02:30:18.081059 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077813    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 02:30:18.081059 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077816    2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 02:30:18.081059 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077818    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 02:30:18.081059 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077821    2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 02:30:18.081059 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077824    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 02:30:18.081059 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077826    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 02:30:18.081059 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077829    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 02:30:18.081059 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077831    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 02:30:18.081059 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077834    2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 02:30:18.081059 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077836    2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 02:30:18.081547 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077839    2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 02:30:18.081547 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077841    2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 02:30:18.081547 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077844    2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 02:30:18.081547 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077846    2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 02:30:18.081547 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077849    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 02:30:18.081547 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077851    2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 02:30:18.081547 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077854    2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 02:30:18.081547 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077856    2570 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 02:30:18.081547 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077859    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 02:30:18.081547 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077862    2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 02:30:18.081547 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077864    2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 02:30:18.081547 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077867    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 02:30:18.081547 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077869    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 02:30:18.081547 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077872    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 02:30:18.081547 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077874    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 02:30:18.081547 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077877    2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 02:30:18.081547 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077879    2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 02:30:18.081547 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077881    2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 02:30:18.081547 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077884    2570 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 02:30:18.081547 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077886    2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 02:30:18.082074 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077889    2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 02:30:18.082074 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077891    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 02:30:18.082074 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077893    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 02:30:18.082074 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077897    2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 02:30:18.082074 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077899    2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 02:30:18.082074 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077902    2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 02:30:18.082074 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077907    2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 02:30:18.082074 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077910    2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 02:30:18.082074 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077912    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 02:30:18.082074 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077914    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 02:30:18.082074 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077917    2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 02:30:18.082074 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077920    2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 02:30:18.082074 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077922    2570 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 02:30:18.082074 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077925    2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 02:30:18.082074 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077928    2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 02:30:18.082074 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077930    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 02:30:18.082074 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077933    2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 02:30:18.082074 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077935    2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 02:30:18.082074 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077938    2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 02:30:18.082074 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.077940    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 02:30:18.082561 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078028    2570 flags.go:64] FLAG: --address="0.0.0.0"
Sep 09 02:30:18.082561 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078066    2570 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 09 02:30:18.082561 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078073    2570 flags.go:64] FLAG: --anonymous-auth="true"
Sep 09 02:30:18.082561 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078077    2570 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 09 02:30:18.082561 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078081    2570 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 09 02:30:18.082561 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078084    2570 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 09 02:30:18.082561 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078088    2570 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 09 02:30:18.082561 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078093    2570 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 09 02:30:18.082561 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078096    2570 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 09 02:30:18.082561 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078098    2570 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 09 02:30:18.082561 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078102    2570 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 09 02:30:18.082561 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078105    2570 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 09 02:30:18.082561 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078108    2570 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 09 02:30:18.082561 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078111    2570 flags.go:64] FLAG: --cgroup-root=""
Sep 09 02:30:18.082561 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078118    2570 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 09 02:30:18.082561 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078120    2570 flags.go:64] FLAG: --client-ca-file=""
Sep 09 02:30:18.082561 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078124    2570 flags.go:64] FLAG: --cloud-provider="external"
Sep 09 02:30:18.082561 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078127    2570 flags.go:64] FLAG: --cluster-dns="[]"
Sep 09 02:30:18.082561 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078133    2570 flags.go:64] FLAG: --cluster-domain=""
Sep 09 02:30:18.082561 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078136    2570 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 09 02:30:18.082561 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078139    2570 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 09 02:30:18.082561 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078142    2570 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 09 02:30:18.082561 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078146    2570 flags.go:64] FLAG: --container-log-max-files="5"
Sep 09 02:30:18.082561 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078150    2570 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 09 02:30:18.083231 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078153    2570 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 09 02:30:18.083231 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078156    2570 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 09 02:30:18.083231 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078159    2570 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 09 02:30:18.083231 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078162    2570 flags.go:64] FLAG: --contention-profiling="false"
Sep 09 02:30:18.083231 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078165    2570 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 09 02:30:18.083231 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078168    2570 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 09 02:30:18.083231 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078171    2570 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 09 02:30:18.083231 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078174    2570 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 09 02:30:18.083231 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078178    2570 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 09 02:30:18.083231 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078181    2570 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 09 02:30:18.083231 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078184    2570 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 09 02:30:18.083231 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078187    2570 flags.go:64] FLAG: --enable-load-reader="false"
Sep 09 02:30:18.083231 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078190    2570 flags.go:64] FLAG: --enable-server="true"
Sep 09 02:30:18.083231 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078194    2570 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 09 02:30:18.083231 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078198    2570 flags.go:64] FLAG: --event-burst="100"
Sep 09 02:30:18.083231 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078201    2570 flags.go:64] FLAG: --event-qps="50"
Sep 09 02:30:18.083231 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078204    2570 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 09 02:30:18.083231 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078207    2570 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 09 02:30:18.083231 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078210    2570 flags.go:64] FLAG: --eviction-hard=""
Sep 09 02:30:18.083231 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078213    2570 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 09 02:30:18.083231 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078216    2570 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 09 02:30:18.083231 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078219    2570 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 09 02:30:18.083231 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078223    2570 flags.go:64] FLAG: --eviction-soft=""
Sep 09 02:30:18.083231 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078226    2570 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 09 02:30:18.083231 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078228    2570 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 09 02:30:18.083855 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078232    2570 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 09 02:30:18.083855 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078236    2570 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 09 02:30:18.083855 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078239    2570 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 09 02:30:18.083855 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078242    2570 flags.go:64] FLAG: --fail-swap-on="true"
Sep 09 02:30:18.083855 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078244    2570 flags.go:64] FLAG: --feature-gates=""
Sep 09 02:30:18.083855 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078248    2570 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 09 02:30:18.083855 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078251    2570 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 09 02:30:18.083855 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078254    2570 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 09 02:30:18.083855 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078257    2570 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 09 02:30:18.083855 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078260    2570 flags.go:64] FLAG: --healthz-port="10248"
Sep 09 02:30:18.083855 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078263    2570 flags.go:64] FLAG: --help="false"
Sep 09 02:30:18.083855 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078266    2570 flags.go:64] FLAG: --hostname-override="ip-10-0-136-188.ec2.internal"
Sep 09 02:30:18.083855 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078269    2570 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 09 02:30:18.083855 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078271    2570 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 09 02:30:18.083855 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078274    2570 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 09 02:30:18.083855 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078277    2570 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 09 02:30:18.083855 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078281    2570 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 09 02:30:18.083855 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078283    2570 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 09 02:30:18.083855 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078286    2570 flags.go:64] FLAG: --image-service-endpoint=""
Sep 09 02:30:18.083855 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078289    2570 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 09 02:30:18.083855 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078292    2570 flags.go:64] FLAG: --kube-api-burst="100"
Sep 09 02:30:18.083855 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078294    2570 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 09 02:30:18.083855 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078297    2570 flags.go:64] FLAG: --kube-api-qps="50"
Sep 09 02:30:18.083855 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078300    2570 flags.go:64] FLAG: --kube-reserved=""
Sep 09 02:30:18.084525 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078303    2570 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 09 02:30:18.084525 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078305    2570 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 09 02:30:18.084525 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078308    2570 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 09 02:30:18.084525 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078311    2570 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 09 02:30:18.084525 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078314    2570 flags.go:64] FLAG: --lock-file=""
Sep 09 02:30:18.084525 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078317    2570 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 09 02:30:18.084525 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078320    2570 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 09 02:30:18.084525 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078323    2570 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 09 02:30:18.084525 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078328    2570 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 09 02:30:18.084525 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078334    2570 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 09 02:30:18.084525 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078337    2570 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 09 02:30:18.084525 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078339    2570 flags.go:64] FLAG: --logging-format="text"
Sep 09 02:30:18.084525 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078342    2570 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 09 02:30:18.084525 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078346    2570 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 09 02:30:18.084525 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078349    2570 flags.go:64] FLAG: --manifest-url=""
Sep 09 02:30:18.084525 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078351    2570 flags.go:64] FLAG: --manifest-url-header=""
Sep 09 02:30:18.084525 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078356    2570 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 09 02:30:18.084525 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078359    2570 flags.go:64] FLAG: --max-open-files="1000000"
Sep 09 02:30:18.084525 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078363    2570 flags.go:64] FLAG: --max-pods="110"
Sep 09 02:30:18.084525 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078366    2570 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 09 02:30:18.084525 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078369    2570 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 09 02:30:18.084525 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078371    2570 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 09 02:30:18.084525 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078374    2570 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 09 02:30:18.084525 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078377    2570 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 09 02:30:18.084525 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078380    2570 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 09 02:30:18.085162 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078383    2570 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 09 02:30:18.085162 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078390    2570 flags.go:64] FLAG: --node-status-max-images="50"
Sep 09 02:30:18.085162 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078393    2570 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 09 02:30:18.085162 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078396    2570 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 09 02:30:18.085162 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078399    2570 flags.go:64] FLAG: --pod-cidr=""
Sep 09 02:30:18.085162 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078402    2570 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 09 02:30:18.085162 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078408    2570 flags.go:64] FLAG: --pod-manifest-path=""
Sep 09 02:30:18.085162 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078411    2570 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 09 02:30:18.085162 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078414    2570 flags.go:64] FLAG: --pods-per-core="0"
Sep 09 02:30:18.085162 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078417    2570 flags.go:64] FLAG: --port="10250"
Sep 09 02:30:18.085162 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078420    2570 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 09 02:30:18.085162 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078423    2570 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-00a780e4f27da6b6a"
Sep 09 02:30:18.085162 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078426    2570 flags.go:64] FLAG: --qos-reserved=""
Sep 09 02:30:18.085162 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078429    2570 flags.go:64] FLAG: --read-only-port="10255"
Sep 09 02:30:18.085162 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078432    2570 flags.go:64] FLAG: --register-node="true"
Sep 09 02:30:18.085162 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078435    2570 flags.go:64] FLAG: --register-with-taints=""
Sep 09 02:30:18.085162 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078439    2570 flags.go:64] FLAG: --registry-burst="10"
Sep 09 02:30:18.085162 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078443    2570 flags.go:64] FLAG: --registry-qps="5"
Sep 09 02:30:18.085162 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078446    2570 flags.go:64] FLAG: --reserved-cpus=""
Sep 09 02:30:18.085162 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078448    2570 flags.go:64] FLAG: --reserved-memory=""
Sep 09 02:30:18.085162 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078452    2570 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 09 02:30:18.085162 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078455    2570 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 09 02:30:18.085162 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078458    2570 flags.go:64] FLAG: --rotate-certificates="false"
Sep 09 02:30:18.085162 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078461    2570 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 09 02:30:18.085162 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078463    2570 flags.go:64] FLAG: --runonce="false"
Sep 09 02:30:18.085816 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078466    2570 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 09 02:30:18.085816 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078469    2570 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 09 02:30:18.085816 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078472    2570 flags.go:64] FLAG: --seccomp-default="false"
Sep 09 02:30:18.085816 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078475    2570 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 09 02:30:18.085816 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078478    2570 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 09 02:30:18.085816 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078480    2570 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 09 02:30:18.085816 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078483    2570 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 09 02:30:18.085816 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078486    2570 flags.go:64] FLAG: --storage-driver-password="root"
Sep 09 02:30:18.085816 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078489    2570 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 09 02:30:18.085816 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078492    2570 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 09 02:30:18.085816 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078495    2570 flags.go:64] FLAG: --storage-driver-user="root"
Sep 09 02:30:18.085816 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078497    2570 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 09 02:30:18.085816 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078500    2570 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 09 02:30:18.085816 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078503    2570 flags.go:64] FLAG: --system-cgroups=""
Sep 09 02:30:18.085816 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078506    2570 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 09 02:30:18.085816 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078511    2570 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 09 02:30:18.085816 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078514    2570 flags.go:64] FLAG: --tls-cert-file=""
Sep 09 02:30:18.085816 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078516    2570 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 09 02:30:18.085816 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078527    2570 flags.go:64] FLAG: --tls-min-version=""
Sep 09 02:30:18.085816 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078530    2570 flags.go:64] FLAG: --tls-private-key-file=""
Sep 09 02:30:18.085816 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078532    2570 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 09 02:30:18.085816 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078535    2570 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 09 02:30:18.085816 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078538    2570 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 09 02:30:18.085816 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078542    2570 flags.go:64] FLAG: --v="2"
Sep 09 02:30:18.085816 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078550    2570 flags.go:64] FLAG: --version="false"
Sep 09 02:30:18.086440 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078555    2570 flags.go:64] FLAG: --vmodule=""
Sep 09 02:30:18.086440 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078559    2570 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 09 02:30:18.086440 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078562    2570 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 09 02:30:18.086440 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.078683    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 02:30:18.086440 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.078687    2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 02:30:18.086440 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.078690    2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 02:30:18.086440 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.078693    2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 02:30:18.086440 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.078695    2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 02:30:18.086440 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.078698    2570 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 02:30:18.086440 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.078702    2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 02:30:18.086440 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.078706    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 02:30:18.086440 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.078709    2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 02:30:18.086440 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.078711    2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 02:30:18.086440 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.078714    2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 02:30:18.086440 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.078717    2570 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 02:30:18.086440 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.078720    2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 02:30:18.086440 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.078722    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 02:30:18.086440 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.078725    2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 02:30:18.086440 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.078727    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 02:30:18.086440 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.078730    2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 02:30:18.086928 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.078733    2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 02:30:18.086928 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.078735    2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 02:30:18.086928 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.078738    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 02:30:18.086928 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.078740    2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 02:30:18.086928 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.078743    2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 02:30:18.086928 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.078745    2570 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 02:30:18.086928 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.078748    2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 02:30:18.086928 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.078751    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 02:30:18.086928 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.078754    2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 02:30:18.086928 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.078757    2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 02:30:18.086928 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.078760    2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 02:30:18.086928 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.078763    2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 02:30:18.086928 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.078765    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 02:30:18.086928 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.078769    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 02:30:18.086928 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.078771    2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 02:30:18.086928 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.078774    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 02:30:18.086928 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.078777    2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 02:30:18.086928 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.078780    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 02:30:18.086928 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.078782    2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 02:30:18.086928 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.078785    2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 02:30:18.087454 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.078787    2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 02:30:18.087454 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.078790    2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 02:30:18.087454 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.078792    2570 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 02:30:18.087454 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.078795    2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 02:30:18.087454 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.078797    2570 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 02:30:18.087454 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.078800    2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 02:30:18.087454 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.078802    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 02:30:18.087454 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.078805    2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 02:30:18.087454 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.078807    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 02:30:18.087454 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.078809    2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 02:30:18.087454 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.078812    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 02:30:18.087454 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.078814    2570 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 02:30:18.087454 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.078817    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 02:30:18.087454 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.078820    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 02:30:18.087454 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.078822    2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 02:30:18.087454 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.078824    2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 02:30:18.087454 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.078827    2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 02:30:18.087454 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.078829    2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 02:30:18.087454 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.078832    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 02:30:18.087454 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.078836    2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 02:30:18.087950 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.078838    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 02:30:18.087950 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.078841    2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 02:30:18.087950 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.078847    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 02:30:18.087950 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.078850    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 02:30:18.087950 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.078852    2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 02:30:18.087950 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.078855    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 02:30:18.087950 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.078858    2570 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 02:30:18.087950 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.078861    2570 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 02:30:18.087950 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.078863    2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 02:30:18.087950 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.078866    2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 02:30:18.087950 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.078869    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 02:30:18.087950 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.078872    2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 02:30:18.087950 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.078876    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 02:30:18.087950 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.078880    2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 02:30:18.087950 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.078882    2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 02:30:18.087950 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.078885    2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 02:30:18.087950 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.078888    2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 02:30:18.087950 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.078890    2570 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 02:30:18.087950 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.078893    2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 02:30:18.087950 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.078895    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 02:30:18.088514 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.078898    2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 02:30:18.088514 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.078901    2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 02:30:18.088514 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.078903    2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 02:30:18.088514 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.078906    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 02:30:18.088514 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.078909    2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 02:30:18.088514 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.078911    2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 02:30:18.088514 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.078914    2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 02:30:18.088514 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.078917    2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 02:30:18.088514 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.078921    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 02:30:18.088514 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.078924    2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 02:30:18.088514 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.078927    2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 02:30:18.088514 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.078931    2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 02:30:18.088514 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.078934    2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 02:30:18.088514 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.078937    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 02:30:18.088514 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.078940    2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 02:30:18.088514 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.078943    2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 02:30:18.088514 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.078946    2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 02:30:18.088514 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.078948    2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 02:30:18.088514 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.078951    2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 02:30:18.089065 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.078954    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 02:30:18.089065 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.078957    2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 02:30:18.089065 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.078959    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 02:30:18.089065 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.078967    2570 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 02:30:18.089065 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.086434    2570 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 09 02:30:18.089065 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.086449    2570 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 09 02:30:18.089065 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.086468    2570 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 09 02:30:18.089065 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.086474    2570 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 09 02:30:18.089065 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.086532    2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 02:30:18.089065 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.086538    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 02:30:18.089065 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.086541    2570 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 02:30:18.089065 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.086544    2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 02:30:18.089065 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.086547    2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 02:30:18.089065 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.086550    2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 02:30:18.089065 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.086553    2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 02:30:18.089443 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.086556    2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 02:30:18.089443 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.086559    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 02:30:18.089443 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.086562    2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 02:30:18.089443 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.086565    2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 02:30:18.089443 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.086568    2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 02:30:18.089443 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.086571    2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 02:30:18.089443 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.086573    2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 02:30:18.089443 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.086576    2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 02:30:18.089443 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.086578    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 02:30:18.089443 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.086581    2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 02:30:18.089443 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.086584    2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 02:30:18.089443 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.086586    2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 02:30:18.089443 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.086590    2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 02:30:18.089443 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.086592    2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 02:30:18.089443 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.086595    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 02:30:18.089443 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.086597    2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 02:30:18.089443 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.086600    2570 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 02:30:18.089443 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.086603    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 02:30:18.089443 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.086606    2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 02:30:18.089443 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.086608    2570 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 02:30:18.089936 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.086611    2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 02:30:18.089936 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.086613    2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 02:30:18.089936 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.086617    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 02:30:18.089936 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.086620    2570 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 02:30:18.089936 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.086622    2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 02:30:18.089936 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.086625    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 02:30:18.089936 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.086627    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 02:30:18.089936 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.086630    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 02:30:18.089936 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.086632    2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 02:30:18.089936 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.086636    2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 02:30:18.089936 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.086640    2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 02:30:18.089936 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.086643    2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 02:30:18.089936 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.086646    2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 02:30:18.089936 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.086649    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 02:30:18.089936 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.086652    2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 02:30:18.089936 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.086654    2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 02:30:18.089936 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.086657    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 02:30:18.089936 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.086659    2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 02:30:18.089936 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.086662    2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 02:30:18.090422 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.086664    2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 02:30:18.090422 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.086667    2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 02:30:18.090422 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.086669    2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 02:30:18.090422 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.086672    2570 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 02:30:18.090422 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.086674    2570 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 02:30:18.090422 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.086677    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 02:30:18.090422 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.086679    2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 02:30:18.090422 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.086682    2570 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 02:30:18.090422 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.086684    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 02:30:18.090422 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.086687    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 02:30:18.090422 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.086689    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 02:30:18.090422 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.086692    2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 02:30:18.090422 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.086695    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 02:30:18.090422 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.086699    2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 02:30:18.090422 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.086703    2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 02:30:18.090422 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.086706    2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 02:30:18.090422 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.086709    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 02:30:18.090422 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.086711    2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 02:30:18.090422 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.086714    2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 02:30:18.090886 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.086717    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 02:30:18.090886 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.086719    2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 02:30:18.090886 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.086722    2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 02:30:18.090886 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.086724    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 02:30:18.090886 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.086727    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 02:30:18.090886 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.086729    2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 02:30:18.090886 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.086731    2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 02:30:18.090886 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.086734    2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 02:30:18.090886 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.086737    2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 02:30:18.090886 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.086739    2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 02:30:18.090886 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.086742    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 02:30:18.090886 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.086744    2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 02:30:18.090886 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.086747    2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 02:30:18.090886 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.086749    2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 02:30:18.090886 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.086752    2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 02:30:18.090886 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.086754    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 02:30:18.090886 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.086757    2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 02:30:18.090886 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.086759    2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 02:30:18.090886 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.086762    2570 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 02:30:18.090886 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.086764    2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 02:30:18.091379 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.086767    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 02:30:18.091379 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.086770    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 02:30:18.091379 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.086772    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 02:30:18.091379 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.086776    2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 02:30:18.091379 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.086780    2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 02:30:18.091379 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.086784    2570 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 02:30:18.091379 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.086787    2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 02:30:18.091379 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.086790    2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 02:30:18.091379 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.086793    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 02:30:18.091379 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.086796    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 02:30:18.091379 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.086799    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 02:30:18.091379 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.086802    2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 02:30:18.091379 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.086805    2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 02:30:18.091379 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.086807    2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 02:30:18.091379 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.086812    2570 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 02:30:18.091745 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.086929    2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 02:30:18.091745 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.086934    2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 02:30:18.091745 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.086937    2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 02:30:18.091745 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.086940    2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 02:30:18.091745 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.086943    2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 02:30:18.091745 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.086946    2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 02:30:18.091745 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.086949    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 02:30:18.091745 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.086953    2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 02:30:18.091745 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.086956    2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 02:30:18.091745 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.086959    2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 02:30:18.091745 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.086962    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 02:30:18.091745 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.086965    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 02:30:18.091745 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.086967    2570 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 02:30:18.091745 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.086970    2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 02:30:18.091745 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.086985    2570 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 02:30:18.091745 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.086988    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 02:30:18.091745 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.086990    2570 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 02:30:18.091745 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.086993    2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 02:30:18.091745 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.086996    2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 02:30:18.091745 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.086999    2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 02:30:18.092247 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.087002    2570 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 02:30:18.092247 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.087004    2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 02:30:18.092247 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.087007    2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 02:30:18.092247 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.087010    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 02:30:18.092247 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.087013    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 02:30:18.092247 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.087015    2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 02:30:18.092247 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.087018    2570 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 02:30:18.092247 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.087020    2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 02:30:18.092247 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.087023    2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 02:30:18.092247 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.087026    2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 02:30:18.092247 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.087030    2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 02:30:18.092247 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.087033    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 02:30:18.092247 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.087036    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 02:30:18.092247 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.087039    2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 02:30:18.092247 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.087041    2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 02:30:18.092247 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.087044    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 02:30:18.092247 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.087047    2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 02:30:18.092247 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.087049    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 02:30:18.092247 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.087052    2570 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 02:30:18.092247 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.087054    2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 02:30:18.092734 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.087057    2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 02:30:18.092734 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.087060    2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 02:30:18.092734 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.087062    2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 02:30:18.092734 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.087065    2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 02:30:18.092734 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.087067    2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 02:30:18.092734 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.087070    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 02:30:18.092734 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.087072    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 02:30:18.092734 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.087075    2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 02:30:18.092734 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.087077    2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 02:30:18.092734 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.087081    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 02:30:18.092734 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.087084    2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 02:30:18.092734 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.087086    2570 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 02:30:18.092734 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.087089    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 02:30:18.092734 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.087091    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 02:30:18.092734 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.087094    2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 02:30:18.092734 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.087096    2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 02:30:18.092734 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.087100    2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 02:30:18.092734 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.087102    2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 02:30:18.092734 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.087105    2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 02:30:18.092734 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.087107    2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 02:30:18.093254 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.087110    2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 02:30:18.093254 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.087112    2570 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 02:30:18.093254 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.087115    2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 02:30:18.093254 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.087117    2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 02:30:18.093254 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.087120    2570 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 02:30:18.093254 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.087123    2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 02:30:18.093254 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.087125    2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 02:30:18.093254 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.087128    2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 02:30:18.093254 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.087130    2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 02:30:18.093254 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.087133    2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 02:30:18.093254 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.087135    2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 02:30:18.093254 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.087137    2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 02:30:18.093254 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.087140    2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 02:30:18.093254 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.087143    2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 02:30:18.093254 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.087146    2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 02:30:18.093254 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.087149    2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 02:30:18.093254 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.087151    2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 02:30:18.093254 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.087154    2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 02:30:18.093254 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.087157    2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 02:30:18.093714 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.087159    2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 02:30:18.093714 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.087161    2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 02:30:18.093714 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.087164    2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 02:30:18.093714 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.087167    2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 02:30:18.093714 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.087169    2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 02:30:18.093714 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.087172    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 02:30:18.093714 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.087174    2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 02:30:18.093714 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.087177    2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 02:30:18.093714 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.087179    2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 02:30:18.093714 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.087181    2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 02:30:18.093714 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.087184    2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 02:30:18.093714 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.087187    2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 02:30:18.093714 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.087189    2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 02:30:18.093714 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.087192    2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 02:30:18.093714 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.087195    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 02:30:18.093714 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.087197    2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 02:30:18.093714 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.087200    2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 02:30:18.093714 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.087203    2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 02:30:18.093714 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.087205    2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 02:30:18.094187 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:18.087207    2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 02:30:18.094187 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.087213    2570 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 02:30:18.094187 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.087308    2570 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 09 02:30:18.094187 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.089551    2570 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 09 02:30:18.094187 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.090554    2570 server.go:1019] "Starting client certificate rotation"
Sep 09 02:30:18.094187 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.090648    2570 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 02:30:18.094187 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.090681    2570 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 02:30:18.094187 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.093337    2570 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 02:30:18.096185 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.096167    2570 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 02:30:18.107842 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.107821    2570 log.go:25] "Validated CRI v1 runtime API"
Sep 09 02:30:18.111247 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.111228    2570 log.go:25] "Validated CRI v1 image API"
Sep 09 02:30:18.112033 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.112016    2570 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 09 02:30:18.115249 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.115229    2570 fs.go:135] Filesystem UUIDs: map[00eade00-e314-4e2e-b63f-7b741b9b6a70:/dev/nvme0n1p4 33576a36-d819-4d48-8fc2-22723b78b620:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Sep 09 02:30:18.115321 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.115247    2570 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 09 02:30:18.121480 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.121367    2570 manager.go:216] Machine: {Timestamp:2026-09-09 02:30:18.118755451 +0000 UTC m=+0.373059079 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099066 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b541fe6079b37d647b60f14889744 SystemUUID:ec2b541f-e607-9b37-d647-b60f14889744 BootID:bda8a0cf-6443-4f25-8a84-0df864fb0360 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ad:a0:5f:0e:c3 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ad:a0:5f:0e:c3 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:e2:bd:a9:1f:6e:4b Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 09 02:30:18.121480 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.121466    2570 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 09 02:30:18.121653 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.121538    2570 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 09 02:30:18.123220 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.123203    2570 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 02:30:18.124839 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.124814    2570 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 09 02:30:18.125018 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.124845    2570 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-188.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 09 02:30:18.126477 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.126465    2570 topology_manager.go:138] "Creating topology manager with none policy"
Sep 09 02:30:18.126532 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.126481    2570 container_manager_linux.go:309] "Creating device plugin manager"
Sep 09 02:30:18.126532 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.126493    2570 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 02:30:18.126532 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.126513    2570 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 02:30:18.128420 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.128408    2570 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 09 02:30:18.131219 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.131205    2570 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 02:30:18.131352 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.131340    2570 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 09 02:30:18.131417 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.131377    2570 kubelet.go:486] "Attempting to sync node with API server"
Sep 09 02:30:18.131417 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.131392    2570 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 09 02:30:18.131417 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.131407    2570 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 09 02:30:18.131533 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.131420    2570 kubelet.go:398] "Adding apiserver pod source"
Sep 09 02:30:18.131533 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.131433    2570 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 09 02:30:18.131766 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.131752    2570 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 02:30:18.131825 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.131775    2570 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 02:30:18.134080 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.134064    2570 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 09 02:30:18.137017 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.137003    2570 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 09 02:30:18.137079 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.137034    2570 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 09 02:30:18.137201 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.137191    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 09 02:30:18.137235 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.137206    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 09 02:30:18.137235 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.137212    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 09 02:30:18.137235 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.137217    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 09 02:30:18.137235 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.137225    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 09 02:30:18.137235 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.137231    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 09 02:30:18.137766 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.137756    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 09 02:30:18.137795 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.137769    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 09 02:30:18.137795 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.137778    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 09 02:30:18.137795 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.137783    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 09 02:30:18.137880 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.137797    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 09 02:30:18.137880 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.137804    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 09 02:30:18.138555 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.138545    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 09 02:30:18.138593 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.138556    2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 09 02:30:18.142225 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.142210    2570 server.go:1268] "Started kubelet"
Sep 09 02:30:18.142380 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.142342    2570 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 09 02:30:18.142468 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.142380    2570 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 09 02:30:18.142468 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.142449    2570 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 09 02:30:18.142949 ip-10-0-136-188 systemd[1]: Started Kubernetes Kubelet.
Sep 09 02:30:18.147759 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.147146    2570 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 09 02:30:18.147759 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.147209    2570 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-188.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 09 02:30:18.147759 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:18.147547    2570 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 09 02:30:18.148226 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:18.148063    2570 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-188.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 09 02:30:18.148643 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:18.148601    2570 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 09 02:30:18.149009 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.148994    2570 server.go:310] "Adding debug handlers to kubelet server"
Sep 09 02:30:18.152730 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:18.151665    2570 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-188.ec2.internal.18d38630199843ac  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-188.ec2.internal,UID:ip-10-0-136-188.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-188.ec2.internal,},FirstTimestamp:2026-09-09 02:30:18.142180268 +0000 UTC m=+0.396483898,LastTimestamp:2026-09-09 02:30:18.142180268 +0000 UTC m=+0.396483898,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-188.ec2.internal,}"
Sep 09 02:30:18.153572 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.153551    2570 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 09 02:30:18.154226 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.154211    2570 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 09 02:30:18.154319 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:18.154284    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-188.ec2.internal\" not found"
Sep 09 02:30:18.154965 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.154946    2570 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 09 02:30:18.154965 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.154949    2570 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 09 02:30:18.155125 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.154987    2570 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 09 02:30:18.155125 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.155011    2570 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 09 02:30:18.155125 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.155024    2570 factory.go:55] Registering systemd factory
Sep 09 02:30:18.155125 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.155033    2570 factory.go:223] Registration of the systemd container factory successfully
Sep 09 02:30:18.155125 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.155046    2570 reconstruct.go:97] "Volume reconstruction finished"
Sep 09 02:30:18.155125 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.155054    2570 reconciler.go:29] "Reconciler: start to sync state"
Sep 09 02:30:18.155375 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.155214    2570 factory.go:153] Registering CRI-O factory
Sep 09 02:30:18.155375 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.155224    2570 factory.go:223] Registration of the crio container factory successfully
Sep 09 02:30:18.155375 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.155241    2570 factory.go:103] Registering Raw factory
Sep 09 02:30:18.155375 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.155254    2570 manager.go:1196] Started watching for new ooms in manager
Sep 09 02:30:18.155667 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.155655    2570 manager.go:349] Starting recovery of all containers
Sep 09 02:30:18.163510 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.163490    2570 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-wqhhq"
Sep 09 02:30:18.165957 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:18.165931    2570 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 09 02:30:18.166139 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:18.166114    2570 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-136-188.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 09 02:30:18.167008 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.166992    2570 manager.go:354] Recovery completed
Sep 09 02:30:18.171045 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.171024    2570 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-wqhhq"
Sep 09 02:30:18.171121 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.171051    2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 02:30:18.175262 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.175245    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-188.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 02:30:18.175335 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.175272    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-188.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 02:30:18.175335 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.175283    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-188.ec2.internal" event="NodeHasSufficientPID"
Sep 09 02:30:18.175810 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.175785    2570 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 09 02:30:18.175876 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.175812    2570 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 09 02:30:18.175876 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.175830    2570 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 02:30:18.175876 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.175798    2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 02:30:18.176578 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.176561    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-188.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 02:30:18.176657 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.176596    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-188.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 02:30:18.176657 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.176611    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-188.ec2.internal" event="NodeHasSufficientPID"
Sep 09 02:30:18.176657 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.176655    2570 policy_none.go:49] "None policy: Start"
Sep 09 02:30:18.176813 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.176671    2570 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 09 02:30:18.176813 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.176683    2570 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 09 02:30:18.177585 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.177574    2570 policy_none.go:47] "Start"
Sep 09 02:30:18.213537 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.213521    2570 manager.go:337] "Starting Device Plugin manager"
Sep 09 02:30:18.213625 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:18.213560    2570 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 09 02:30:18.213625 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.213570    2570 server.go:87] "Starting device plugin registration server"
Sep 09 02:30:18.213817 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.213792    2570 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 09 02:30:18.213872 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.213807    2570 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 09 02:30:18.213960 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.213947    2570 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 09 02:30:18.214344 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.214033    2570 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 09 02:30:18.214344 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.214046    2570 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 09 02:30:18.214467 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:18.214419    2570 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 09 02:30:18.214467 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:18.214464    2570 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-188.ec2.internal\" not found"
Sep 09 02:30:18.215026 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.214994    2570 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 09 02:30:18.216137 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.216120    2570 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 09 02:30:18.216209 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.216176    2570 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 09 02:30:18.216209 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.216196    2570 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 09 02:30:18.216291 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:18.216238    2570 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 09 02:30:18.220302 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.220285    2570 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 02:30:18.314154 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.314130    2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 02:30:18.315002 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.314987    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-188.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 02:30:18.315061 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.315018    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-188.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 02:30:18.315061 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.315031    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-188.ec2.internal" event="NodeHasSufficientPID"
Sep 09 02:30:18.315061 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.315051    2570 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-136-188.ec2.internal"
Sep 09 02:30:18.317103 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.317087    2570 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-136-188.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-188.ec2.internal"]
Sep 09 02:30:18.317162 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.317137    2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 02:30:18.317844 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.317829    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-188.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 02:30:18.317898 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.317854    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-188.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 02:30:18.317898 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.317864    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-188.ec2.internal" event="NodeHasSufficientPID"
Sep 09 02:30:18.318882 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.318870    2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 02:30:18.319022 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.318986    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-188.ec2.internal"
Sep 09 02:30:18.319022 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.319012    2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 02:30:18.319534 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.319516    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-188.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 02:30:18.319616 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.319521    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-188.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 02:30:18.319616 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.319546    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-188.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 02:30:18.319616 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.319561    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-188.ec2.internal" event="NodeHasSufficientPID"
Sep 09 02:30:18.319616 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.319563    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-188.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 02:30:18.319616 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.319577    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-188.ec2.internal" event="NodeHasSufficientPID"
Sep 09 02:30:18.320607 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.320593    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-188.ec2.internal"
Sep 09 02:30:18.320655 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.320617    2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 02:30:18.321294 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.321279    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-188.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 02:30:18.321359 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.321307    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-188.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 02:30:18.321359 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.321321    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-188.ec2.internal" event="NodeHasSufficientPID"
Sep 09 02:30:18.328392 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.328379    2570 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-136-188.ec2.internal"
Sep 09 02:30:18.328438 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:18.328398    2570 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-188.ec2.internal\": node \"ip-10-0-136-188.ec2.internal\" not found"
Sep 09 02:30:18.346734 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:18.346560    2570 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-188.ec2.internal\" not found" node="ip-10-0-136-188.ec2.internal"
Sep 09 02:30:18.350876 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:18.350861    2570 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-188.ec2.internal\" not found" node="ip-10-0-136-188.ec2.internal"
Sep 09 02:30:18.354854 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:18.354838    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-188.ec2.internal\" not found"
Sep 09 02:30:18.356996 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.356964    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/871e64cd6b8e47e52f74676ffa7cfe0c-config\") pod \"kube-apiserver-proxy-ip-10-0-136-188.ec2.internal\" (UID: \"871e64cd6b8e47e52f74676ffa7cfe0c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-188.ec2.internal"
Sep 09 02:30:18.357071 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.357003    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7c9f79a4d290b3116ce4e01e83369531-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-188.ec2.internal\" (UID: \"7c9f79a4d290b3116ce4e01e83369531\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-188.ec2.internal"
Sep 09 02:30:18.357071 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.357018    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7c9f79a4d290b3116ce4e01e83369531-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-188.ec2.internal\" (UID: \"7c9f79a4d290b3116ce4e01e83369531\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-188.ec2.internal"
Sep 09 02:30:18.455556 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:18.455540    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-188.ec2.internal\" not found"
Sep 09 02:30:18.457749 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.457732    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/871e64cd6b8e47e52f74676ffa7cfe0c-config\") pod \"kube-apiserver-proxy-ip-10-0-136-188.ec2.internal\" (UID: \"871e64cd6b8e47e52f74676ffa7cfe0c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-188.ec2.internal"
Sep 09 02:30:18.457822 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.457761    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7c9f79a4d290b3116ce4e01e83369531-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-188.ec2.internal\" (UID: \"7c9f79a4d290b3116ce4e01e83369531\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-188.ec2.internal"
Sep 09 02:30:18.457822 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.457785    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7c9f79a4d290b3116ce4e01e83369531-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-188.ec2.internal\" (UID: \"7c9f79a4d290b3116ce4e01e83369531\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-188.ec2.internal"
Sep 09 02:30:18.457894 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.457838    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/871e64cd6b8e47e52f74676ffa7cfe0c-config\") pod \"kube-apiserver-proxy-ip-10-0-136-188.ec2.internal\" (UID: \"871e64cd6b8e47e52f74676ffa7cfe0c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-188.ec2.internal"
Sep 09 02:30:18.457894 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.457871    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7c9f79a4d290b3116ce4e01e83369531-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-188.ec2.internal\" (UID: \"7c9f79a4d290b3116ce4e01e83369531\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-188.ec2.internal"
Sep 09 02:30:18.457894 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.457838    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7c9f79a4d290b3116ce4e01e83369531-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-188.ec2.internal\" (UID: \"7c9f79a4d290b3116ce4e01e83369531\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-188.ec2.internal"
Sep 09 02:30:18.555659 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:18.555636    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-188.ec2.internal\" not found"
Sep 09 02:30:18.651719 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.651658    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-188.ec2.internal"
Sep 09 02:30:18.654934 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:18.654914    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-188.ec2.internal"
Sep 09 02:30:18.656461 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:18.656122    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-188.ec2.internal\" not found"
Sep 09 02:30:18.756486 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:18.756460    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-188.ec2.internal\" not found"
Sep 09 02:30:18.856917 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:18.856900    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-188.ec2.internal\" not found"
Sep 09 02:30:18.957324 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:18.957264    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-188.ec2.internal\" not found"
Sep 09 02:30:19.058262 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:19.058236    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-188.ec2.internal\" not found"
Sep 09 02:30:19.090483 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:19.090458    2570 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 09 02:30:19.090891 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:19.090607    2570 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 09 02:30:19.158917 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:19.158896    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-188.ec2.internal\" not found"
Sep 09 02:30:19.162336 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:19.162318    2570 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 02:30:19.173750 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:19.173726    2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-08 02:25:18 +0000 UTC" deadline="2028-03-26 17:06:51.760149679 +0000 UTC"
Sep 09 02:30:19.173750 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:19.173748    2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13550h36m32.586404142s"
Sep 09 02:30:19.233876 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:19.233855    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod871e64cd6b8e47e52f74676ffa7cfe0c.slice/crio-65a3c3aa363dfb01f2c6294ffccf6b2d3d158191d369f5b08a59d2e0a8891e52 WatchSource:0}: Error finding container 65a3c3aa363dfb01f2c6294ffccf6b2d3d158191d369f5b08a59d2e0a8891e52: Status 404 returned error can't find the container with id 65a3c3aa363dfb01f2c6294ffccf6b2d3d158191d369f5b08a59d2e0a8891e52
Sep 09 02:30:19.234274 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:19.234251    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c9f79a4d290b3116ce4e01e83369531.slice/crio-7fcf4f71ced81d86d8cc81a54a565832a07f91bd2c34aadb17ea3fe726e748fc WatchSource:0}: Error finding container 7fcf4f71ced81d86d8cc81a54a565832a07f91bd2c34aadb17ea3fe726e748fc: Status 404 returned error can't find the container with id 7fcf4f71ced81d86d8cc81a54a565832a07f91bd2c34aadb17ea3fe726e748fc
Sep 09 02:30:19.239707 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:19.239691    2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 02:30:19.259759 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:19.259739    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-188.ec2.internal\" not found"
Sep 09 02:30:19.360134 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:19.360119    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-188.ec2.internal\" not found"
Sep 09 02:30:19.460524 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:19.460506    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-188.ec2.internal\" not found"
Sep 09 02:30:19.560603 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:19.560548    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-188.ec2.internal\" not found"
Sep 09 02:30:19.565117 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:19.565096    2570 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 02:30:19.660653 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:19.660625    2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-188.ec2.internal\" not found"
Sep 09 02:30:19.707429 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:19.707403    2570 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 02:30:19.755074 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:19.755047    2570 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-188.ec2.internal"
Sep 09 02:30:19.767482 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:19.767461    2570 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 02:30:19.768706 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:19.768688    2570 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-188.ec2.internal"
Sep 09 02:30:19.780777 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:19.780759    2570 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 02:30:20.132545 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.132503    2570 apiserver.go:52] "Watching apiserver"
Sep 09 02:30:20.142194 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.142168    2570 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 09 02:30:20.144038 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.143029    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-zw227","openshift-multus/multus-phjk6","openshift-multus/network-metrics-daemon-zrljs","openshift-network-diagnostics/network-check-target-6gkgn","kube-system/konnectivity-agent-kxvb7","openshift-cluster-node-tuning-operator/tuned-sv9fq","openshift-dns/node-resolver-d7t89","openshift-image-registry/node-ca-zg4dm","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-188.ec2.internal","openshift-network-operator/iptables-alerter-h2sdl","openshift-ovn-kubernetes/ovnkube-node-xcqxh","kube-system/global-pull-secret-syncer-952h4","kube-system/kube-apiserver-proxy-ip-10-0-136-188.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2f82x"]
Sep 09 02:30:20.146093 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.146069    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-d7t89"
Sep 09 02:30:20.148362 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.148338    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xcqxh"
Sep 09 02:30:20.149070 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.149016    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-sj7xn\""
Sep 09 02:30:20.149070 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.149037    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 09 02:30:20.149264 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.149250    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 09 02:30:20.150836 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.150726    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zrljs"
Sep 09 02:30:20.150836 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:20.150792    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zrljs" podUID="d786708a-bbb1-4250-a2d3-84dc2299d71a"
Sep 09 02:30:20.151459 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.151226    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-vbnll\""
Sep 09 02:30:20.151459 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.151260    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 09 02:30:20.151459 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.151300    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 09 02:30:20.153090 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.153069    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6gkgn"
Sep 09 02:30:20.153196 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:20.153175    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6gkgn" podUID="ac0a0b68-d43e-4ca8-8c4d-39bbe5e7519c"
Sep 09 02:30:20.153334 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.153313    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 09 02:30:20.153334 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.153321    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 09 02:30:20.153499 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.153344    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 09 02:30:20.153602 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.153587    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 09 02:30:20.154669 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.154160    2570 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 09 02:30:20.155561 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.155540    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-kxvb7"
Sep 09 02:30:20.158729 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.158711    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-sv9fq"
Sep 09 02:30:20.158812 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.158762    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 09 02:30:20.159091 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.159071    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-qhx7r\""
Sep 09 02:30:20.159204 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.159190    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 09 02:30:20.161914 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.161897    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2f82x"
Sep 09 02:30:20.162170 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.162149    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 09 02:30:20.162478 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.162459    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-6b4bg\""
Sep 09 02:30:20.162798 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.162782    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 09 02:30:20.164317 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.164296    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zg4dm"
Sep 09 02:30:20.164863 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.164844    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 09 02:30:20.165421 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.165404    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 09 02:30:20.166041 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.166024    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/57852f55-dada-41bc-927d-466452a75957-host-kubelet\") pod \"ovnkube-node-xcqxh\" (UID: \"57852f55-dada-41bc-927d-466452a75957\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcqxh"
Sep 09 02:30:20.166133 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.166063    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 09 02:30:20.166133 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.166075    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/57852f55-dada-41bc-927d-466452a75957-systemd-units\") pod \"ovnkube-node-xcqxh\" (UID: \"57852f55-dada-41bc-927d-466452a75957\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcqxh"
Sep 09 02:30:20.166133 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.166095    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-zr6gb\""
Sep 09 02:30:20.166133 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.166104    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d786708a-bbb1-4250-a2d3-84dc2299d71a-metrics-certs\") pod \"network-metrics-daemon-zrljs\" (UID: \"d786708a-bbb1-4250-a2d3-84dc2299d71a\") " pod="openshift-multus/network-metrics-daemon-zrljs"
Sep 09 02:30:20.166339 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.166144    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a0d4f170-d793-4602-9c77-afdf152cd687-host\") pod \"tuned-sv9fq\" (UID: \"a0d4f170-d793-4602-9c77-afdf152cd687\") " pod="openshift-cluster-node-tuning-operator/tuned-sv9fq"
Sep 09 02:30:20.166339 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.166172    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90413a10-3c11-4127-b7c8-6e8ac09b1fc9-tmp-dir\") pod \"node-resolver-d7t89\" (UID: \"90413a10-3c11-4127-b7c8-6e8ac09b1fc9\") " pod="openshift-dns/node-resolver-d7t89"
Sep 09 02:30:20.166339 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.166198    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6ddt\" (UniqueName: \"kubernetes.io/projected/90413a10-3c11-4127-b7c8-6e8ac09b1fc9-kube-api-access-r6ddt\") pod \"node-resolver-d7t89\" (UID: \"90413a10-3c11-4127-b7c8-6e8ac09b1fc9\") " pod="openshift-dns/node-resolver-d7t89"
Sep 09 02:30:20.166339 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.166229    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/57852f55-dada-41bc-927d-466452a75957-host-slash\") pod \"ovnkube-node-xcqxh\" (UID: \"57852f55-dada-41bc-927d-466452a75957\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcqxh"
Sep 09 02:30:20.166339 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.166253    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/57852f55-dada-41bc-927d-466452a75957-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xcqxh\" (UID: \"57852f55-dada-41bc-927d-466452a75957\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcqxh"
Sep 09 02:30:20.166339 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.166284    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/57852f55-dada-41bc-927d-466452a75957-ovn-node-metrics-cert\") pod \"ovnkube-node-xcqxh\" (UID: \"57852f55-dada-41bc-927d-466452a75957\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcqxh"
Sep 09 02:30:20.166339 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.166307    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a0d4f170-d793-4602-9c77-afdf152cd687-etc-modprobe-d\") pod \"tuned-sv9fq\" (UID: \"a0d4f170-d793-4602-9c77-afdf152cd687\") " pod="openshift-cluster-node-tuning-operator/tuned-sv9fq"
Sep 09 02:30:20.166339 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.166329    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/57852f55-dada-41bc-927d-466452a75957-var-lib-openvswitch\") pod \"ovnkube-node-xcqxh\" (UID: \"57852f55-dada-41bc-927d-466452a75957\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcqxh"
Sep 09 02:30:20.166705 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.166353    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a0d4f170-d793-4602-9c77-afdf152cd687-etc-sysctl-conf\") pod \"tuned-sv9fq\" (UID: \"a0d4f170-d793-4602-9c77-afdf152cd687\") " pod="openshift-cluster-node-tuning-operator/tuned-sv9fq"
Sep 09 02:30:20.166705 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.166402    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a0d4f170-d793-4602-9c77-afdf152cd687-run\") pod \"tuned-sv9fq\" (UID: \"a0d4f170-d793-4602-9c77-afdf152cd687\") " pod="openshift-cluster-node-tuning-operator/tuned-sv9fq"
Sep 09 02:30:20.166705 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.166426    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzrbq\" (UniqueName: \"kubernetes.io/projected/a0d4f170-d793-4602-9c77-afdf152cd687-kube-api-access-rzrbq\") pod \"tuned-sv9fq\" (UID: \"a0d4f170-d793-4602-9c77-afdf152cd687\") " pod="openshift-cluster-node-tuning-operator/tuned-sv9fq"
Sep 09 02:30:20.166705 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.166451    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/57852f55-dada-41bc-927d-466452a75957-host-run-netns\") pod \"ovnkube-node-xcqxh\" (UID: \"57852f55-dada-41bc-927d-466452a75957\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcqxh"
Sep 09 02:30:20.166705 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.166475    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drrcs\" (UniqueName: \"kubernetes.io/projected/ac0a0b68-d43e-4ca8-8c4d-39bbe5e7519c-kube-api-access-drrcs\") pod \"network-check-target-6gkgn\" (UID: \"ac0a0b68-d43e-4ca8-8c4d-39bbe5e7519c\") " pod="openshift-network-diagnostics/network-check-target-6gkgn"
Sep 09 02:30:20.166705 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.166496    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a0d4f170-d793-4602-9c77-afdf152cd687-etc-systemd\") pod \"tuned-sv9fq\" (UID: \"a0d4f170-d793-4602-9c77-afdf152cd687\") " pod="openshift-cluster-node-tuning-operator/tuned-sv9fq"
Sep 09 02:30:20.166705 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.166531    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/57852f55-dada-41bc-927d-466452a75957-etc-openvswitch\") pod \"ovnkube-node-xcqxh\" (UID: \"57852f55-dada-41bc-927d-466452a75957\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcqxh"
Sep 09 02:30:20.166705 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.166553    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/57852f55-dada-41bc-927d-466452a75957-node-log\") pod \"ovnkube-node-xcqxh\" (UID: \"57852f55-dada-41bc-927d-466452a75957\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcqxh"
Sep 09 02:30:20.166705 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.166575    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/57852f55-dada-41bc-927d-466452a75957-host-cni-netd\") pod \"ovnkube-node-xcqxh\" (UID: \"57852f55-dada-41bc-927d-466452a75957\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcqxh"
Sep 09 02:30:20.166705 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.166610    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/57852f55-dada-41bc-927d-466452a75957-ovnkube-config\") pod \"ovnkube-node-xcqxh\" (UID: \"57852f55-dada-41bc-927d-466452a75957\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcqxh"
Sep 09 02:30:20.166705 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.166634    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a0d4f170-d793-4602-9c77-afdf152cd687-etc-kubernetes\") pod \"tuned-sv9fq\" (UID: \"a0d4f170-d793-4602-9c77-afdf152cd687\") " pod="openshift-cluster-node-tuning-operator/tuned-sv9fq"
Sep 09 02:30:20.166705 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.166679    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/57852f55-dada-41bc-927d-466452a75957-host-run-ovn-kubernetes\") pod \"ovnkube-node-xcqxh\" (UID: \"57852f55-dada-41bc-927d-466452a75957\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcqxh"
Sep 09 02:30:20.166705 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.166706    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/57852f55-dada-41bc-927d-466452a75957-env-overrides\") pod \"ovnkube-node-xcqxh\" (UID: \"57852f55-dada-41bc-927d-466452a75957\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcqxh"
Sep 09 02:30:20.167290 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.166732    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/679d664f-a0b2-4ad9-94e4-014aef8f2e16-agent-certs\") pod \"konnectivity-agent-kxvb7\" (UID: \"679d664f-a0b2-4ad9-94e4-014aef8f2e16\") " pod="kube-system/konnectivity-agent-kxvb7"
Sep 09 02:30:20.167290 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.166758    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a0d4f170-d793-4602-9c77-afdf152cd687-etc-sysconfig\") pod \"tuned-sv9fq\" (UID: \"a0d4f170-d793-4602-9c77-afdf152cd687\") " pod="openshift-cluster-node-tuning-operator/tuned-sv9fq"
Sep 09 02:30:20.167290 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.166685    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-phjk6"
Sep 09 02:30:20.167290 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.166801    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a0d4f170-d793-4602-9c77-afdf152cd687-etc-sysctl-d\") pod \"tuned-sv9fq\" (UID: \"a0d4f170-d793-4602-9c77-afdf152cd687\") " pod="openshift-cluster-node-tuning-operator/tuned-sv9fq"
Sep 09 02:30:20.167290 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.166830    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/90413a10-3c11-4127-b7c8-6e8ac09b1fc9-hosts-file\") pod \"node-resolver-d7t89\" (UID: \"90413a10-3c11-4127-b7c8-6e8ac09b1fc9\") " pod="openshift-dns/node-resolver-d7t89"
Sep 09 02:30:20.167290 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.166894    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/57852f55-dada-41bc-927d-466452a75957-log-socket\") pod \"ovnkube-node-xcqxh\" (UID: \"57852f55-dada-41bc-927d-466452a75957\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcqxh"
Sep 09 02:30:20.167290 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.166931    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/57852f55-dada-41bc-927d-466452a75957-host-cni-bin\") pod \"ovnkube-node-xcqxh\" (UID: \"57852f55-dada-41bc-927d-466452a75957\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcqxh"
Sep 09 02:30:20.167290 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.166968    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/57852f55-dada-41bc-927d-466452a75957-ovnkube-script-lib\") pod \"ovnkube-node-xcqxh\" (UID: \"57852f55-dada-41bc-927d-466452a75957\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcqxh"
Sep 09 02:30:20.167290 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.167038    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bm8w\" (UniqueName: \"kubernetes.io/projected/57852f55-dada-41bc-927d-466452a75957-kube-api-access-7bm8w\") pod \"ovnkube-node-xcqxh\" (UID: \"57852f55-dada-41bc-927d-466452a75957\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcqxh"
Sep 09 02:30:20.167290 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.167074    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vnp9\" (UniqueName: \"kubernetes.io/projected/d786708a-bbb1-4250-a2d3-84dc2299d71a-kube-api-access-5vnp9\") pod \"network-metrics-daemon-zrljs\" (UID: \"d786708a-bbb1-4250-a2d3-84dc2299d71a\") " pod="openshift-multus/network-metrics-daemon-zrljs"
Sep 09 02:30:20.167290 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.167111    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a0d4f170-d793-4602-9c77-afdf152cd687-sys\") pod \"tuned-sv9fq\" (UID: \"a0d4f170-d793-4602-9c77-afdf152cd687\") " pod="openshift-cluster-node-tuning-operator/tuned-sv9fq"
Sep 09 02:30:20.167820 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.167327    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/57852f55-dada-41bc-927d-466452a75957-run-systemd\") pod \"ovnkube-node-xcqxh\" (UID: \"57852f55-dada-41bc-927d-466452a75957\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcqxh"
Sep 09 02:30:20.167820 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.167355    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 09 02:30:20.167820 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.167391    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/57852f55-dada-41bc-927d-466452a75957-run-openvswitch\") pod \"ovnkube-node-xcqxh\" (UID: \"57852f55-dada-41bc-927d-466452a75957\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcqxh"
Sep 09 02:30:20.167820 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.167437    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/57852f55-dada-41bc-927d-466452a75957-run-ovn\") pod \"ovnkube-node-xcqxh\" (UID: \"57852f55-dada-41bc-927d-466452a75957\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcqxh"
Sep 09 02:30:20.167820 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.167477    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/679d664f-a0b2-4ad9-94e4-014aef8f2e16-konnectivity-ca\") pod \"konnectivity-agent-kxvb7\" (UID: \"679d664f-a0b2-4ad9-94e4-014aef8f2e16\") " pod="kube-system/konnectivity-agent-kxvb7"
Sep 09 02:30:20.167820 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.167522    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a0d4f170-d793-4602-9c77-afdf152cd687-lib-modules\") pod \"tuned-sv9fq\" (UID: \"a0d4f170-d793-4602-9c77-afdf152cd687\") " pod="openshift-cluster-node-tuning-operator/tuned-sv9fq"
Sep 09 02:30:20.167820 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.167563    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a0d4f170-d793-4602-9c77-afdf152cd687-var-lib-kubelet\") pod \"tuned-sv9fq\" (UID: \"a0d4f170-d793-4602-9c77-afdf152cd687\") " pod="openshift-cluster-node-tuning-operator/tuned-sv9fq"
Sep 09 02:30:20.167820 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.167599    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a0d4f170-d793-4602-9c77-afdf152cd687-etc-tuned\") pod \"tuned-sv9fq\" (UID: \"a0d4f170-d793-4602-9c77-afdf152cd687\") " pod="openshift-cluster-node-tuning-operator/tuned-sv9fq"
Sep 09 02:30:20.167820 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.167624    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a0d4f170-d793-4602-9c77-afdf152cd687-tmp\") pod \"tuned-sv9fq\" (UID: \"a0d4f170-d793-4602-9c77-afdf152cd687\") " pod="openshift-cluster-node-tuning-operator/tuned-sv9fq"
Sep 09 02:30:20.167820 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.167740    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 09 02:30:20.167820 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.167774    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 09 02:30:20.167820 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.167799    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-rpqjd\""
Sep 09 02:30:20.170170 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.169667    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-hxskh\""
Sep 09 02:30:20.170170 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.169758    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-h2sdl"
Sep 09 02:30:20.170170 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.169784    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 09 02:30:20.170170 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.169813    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 09 02:30:20.170170 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.169897    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 09 02:30:20.171327 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.171306    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 09 02:30:20.172315 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.172282    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-zw227"
Sep 09 02:30:20.173879 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.173852    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 09 02:30:20.174426 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.174274    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 09 02:30:20.174744 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.174724    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 09 02:30:20.175725 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.175704    2570 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 02:30:20.176088 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.176066    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-c2dc5\""
Sep 09 02:30:20.176720 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.176702    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 09 02:30:20.176803 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.176772    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-fs8xm\""
Sep 09 02:30:20.178006 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.177807    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-952h4"
Sep 09 02:30:20.178006 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:20.177996    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-952h4" podUID="37ef684a-ce44-43fc-a803-519ce7d4d225"
Sep 09 02:30:20.179758 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.179741    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 09 02:30:20.195640 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.195617    2570 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-qzxmz"
Sep 09 02:30:20.205308 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.205287    2570 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-qzxmz"
Sep 09 02:30:20.222267 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.222207    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-188.ec2.internal" event={"ID":"871e64cd6b8e47e52f74676ffa7cfe0c","Type":"ContainerStarted","Data":"65a3c3aa363dfb01f2c6294ffccf6b2d3d158191d369f5b08a59d2e0a8891e52"}
Sep 09 02:30:20.223373 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.223347    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-188.ec2.internal" event={"ID":"7c9f79a4d290b3116ce4e01e83369531","Type":"ContainerStarted","Data":"7fcf4f71ced81d86d8cc81a54a565832a07f91bd2c34aadb17ea3fe726e748fc"}
Sep 09 02:30:20.255884 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.255858    2570 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 09 02:30:20.268463 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.268404    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/37ef684a-ce44-43fc-a803-519ce7d4d225-original-pull-secret\") pod \"global-pull-secret-syncer-952h4\" (UID: \"37ef684a-ce44-43fc-a803-519ce7d4d225\") " pod="kube-system/global-pull-secret-syncer-952h4"
Sep 09 02:30:20.268597 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.268470    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6w59\" (UniqueName: \"kubernetes.io/projected/721fa3c1-a7d2-4245-a7bd-76eb7facfc36-kube-api-access-k6w59\") pod \"node-ca-zg4dm\" (UID: \"721fa3c1-a7d2-4245-a7bd-76eb7facfc36\") " pod="openshift-image-registry/node-ca-zg4dm"
Sep 09 02:30:20.268597 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.268501    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/57852f55-dada-41bc-927d-466452a75957-run-systemd\") pod \"ovnkube-node-xcqxh\" (UID: \"57852f55-dada-41bc-927d-466452a75957\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcqxh"
Sep 09 02:30:20.268597 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.268551    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/57852f55-dada-41bc-927d-466452a75957-run-systemd\") pod \"ovnkube-node-xcqxh\" (UID: \"57852f55-dada-41bc-927d-466452a75957\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcqxh"
Sep 09 02:30:20.268597 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.268588    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/679d664f-a0b2-4ad9-94e4-014aef8f2e16-konnectivity-ca\") pod \"konnectivity-agent-kxvb7\" (UID: \"679d664f-a0b2-4ad9-94e4-014aef8f2e16\") " pod="kube-system/konnectivity-agent-kxvb7"
Sep 09 02:30:20.268798 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.268640    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a0d4f170-d793-4602-9c77-afdf152cd687-var-lib-kubelet\") pod \"tuned-sv9fq\" (UID: \"a0d4f170-d793-4602-9c77-afdf152cd687\") " pod="openshift-cluster-node-tuning-operator/tuned-sv9fq"
Sep 09 02:30:20.268798 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.268772    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a0d4f170-d793-4602-9c77-afdf152cd687-var-lib-kubelet\") pod \"tuned-sv9fq\" (UID: \"a0d4f170-d793-4602-9c77-afdf152cd687\") " pod="openshift-cluster-node-tuning-operator/tuned-sv9fq"
Sep 09 02:30:20.268798 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.268784    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2m477\" (UniqueName: \"kubernetes.io/projected/d33ba4b9-3e00-405a-bc2f-9144dbfa5c8f-kube-api-access-2m477\") pod \"multus-additional-cni-plugins-zw227\" (UID: \"d33ba4b9-3e00-405a-bc2f-9144dbfa5c8f\") " pod="openshift-multus/multus-additional-cni-plugins-zw227"
Sep 09 02:30:20.268938 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.268826    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/721fa3c1-a7d2-4245-a7bd-76eb7facfc36-host\") pod \"node-ca-zg4dm\" (UID: \"721fa3c1-a7d2-4245-a7bd-76eb7facfc36\") " pod="openshift-image-registry/node-ca-zg4dm"
Sep 09 02:30:20.268938 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.268854    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/57852f55-dada-41bc-927d-466452a75957-host-kubelet\") pod \"ovnkube-node-xcqxh\" (UID: \"57852f55-dada-41bc-927d-466452a75957\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcqxh"
Sep 09 02:30:20.268938 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.268876    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/57852f55-dada-41bc-927d-466452a75957-systemd-units\") pod \"ovnkube-node-xcqxh\" (UID: \"57852f55-dada-41bc-927d-466452a75957\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcqxh"
Sep 09 02:30:20.268938 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.268899    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d786708a-bbb1-4250-a2d3-84dc2299d71a-metrics-certs\") pod \"network-metrics-daemon-zrljs\" (UID: \"d786708a-bbb1-4250-a2d3-84dc2299d71a\") " pod="openshift-multus/network-metrics-daemon-zrljs"
Sep 09 02:30:20.268938 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.268924    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ddccb44f-2b0d-4cf3-9419-31b5211ab2f5-cni-binary-copy\") pod \"multus-phjk6\" (UID: \"ddccb44f-2b0d-4cf3-9419-31b5211ab2f5\") " pod="openshift-multus/multus-phjk6"
Sep 09 02:30:20.269214 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.268946    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d33ba4b9-3e00-405a-bc2f-9144dbfa5c8f-cni-binary-copy\") pod \"multus-additional-cni-plugins-zw227\" (UID: \"d33ba4b9-3e00-405a-bc2f-9144dbfa5c8f\") " pod="openshift-multus/multus-additional-cni-plugins-zw227"
Sep 09 02:30:20.269214 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.268953    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/57852f55-dada-41bc-927d-466452a75957-host-kubelet\") pod \"ovnkube-node-xcqxh\" (UID: \"57852f55-dada-41bc-927d-466452a75957\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcqxh"
Sep 09 02:30:20.269214 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.268962    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/57852f55-dada-41bc-927d-466452a75957-systemd-units\") pod \"ovnkube-node-xcqxh\" (UID: \"57852f55-dada-41bc-927d-466452a75957\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcqxh"
Sep 09 02:30:20.269214 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.268971    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mhkj\" (UniqueName: \"kubernetes.io/projected/c3e9e8ee-29c7-4fe7-8bcd-7f2363992b3c-kube-api-access-2mhkj\") pod \"aws-ebs-csi-driver-node-2f82x\" (UID: \"c3e9e8ee-29c7-4fe7-8bcd-7f2363992b3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2f82x"
Sep 09 02:30:20.269214 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.269018    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6ddt\" (UniqueName: \"kubernetes.io/projected/90413a10-3c11-4127-b7c8-6e8ac09b1fc9-kube-api-access-r6ddt\") pod \"node-resolver-d7t89\" (UID: \"90413a10-3c11-4127-b7c8-6e8ac09b1fc9\") " pod="openshift-dns/node-resolver-d7t89"
Sep 09 02:30:20.269214 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:20.269110    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 02:30:20.269214 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.269121    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/57852f55-dada-41bc-927d-466452a75957-host-slash\") pod \"ovnkube-node-xcqxh\" (UID: \"57852f55-dada-41bc-927d-466452a75957\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcqxh"
Sep 09 02:30:20.269214 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.269142    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/57852f55-dada-41bc-927d-466452a75957-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xcqxh\" (UID: \"57852f55-dada-41bc-927d-466452a75957\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcqxh"
Sep 09 02:30:20.269214 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.269171    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a0d4f170-d793-4602-9c77-afdf152cd687-etc-modprobe-d\") pod \"tuned-sv9fq\" (UID: \"a0d4f170-d793-4602-9c77-afdf152cd687\") " pod="openshift-cluster-node-tuning-operator/tuned-sv9fq"
Sep 09 02:30:20.269596 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:20.269226    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d786708a-bbb1-4250-a2d3-84dc2299d71a-metrics-certs podName:d786708a-bbb1-4250-a2d3-84dc2299d71a nodeName:}" failed. No retries permitted until 2026-09-09 02:30:20.769193047 +0000 UTC m=+3.023496676 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d786708a-bbb1-4250-a2d3-84dc2299d71a-metrics-certs") pod "network-metrics-daemon-zrljs" (UID: "d786708a-bbb1-4250-a2d3-84dc2299d71a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 02:30:20.269596 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.269279    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/57852f55-dada-41bc-927d-466452a75957-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xcqxh\" (UID: \"57852f55-dada-41bc-927d-466452a75957\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcqxh"
Sep 09 02:30:20.269596 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.269311    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/57852f55-dada-41bc-927d-466452a75957-host-slash\") pod \"ovnkube-node-xcqxh\" (UID: \"57852f55-dada-41bc-927d-466452a75957\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcqxh"
Sep 09 02:30:20.269596 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.269342    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ddccb44f-2b0d-4cf3-9419-31b5211ab2f5-hostroot\") pod \"multus-phjk6\" (UID: \"ddccb44f-2b0d-4cf3-9419-31b5211ab2f5\") " pod="openshift-multus/multus-phjk6"
Sep 09 02:30:20.269596 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.269338    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a0d4f170-d793-4602-9c77-afdf152cd687-etc-modprobe-d\") pod \"tuned-sv9fq\" (UID: \"a0d4f170-d793-4602-9c77-afdf152cd687\") " pod="openshift-cluster-node-tuning-operator/tuned-sv9fq"
Sep 09 02:30:20.269596 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.269390    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ddccb44f-2b0d-4cf3-9419-31b5211ab2f5-etc-kubernetes\") pod \"multus-phjk6\" (UID: \"ddccb44f-2b0d-4cf3-9419-31b5211ab2f5\") " pod="openshift-multus/multus-phjk6"
Sep 09 02:30:20.269596 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.269433    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/67afff59-c0d9-43fb-9b1f-581ceab88dbd-iptables-alerter-script\") pod \"iptables-alerter-h2sdl\" (UID: \"67afff59-c0d9-43fb-9b1f-581ceab88dbd\") " pod="openshift-network-operator/iptables-alerter-h2sdl"
Sep 09 02:30:20.269596 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.269465    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d33ba4b9-3e00-405a-bc2f-9144dbfa5c8f-cnibin\") pod \"multus-additional-cni-plugins-zw227\" (UID: \"d33ba4b9-3e00-405a-bc2f-9144dbfa5c8f\") " pod="openshift-multus/multus-additional-cni-plugins-zw227"
Sep 09 02:30:20.269596 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.269495    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a0d4f170-d793-4602-9c77-afdf152cd687-run\") pod \"tuned-sv9fq\" (UID: \"a0d4f170-d793-4602-9c77-afdf152cd687\") " pod="openshift-cluster-node-tuning-operator/tuned-sv9fq"
Sep 09 02:30:20.269596 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.269519    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzrbq\" (UniqueName: \"kubernetes.io/projected/a0d4f170-d793-4602-9c77-afdf152cd687-kube-api-access-rzrbq\") pod \"tuned-sv9fq\" (UID: \"a0d4f170-d793-4602-9c77-afdf152cd687\") " pod="openshift-cluster-node-tuning-operator/tuned-sv9fq"
Sep 09 02:30:20.269596 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.269555    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ddccb44f-2b0d-4cf3-9419-31b5211ab2f5-host-var-lib-cni-multus\") pod \"multus-phjk6\" (UID: \"ddccb44f-2b0d-4cf3-9419-31b5211ab2f5\") " pod="openshift-multus/multus-phjk6"
Sep 09 02:30:20.269596 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.269580    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/67afff59-c0d9-43fb-9b1f-581ceab88dbd-host-slash\") pod \"iptables-alerter-h2sdl\" (UID: \"67afff59-c0d9-43fb-9b1f-581ceab88dbd\") " pod="openshift-network-operator/iptables-alerter-h2sdl"
Sep 09 02:30:20.270147 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.269599    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c3e9e8ee-29c7-4fe7-8bcd-7f2363992b3c-socket-dir\") pod \"aws-ebs-csi-driver-node-2f82x\" (UID: \"c3e9e8ee-29c7-4fe7-8bcd-7f2363992b3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2f82x"
Sep 09 02:30:20.270147 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.269651    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a0d4f170-d793-4602-9c77-afdf152cd687-run\") pod \"tuned-sv9fq\" (UID: \"a0d4f170-d793-4602-9c77-afdf152cd687\") " pod="openshift-cluster-node-tuning-operator/tuned-sv9fq"
Sep 09 02:30:20.270147 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.269651    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c3e9e8ee-29c7-4fe7-8bcd-7f2363992b3c-etc-selinux\") pod \"aws-ebs-csi-driver-node-2f82x\" (UID: \"c3e9e8ee-29c7-4fe7-8bcd-7f2363992b3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2f82x"
Sep 09 02:30:20.270147 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.269686    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/57852f55-dada-41bc-927d-466452a75957-host-run-netns\") pod \"ovnkube-node-xcqxh\" (UID: \"57852f55-dada-41bc-927d-466452a75957\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcqxh"
Sep 09 02:30:20.270147 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.269711    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drrcs\" (UniqueName: \"kubernetes.io/projected/ac0a0b68-d43e-4ca8-8c4d-39bbe5e7519c-kube-api-access-drrcs\") pod \"network-check-target-6gkgn\" (UID: \"ac0a0b68-d43e-4ca8-8c4d-39bbe5e7519c\") " pod="openshift-network-diagnostics/network-check-target-6gkgn"
Sep 09 02:30:20.270147 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.269741    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a0d4f170-d793-4602-9c77-afdf152cd687-etc-systemd\") pod \"tuned-sv9fq\" (UID: \"a0d4f170-d793-4602-9c77-afdf152cd687\") " pod="openshift-cluster-node-tuning-operator/tuned-sv9fq"
Sep 09 02:30:20.270147 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.269748    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/57852f55-dada-41bc-927d-466452a75957-host-run-netns\") pod \"ovnkube-node-xcqxh\" (UID: \"57852f55-dada-41bc-927d-466452a75957\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcqxh"
Sep 09 02:30:20.270147 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.269765    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/721fa3c1-a7d2-4245-a7bd-76eb7facfc36-serviceca\") pod \"node-ca-zg4dm\" (UID: \"721fa3c1-a7d2-4245-a7bd-76eb7facfc36\") " pod="openshift-image-registry/node-ca-zg4dm"
Sep 09 02:30:20.270147 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.269789    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/57852f55-dada-41bc-927d-466452a75957-host-cni-netd\") pod \"ovnkube-node-xcqxh\" (UID: \"57852f55-dada-41bc-927d-466452a75957\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcqxh"
Sep 09 02:30:20.270147 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.269819    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a0d4f170-d793-4602-9c77-afdf152cd687-etc-systemd\") pod \"tuned-sv9fq\" (UID: \"a0d4f170-d793-4602-9c77-afdf152cd687\") " pod="openshift-cluster-node-tuning-operator/tuned-sv9fq"
Sep 09 02:30:20.270147 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.269823    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a0d4f170-d793-4602-9c77-afdf152cd687-etc-kubernetes\") pod \"tuned-sv9fq\" (UID: \"a0d4f170-d793-4602-9c77-afdf152cd687\") " pod="openshift-cluster-node-tuning-operator/tuned-sv9fq"
Sep 09 02:30:20.270147 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.269852    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ddccb44f-2b0d-4cf3-9419-31b5211ab2f5-multus-cni-dir\") pod \"multus-phjk6\" (UID: \"ddccb44f-2b0d-4cf3-9419-31b5211ab2f5\") " pod="openshift-multus/multus-phjk6"
Sep 09 02:30:20.270147 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.269865    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a0d4f170-d793-4602-9c77-afdf152cd687-etc-kubernetes\") pod \"tuned-sv9fq\" (UID: \"a0d4f170-d793-4602-9c77-afdf152cd687\") " pod="openshift-cluster-node-tuning-operator/tuned-sv9fq"
Sep 09 02:30:20.270147 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.269892    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/57852f55-dada-41bc-927d-466452a75957-host-cni-netd\") pod \"ovnkube-node-xcqxh\" (UID: \"57852f55-dada-41bc-927d-466452a75957\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcqxh"
Sep 09 02:30:20.270147 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.269930    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ddccb44f-2b0d-4cf3-9419-31b5211ab2f5-host-run-k8s-cni-cncf-io\") pod \"multus-phjk6\" (UID: \"ddccb44f-2b0d-4cf3-9419-31b5211ab2f5\") " pod="openshift-multus/multus-phjk6"
Sep 09 02:30:20.270147 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.269962    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ddccb44f-2b0d-4cf3-9419-31b5211ab2f5-host-var-lib-kubelet\") pod \"multus-phjk6\" (UID: \"ddccb44f-2b0d-4cf3-9419-31b5211ab2f5\") " pod="openshift-multus/multus-phjk6"
Sep 09 02:30:20.270147 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.270004    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d33ba4b9-3e00-405a-bc2f-9144dbfa5c8f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zw227\" (UID: \"d33ba4b9-3e00-405a-bc2f-9144dbfa5c8f\") " pod="openshift-multus/multus-additional-cni-plugins-zw227"
Sep 09 02:30:20.270888 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.270033    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/57852f55-dada-41bc-927d-466452a75957-host-run-ovn-kubernetes\") pod \"ovnkube-node-xcqxh\" (UID: \"57852f55-dada-41bc-927d-466452a75957\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcqxh"
Sep 09 02:30:20.270888 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.270056    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/57852f55-dada-41bc-927d-466452a75957-env-overrides\") pod \"ovnkube-node-xcqxh\" (UID: \"57852f55-dada-41bc-927d-466452a75957\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcqxh"
Sep 09 02:30:20.270888 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.270082    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a0d4f170-d793-4602-9c77-afdf152cd687-etc-sysctl-d\") pod \"tuned-sv9fq\" (UID: \"a0d4f170-d793-4602-9c77-afdf152cd687\") " pod="openshift-cluster-node-tuning-operator/tuned-sv9fq"
Sep 09 02:30:20.270888 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.270104    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvz5v\" (UniqueName: \"kubernetes.io/projected/ddccb44f-2b0d-4cf3-9419-31b5211ab2f5-kube-api-access-tvz5v\") pod \"multus-phjk6\" (UID: \"ddccb44f-2b0d-4cf3-9419-31b5211ab2f5\") " pod="openshift-multus/multus-phjk6"
Sep 09 02:30:20.270888 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.270127    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/90413a10-3c11-4127-b7c8-6e8ac09b1fc9-hosts-file\") pod \"node-resolver-d7t89\" (UID: \"90413a10-3c11-4127-b7c8-6e8ac09b1fc9\") " pod="openshift-dns/node-resolver-d7t89"
Sep 09 02:30:20.270888 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.270133    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/57852f55-dada-41bc-927d-466452a75957-host-run-ovn-kubernetes\") pod \"ovnkube-node-xcqxh\" (UID: \"57852f55-dada-41bc-927d-466452a75957\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcqxh"
Sep 09 02:30:20.270888 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.270157    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/57852f55-dada-41bc-927d-466452a75957-log-socket\") pod \"ovnkube-node-xcqxh\" (UID: \"57852f55-dada-41bc-927d-466452a75957\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcqxh"
Sep 09 02:30:20.270888 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.270202    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/90413a10-3c11-4127-b7c8-6e8ac09b1fc9-hosts-file\") pod \"node-resolver-d7t89\" (UID: \"90413a10-3c11-4127-b7c8-6e8ac09b1fc9\") " pod="openshift-dns/node-resolver-d7t89"
Sep 09 02:30:20.270888 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.270222    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/57852f55-dada-41bc-927d-466452a75957-ovnkube-script-lib\") pod \"ovnkube-node-xcqxh\" (UID: \"57852f55-dada-41bc-927d-466452a75957\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcqxh"
Sep 09 02:30:20.270888 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.270250    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/57852f55-dada-41bc-927d-466452a75957-log-socket\") pod \"ovnkube-node-xcqxh\" (UID: \"57852f55-dada-41bc-927d-466452a75957\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcqxh"
Sep 09 02:30:20.270888 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.270261    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bm8w\" (UniqueName: \"kubernetes.io/projected/57852f55-dada-41bc-927d-466452a75957-kube-api-access-7bm8w\") pod \"ovnkube-node-xcqxh\" (UID: \"57852f55-dada-41bc-927d-466452a75957\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcqxh"
Sep 09 02:30:20.270888 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.270283    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a0d4f170-d793-4602-9c77-afdf152cd687-sys\") pod \"tuned-sv9fq\" (UID: \"a0d4f170-d793-4602-9c77-afdf152cd687\") " pod="openshift-cluster-node-tuning-operator/tuned-sv9fq"
Sep 09 02:30:20.270888 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.270298    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a0d4f170-d793-4602-9c77-afdf152cd687-etc-sysctl-d\") pod \"tuned-sv9fq\" (UID: \"a0d4f170-d793-4602-9c77-afdf152cd687\") " pod="openshift-cluster-node-tuning-operator/tuned-sv9fq"
Sep 09 02:30:20.270888 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.270307    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d33ba4b9-3e00-405a-bc2f-9144dbfa5c8f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zw227\" (UID: \"d33ba4b9-3e00-405a-bc2f-9144dbfa5c8f\") " pod="openshift-multus/multus-additional-cni-plugins-zw227"
Sep 09 02:30:20.270888 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.270361    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a0d4f170-d793-4602-9c77-afdf152cd687-sys\") pod \"tuned-sv9fq\" (UID: \"a0d4f170-d793-4602-9c77-afdf152cd687\") " pod="openshift-cluster-node-tuning-operator/tuned-sv9fq"
Sep 09 02:30:20.270888 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.270359    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/57852f55-dada-41bc-927d-466452a75957-run-openvswitch\") pod \"ovnkube-node-xcqxh\" (UID: \"57852f55-dada-41bc-927d-466452a75957\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcqxh"
Sep 09 02:30:20.270888 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.270427    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/57852f55-dada-41bc-927d-466452a75957-run-openvswitch\") pod \"ovnkube-node-xcqxh\" (UID: \"57852f55-dada-41bc-927d-466452a75957\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcqxh"
Sep 09 02:30:20.271699 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.270527    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/57852f55-dada-41bc-927d-466452a75957-run-ovn\") pod \"ovnkube-node-xcqxh\" (UID: \"57852f55-dada-41bc-927d-466452a75957\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcqxh"
Sep 09 02:30:20.271699 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.270543    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/57852f55-dada-41bc-927d-466452a75957-env-overrides\") pod \"ovnkube-node-xcqxh\" (UID: \"57852f55-dada-41bc-927d-466452a75957\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcqxh"
Sep 09 02:30:20.271699 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.270555    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a0d4f170-d793-4602-9c77-afdf152cd687-lib-modules\") pod \"tuned-sv9fq\" (UID: \"a0d4f170-d793-4602-9c77-afdf152cd687\") " pod="openshift-cluster-node-tuning-operator/tuned-sv9fq"
Sep 09 02:30:20.271699 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.270582    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a0d4f170-d793-4602-9c77-afdf152cd687-tmp\") pod \"tuned-sv9fq\" (UID: \"a0d4f170-d793-4602-9c77-afdf152cd687\") " pod="openshift-cluster-node-tuning-operator/tuned-sv9fq"
Sep 09 02:30:20.271699 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.270620    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ddccb44f-2b0d-4cf3-9419-31b5211ab2f5-multus-daemon-config\") pod \"multus-phjk6\" (UID: \"ddccb44f-2b0d-4cf3-9419-31b5211ab2f5\") " pod="openshift-multus/multus-phjk6"
Sep 09 02:30:20.271699 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.270647    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/37ef684a-ce44-43fc-a803-519ce7d4d225-dbus\") pod \"global-pull-secret-syncer-952h4\" (UID: \"37ef684a-ce44-43fc-a803-519ce7d4d225\") " pod="kube-system/global-pull-secret-syncer-952h4"
Sep 09 02:30:20.271699 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.270653    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/57852f55-dada-41bc-927d-466452a75957-run-ovn\") pod \"ovnkube-node-xcqxh\" (UID: \"57852f55-dada-41bc-927d-466452a75957\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcqxh"
Sep 09 02:30:20.271699 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.270676    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a0d4f170-d793-4602-9c77-afdf152cd687-host\") pod \"tuned-sv9fq\" (UID: \"a0d4f170-d793-4602-9c77-afdf152cd687\") " pod="openshift-cluster-node-tuning-operator/tuned-sv9fq"
Sep 09 02:30:20.271699 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.270721    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a0d4f170-d793-4602-9c77-afdf152cd687-host\") pod \"tuned-sv9fq\" (UID: \"a0d4f170-d793-4602-9c77-afdf152cd687\") " pod="openshift-cluster-node-tuning-operator/tuned-sv9fq"
Sep 09 02:30:20.271699 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.270714    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ddccb44f-2b0d-4cf3-9419-31b5211ab2f5-system-cni-dir\") pod \"multus-phjk6\" (UID: \"ddccb44f-2b0d-4cf3-9419-31b5211ab2f5\") " pod="openshift-multus/multus-phjk6"
Sep 09 02:30:20.271699 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.270762    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/57852f55-dada-41bc-927d-466452a75957-ovnkube-script-lib\") pod \"ovnkube-node-xcqxh\" (UID: \"57852f55-dada-41bc-927d-466452a75957\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcqxh"
Sep 09 02:30:20.271699 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.270765    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ddccb44f-2b0d-4cf3-9419-31b5211ab2f5-host-var-lib-cni-bin\") pod \"multus-phjk6\" (UID: \"ddccb44f-2b0d-4cf3-9419-31b5211ab2f5\") " pod="openshift-multus/multus-phjk6"
Sep 09 02:30:20.271699 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.270808    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a0d4f170-d793-4602-9c77-afdf152cd687-lib-modules\") pod \"tuned-sv9fq\" (UID: \"a0d4f170-d793-4602-9c77-afdf152cd687\") " pod="openshift-cluster-node-tuning-operator/tuned-sv9fq"
Sep 09 02:30:20.271699 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.270811    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ddccb44f-2b0d-4cf3-9419-31b5211ab2f5-host-run-multus-certs\") pod \"multus-phjk6\" (UID: \"ddccb44f-2b0d-4cf3-9419-31b5211ab2f5\") " pod="openshift-multus/multus-phjk6"
Sep 09 02:30:20.271699 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.270879    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/679d664f-a0b2-4ad9-94e4-014aef8f2e16-konnectivity-ca\") pod \"konnectivity-agent-kxvb7\" (UID: \"679d664f-a0b2-4ad9-94e4-014aef8f2e16\") " pod="kube-system/konnectivity-agent-kxvb7"
Sep 09 02:30:20.271699 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.270900    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcc4f\" (UniqueName: \"kubernetes.io/projected/67afff59-c0d9-43fb-9b1f-581ceab88dbd-kube-api-access-rcc4f\") pod \"iptables-alerter-h2sdl\" (UID: \"67afff59-c0d9-43fb-9b1f-581ceab88dbd\") " pod="openshift-network-operator/iptables-alerter-h2sdl"
Sep 09 02:30:20.271699 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.270933    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c3e9e8ee-29c7-4fe7-8bcd-7f2363992b3c-device-dir\") pod \"aws-ebs-csi-driver-node-2f82x\" (UID: \"c3e9e8ee-29c7-4fe7-8bcd-7f2363992b3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2f82x"
Sep 09 02:30:20.272494 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.270966    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90413a10-3c11-4127-b7c8-6e8ac09b1fc9-tmp-dir\") pod \"node-resolver-d7t89\" (UID: \"90413a10-3c11-4127-b7c8-6e8ac09b1fc9\") " pod="openshift-dns/node-resolver-d7t89"
Sep 09 02:30:20.272494 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.271070    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/57852f55-dada-41bc-927d-466452a75957-ovn-node-metrics-cert\") pod \"ovnkube-node-xcqxh\" (UID: \"57852f55-dada-41bc-927d-466452a75957\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcqxh"
Sep 09 02:30:20.272494 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.271100    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d33ba4b9-3e00-405a-bc2f-9144dbfa5c8f-system-cni-dir\") pod \"multus-additional-cni-plugins-zw227\" (UID: \"d33ba4b9-3e00-405a-bc2f-9144dbfa5c8f\") " pod="openshift-multus/multus-additional-cni-plugins-zw227"
Sep 09 02:30:20.272494 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.271144    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/57852f55-dada-41bc-927d-466452a75957-var-lib-openvswitch\") pod \"ovnkube-node-xcqxh\" (UID: \"57852f55-dada-41bc-927d-466452a75957\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcqxh"
Sep 09 02:30:20.272494 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.271159    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a0d4f170-d793-4602-9c77-afdf152cd687-etc-sysctl-conf\") pod \"tuned-sv9fq\" (UID: \"a0d4f170-d793-4602-9c77-afdf152cd687\") " pod="openshift-cluster-node-tuning-operator/tuned-sv9fq"
Sep 09 02:30:20.272494 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.271189    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ddccb44f-2b0d-4cf3-9419-31b5211ab2f5-multus-socket-dir-parent\") pod \"multus-phjk6\" (UID: \"ddccb44f-2b0d-4cf3-9419-31b5211ab2f5\") " pod="openshift-multus/multus-phjk6"
Sep 09 02:30:20.272494 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.271226    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ddccb44f-2b0d-4cf3-9419-31b5211ab2f5-host-run-netns\") pod \"multus-phjk6\" (UID: \"ddccb44f-2b0d-4cf3-9419-31b5211ab2f5\") " pod="openshift-multus/multus-phjk6"
Sep 09 02:30:20.272494 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.271269    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d33ba4b9-3e00-405a-bc2f-9144dbfa5c8f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zw227\" (UID: \"d33ba4b9-3e00-405a-bc2f-9144dbfa5c8f\") " pod="openshift-multus/multus-additional-cni-plugins-zw227"
Sep 09 02:30:20.272494 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.271298    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c3e9e8ee-29c7-4fe7-8bcd-7f2363992b3c-registration-dir\") pod \"aws-ebs-csi-driver-node-2f82x\" (UID: \"c3e9e8ee-29c7-4fe7-8bcd-7f2363992b3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2f82x"
Sep 09 02:30:20.272494 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.271318    2570 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 09 02:30:20.272494 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.271330    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c3e9e8ee-29c7-4fe7-8bcd-7f2363992b3c-sys-fs\") pod \"aws-ebs-csi-driver-node-2f82x\" (UID: \"c3e9e8ee-29c7-4fe7-8bcd-7f2363992b3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2f82x"
Sep 09 02:30:20.272494 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.271347    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ddccb44f-2b0d-4cf3-9419-31b5211ab2f5-cnibin\") pod \"multus-phjk6\" (UID: \"ddccb44f-2b0d-4cf3-9419-31b5211ab2f5\") " pod="openshift-multus/multus-phjk6"
Sep 09 02:30:20.272494 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.271380    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/57852f55-dada-41bc-927d-466452a75957-etc-openvswitch\") pod \"ovnkube-node-xcqxh\" (UID: \"57852f55-dada-41bc-927d-466452a75957\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcqxh"
Sep 09 02:30:20.272494 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.271399    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/57852f55-dada-41bc-927d-466452a75957-node-log\") pod \"ovnkube-node-xcqxh\" (UID: \"57852f55-dada-41bc-927d-466452a75957\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcqxh"
Sep 09 02:30:20.272494 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.271418    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/57852f55-dada-41bc-927d-466452a75957-ovnkube-config\") pod \"ovnkube-node-xcqxh\" (UID: \"57852f55-dada-41bc-927d-466452a75957\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcqxh"
Sep 09 02:30:20.272494 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.271432    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/679d664f-a0b2-4ad9-94e4-014aef8f2e16-agent-certs\") pod \"konnectivity-agent-kxvb7\" (UID: \"679d664f-a0b2-4ad9-94e4-014aef8f2e16\") " pod="kube-system/konnectivity-agent-kxvb7"
Sep 09 02:30:20.272494 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.271452    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a0d4f170-d793-4602-9c77-afdf152cd687-etc-sysconfig\") pod \"tuned-sv9fq\" (UID: \"a0d4f170-d793-4602-9c77-afdf152cd687\") " pod="openshift-cluster-node-tuning-operator/tuned-sv9fq"
Sep 09 02:30:20.273359 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.271491    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ddccb44f-2b0d-4cf3-9419-31b5211ab2f5-multus-conf-dir\") pod \"multus-phjk6\" (UID: \"ddccb44f-2b0d-4cf3-9419-31b5211ab2f5\") " pod="openshift-multus/multus-phjk6"
Sep 09 02:30:20.273359 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.271506    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/37ef684a-ce44-43fc-a803-519ce7d4d225-kubelet-config\") pod \"global-pull-secret-syncer-952h4\" (UID: \"37ef684a-ce44-43fc-a803-519ce7d4d225\") " pod="kube-system/global-pull-secret-syncer-952h4"
Sep 09 02:30:20.273359 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.271531    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c3e9e8ee-29c7-4fe7-8bcd-7f2363992b3c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2f82x\" (UID: \"c3e9e8ee-29c7-4fe7-8bcd-7f2363992b3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2f82x"
Sep 09 02:30:20.273359 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.271549    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/57852f55-dada-41bc-927d-466452a75957-host-cni-bin\") pod \"ovnkube-node-xcqxh\" (UID: \"57852f55-dada-41bc-927d-466452a75957\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcqxh"
Sep 09 02:30:20.273359 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.271573    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vnp9\" (UniqueName: \"kubernetes.io/projected/d786708a-bbb1-4250-a2d3-84dc2299d71a-kube-api-access-5vnp9\") pod \"network-metrics-daemon-zrljs\" (UID: \"d786708a-bbb1-4250-a2d3-84dc2299d71a\") " pod="openshift-multus/network-metrics-daemon-zrljs"
Sep 09 02:30:20.273359 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.271590    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a0d4f170-d793-4602-9c77-afdf152cd687-etc-tuned\") pod \"tuned-sv9fq\" (UID: \"a0d4f170-d793-4602-9c77-afdf152cd687\") " pod="openshift-cluster-node-tuning-operator/tuned-sv9fq"
Sep 09 02:30:20.273359 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.271666    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/57852f55-dada-41bc-927d-466452a75957-etc-openvswitch\") pod \"ovnkube-node-xcqxh\" (UID: \"57852f55-dada-41bc-927d-466452a75957\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcqxh"
Sep 09 02:30:20.273359 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.271304    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90413a10-3c11-4127-b7c8-6e8ac09b1fc9-tmp-dir\") pod \"node-resolver-d7t89\" (UID: \"90413a10-3c11-4127-b7c8-6e8ac09b1fc9\") " pod="openshift-dns/node-resolver-d7t89"
Sep 09 02:30:20.273359 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.271771    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a0d4f170-d793-4602-9c77-afdf152cd687-etc-sysctl-conf\") pod \"tuned-sv9fq\" (UID: \"a0d4f170-d793-4602-9c77-afdf152cd687\") " pod="openshift-cluster-node-tuning-operator/tuned-sv9fq"
Sep 09 02:30:20.273359 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.271824    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/57852f55-dada-41bc-927d-466452a75957-node-log\") pod \"ovnkube-node-xcqxh\" (UID: \"57852f55-dada-41bc-927d-466452a75957\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcqxh"
Sep 09 02:30:20.273359 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.271956    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/57852f55-dada-41bc-927d-466452a75957-host-cni-bin\") pod \"ovnkube-node-xcqxh\" (UID: \"57852f55-dada-41bc-927d-466452a75957\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcqxh"
Sep 09 02:30:20.273359 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.272331    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/57852f55-dada-41bc-927d-466452a75957-ovnkube-config\") pod \"ovnkube-node-xcqxh\" (UID: \"57852f55-dada-41bc-927d-466452a75957\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcqxh"
Sep 09 02:30:20.273359 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.272415    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a0d4f170-d793-4602-9c77-afdf152cd687-etc-sysconfig\") pod \"tuned-sv9fq\" (UID: \"a0d4f170-d793-4602-9c77-afdf152cd687\") " pod="openshift-cluster-node-tuning-operator/tuned-sv9fq"
Sep 09 02:30:20.273359 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.272656    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/57852f55-dada-41bc-927d-466452a75957-var-lib-openvswitch\") pod \"ovnkube-node-xcqxh\" (UID: \"57852f55-dada-41bc-927d-466452a75957\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcqxh"
Sep 09 02:30:20.275566 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.275544    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a0d4f170-d793-4602-9c77-afdf152cd687-etc-tuned\") pod \"tuned-sv9fq\" (UID: \"a0d4f170-d793-4602-9c77-afdf152cd687\") " pod="openshift-cluster-node-tuning-operator/tuned-sv9fq"
Sep 09 02:30:20.275690 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.275671    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/57852f55-dada-41bc-927d-466452a75957-ovn-node-metrics-cert\") pod \"ovnkube-node-xcqxh\" (UID: \"57852f55-dada-41bc-927d-466452a75957\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcqxh"
Sep 09 02:30:20.275787 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.275754    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/679d664f-a0b2-4ad9-94e4-014aef8f2e16-agent-certs\") pod \"konnectivity-agent-kxvb7\" (UID: \"679d664f-a0b2-4ad9-94e4-014aef8f2e16\") " pod="kube-system/konnectivity-agent-kxvb7"
Sep 09 02:30:20.276062 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.276042    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a0d4f170-d793-4602-9c77-afdf152cd687-tmp\") pod \"tuned-sv9fq\" (UID: \"a0d4f170-d793-4602-9c77-afdf152cd687\") " pod="openshift-cluster-node-tuning-operator/tuned-sv9fq"
Sep 09 02:30:20.280921 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:20.280900    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 02:30:20.281024 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:20.280934    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 02:30:20.281024 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:20.280950    2570 projected.go:196] Error preparing data for projected volume kube-api-access-drrcs for pod openshift-network-diagnostics/network-check-target-6gkgn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 02:30:20.281141 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:20.281036    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ac0a0b68-d43e-4ca8-8c4d-39bbe5e7519c-kube-api-access-drrcs podName:ac0a0b68-d43e-4ca8-8c4d-39bbe5e7519c nodeName:}" failed. No retries permitted until 2026-09-09 02:30:20.781018596 +0000 UTC m=+3.035322223 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-drrcs" (UniqueName: "kubernetes.io/projected/ac0a0b68-d43e-4ca8-8c4d-39bbe5e7519c-kube-api-access-drrcs") pod "network-check-target-6gkgn" (UID: "ac0a0b68-d43e-4ca8-8c4d-39bbe5e7519c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 02:30:20.284195 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.284169    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bm8w\" (UniqueName: \"kubernetes.io/projected/57852f55-dada-41bc-927d-466452a75957-kube-api-access-7bm8w\") pod \"ovnkube-node-xcqxh\" (UID: \"57852f55-dada-41bc-927d-466452a75957\") " pod="openshift-ovn-kubernetes/ovnkube-node-xcqxh"
Sep 09 02:30:20.284670 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.284650    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzrbq\" (UniqueName: \"kubernetes.io/projected/a0d4f170-d793-4602-9c77-afdf152cd687-kube-api-access-rzrbq\") pod \"tuned-sv9fq\" (UID: \"a0d4f170-d793-4602-9c77-afdf152cd687\") " pod="openshift-cluster-node-tuning-operator/tuned-sv9fq"
Sep 09 02:30:20.284670 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.284650    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6ddt\" (UniqueName: \"kubernetes.io/projected/90413a10-3c11-4127-b7c8-6e8ac09b1fc9-kube-api-access-r6ddt\") pod \"node-resolver-d7t89\" (UID: \"90413a10-3c11-4127-b7c8-6e8ac09b1fc9\") " pod="openshift-dns/node-resolver-d7t89"
Sep 09 02:30:20.285564 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.285531    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vnp9\" (UniqueName: \"kubernetes.io/projected/d786708a-bbb1-4250-a2d3-84dc2299d71a-kube-api-access-5vnp9\") pod \"network-metrics-daemon-zrljs\" (UID: \"d786708a-bbb1-4250-a2d3-84dc2299d71a\") " pod="openshift-multus/network-metrics-daemon-zrljs"
Sep 09 02:30:20.371899 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.371863    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ddccb44f-2b0d-4cf3-9419-31b5211ab2f5-etc-kubernetes\") pod \"multus-phjk6\" (UID: \"ddccb44f-2b0d-4cf3-9419-31b5211ab2f5\") " pod="openshift-multus/multus-phjk6"
Sep 09 02:30:20.371899 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.371903    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/67afff59-c0d9-43fb-9b1f-581ceab88dbd-iptables-alerter-script\") pod \"iptables-alerter-h2sdl\" (UID: \"67afff59-c0d9-43fb-9b1f-581ceab88dbd\") " pod="openshift-network-operator/iptables-alerter-h2sdl"
Sep 09 02:30:20.372180 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.371995    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ddccb44f-2b0d-4cf3-9419-31b5211ab2f5-etc-kubernetes\") pod \"multus-phjk6\" (UID: \"ddccb44f-2b0d-4cf3-9419-31b5211ab2f5\") " pod="openshift-multus/multus-phjk6"
Sep 09 02:30:20.372180 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.372030    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d33ba4b9-3e00-405a-bc2f-9144dbfa5c8f-cnibin\") pod \"multus-additional-cni-plugins-zw227\" (UID: \"d33ba4b9-3e00-405a-bc2f-9144dbfa5c8f\") " pod="openshift-multus/multus-additional-cni-plugins-zw227"
Sep 09 02:30:20.372180 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.372117    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d33ba4b9-3e00-405a-bc2f-9144dbfa5c8f-cnibin\") pod \"multus-additional-cni-plugins-zw227\" (UID: \"d33ba4b9-3e00-405a-bc2f-9144dbfa5c8f\") " pod="openshift-multus/multus-additional-cni-plugins-zw227"
Sep 09 02:30:20.372180 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.372130    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ddccb44f-2b0d-4cf3-9419-31b5211ab2f5-host-var-lib-cni-multus\") pod \"multus-phjk6\" (UID: \"ddccb44f-2b0d-4cf3-9419-31b5211ab2f5\") " pod="openshift-multus/multus-phjk6"
Sep 09 02:30:20.372180 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.372172    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/67afff59-c0d9-43fb-9b1f-581ceab88dbd-host-slash\") pod \"iptables-alerter-h2sdl\" (UID: \"67afff59-c0d9-43fb-9b1f-581ceab88dbd\") " pod="openshift-network-operator/iptables-alerter-h2sdl"
Sep 09 02:30:20.372408 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.372202    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c3e9e8ee-29c7-4fe7-8bcd-7f2363992b3c-socket-dir\") pod \"aws-ebs-csi-driver-node-2f82x\" (UID: \"c3e9e8ee-29c7-4fe7-8bcd-7f2363992b3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2f82x"
Sep 09 02:30:20.372408 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.372229    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c3e9e8ee-29c7-4fe7-8bcd-7f2363992b3c-etc-selinux\") pod \"aws-ebs-csi-driver-node-2f82x\" (UID: \"c3e9e8ee-29c7-4fe7-8bcd-7f2363992b3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2f82x"
Sep 09 02:30:20.372408 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.372230    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/67afff59-c0d9-43fb-9b1f-581ceab88dbd-host-slash\") pod \"iptables-alerter-h2sdl\" (UID: \"67afff59-c0d9-43fb-9b1f-581ceab88dbd\") " pod="openshift-network-operator/iptables-alerter-h2sdl"
Sep 09 02:30:20.372408 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.372174    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ddccb44f-2b0d-4cf3-9419-31b5211ab2f5-host-var-lib-cni-multus\") pod \"multus-phjk6\" (UID: \"ddccb44f-2b0d-4cf3-9419-31b5211ab2f5\") " pod="openshift-multus/multus-phjk6"
Sep 09 02:30:20.372408 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.372268    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/721fa3c1-a7d2-4245-a7bd-76eb7facfc36-serviceca\") pod \"node-ca-zg4dm\" (UID: \"721fa3c1-a7d2-4245-a7bd-76eb7facfc36\") " pod="openshift-image-registry/node-ca-zg4dm"
Sep 09 02:30:20.372408 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.372294    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ddccb44f-2b0d-4cf3-9419-31b5211ab2f5-multus-cni-dir\") pod \"multus-phjk6\" (UID: \"ddccb44f-2b0d-4cf3-9419-31b5211ab2f5\") " pod="openshift-multus/multus-phjk6"
Sep 09 02:30:20.372408 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.372318    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ddccb44f-2b0d-4cf3-9419-31b5211ab2f5-host-run-k8s-cni-cncf-io\") pod \"multus-phjk6\" (UID: \"ddccb44f-2b0d-4cf3-9419-31b5211ab2f5\") " pod="openshift-multus/multus-phjk6"
Sep 09 02:30:20.372408 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.372341    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ddccb44f-2b0d-4cf3-9419-31b5211ab2f5-host-var-lib-kubelet\") pod \"multus-phjk6\" (UID: \"ddccb44f-2b0d-4cf3-9419-31b5211ab2f5\") " pod="openshift-multus/multus-phjk6"
Sep 09 02:30:20.372408 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.372354    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c3e9e8ee-29c7-4fe7-8bcd-7f2363992b3c-etc-selinux\") pod \"aws-ebs-csi-driver-node-2f82x\" (UID: \"c3e9e8ee-29c7-4fe7-8bcd-7f2363992b3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2f82x"
Sep 09 02:30:20.372408 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.372366    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d33ba4b9-3e00-405a-bc2f-9144dbfa5c8f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zw227\" (UID: \"d33ba4b9-3e00-405a-bc2f-9144dbfa5c8f\") " pod="openshift-multus/multus-additional-cni-plugins-zw227"
Sep 09 02:30:20.372408 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.372386    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c3e9e8ee-29c7-4fe7-8bcd-7f2363992b3c-socket-dir\") pod \"aws-ebs-csi-driver-node-2f82x\" (UID: \"c3e9e8ee-29c7-4fe7-8bcd-7f2363992b3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2f82x"
Sep 09 02:30:20.372408 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.372399    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvz5v\" (UniqueName: \"kubernetes.io/projected/ddccb44f-2b0d-4cf3-9419-31b5211ab2f5-kube-api-access-tvz5v\") pod \"multus-phjk6\" (UID: \"ddccb44f-2b0d-4cf3-9419-31b5211ab2f5\") " pod="openshift-multus/multus-phjk6"
Sep 09 02:30:20.373004 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.372424    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d33ba4b9-3e00-405a-bc2f-9144dbfa5c8f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zw227\" (UID: \"d33ba4b9-3e00-405a-bc2f-9144dbfa5c8f\") " pod="openshift-multus/multus-additional-cni-plugins-zw227"
Sep 09 02:30:20.373004 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.372438    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ddccb44f-2b0d-4cf3-9419-31b5211ab2f5-host-var-lib-kubelet\") pod \"multus-phjk6\" (UID: \"ddccb44f-2b0d-4cf3-9419-31b5211ab2f5\") " pod="openshift-multus/multus-phjk6"
Sep 09 02:30:20.373004 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.372441    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ddccb44f-2b0d-4cf3-9419-31b5211ab2f5-multus-daemon-config\") pod \"multus-phjk6\" (UID: \"ddccb44f-2b0d-4cf3-9419-31b5211ab2f5\") " pod="openshift-multus/multus-phjk6"
Sep 09 02:30:20.373004 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.372486    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ddccb44f-2b0d-4cf3-9419-31b5211ab2f5-multus-cni-dir\") pod \"multus-phjk6\" (UID: \"ddccb44f-2b0d-4cf3-9419-31b5211ab2f5\") " pod="openshift-multus/multus-phjk6"
Sep 09 02:30:20.373004 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.372505    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/67afff59-c0d9-43fb-9b1f-581ceab88dbd-iptables-alerter-script\") pod \"iptables-alerter-h2sdl\" (UID: \"67afff59-c0d9-43fb-9b1f-581ceab88dbd\") " pod="openshift-network-operator/iptables-alerter-h2sdl"
Sep 09 02:30:20.373004 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.372529    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ddccb44f-2b0d-4cf3-9419-31b5211ab2f5-host-run-k8s-cni-cncf-io\") pod \"multus-phjk6\" (UID: \"ddccb44f-2b0d-4cf3-9419-31b5211ab2f5\") " pod="openshift-multus/multus-phjk6"
Sep 09 02:30:20.373004 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.372489    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/37ef684a-ce44-43fc-a803-519ce7d4d225-dbus\") pod \"global-pull-secret-syncer-952h4\" (UID: \"37ef684a-ce44-43fc-a803-519ce7d4d225\") " pod="kube-system/global-pull-secret-syncer-952h4"
Sep 09 02:30:20.373004 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.372717    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d33ba4b9-3e00-405a-bc2f-9144dbfa5c8f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zw227\" (UID: \"d33ba4b9-3e00-405a-bc2f-9144dbfa5c8f\") " pod="openshift-multus/multus-additional-cni-plugins-zw227"
Sep 09 02:30:20.373004 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.372782    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/37ef684a-ce44-43fc-a803-519ce7d4d225-dbus\") pod \"global-pull-secret-syncer-952h4\" (UID: \"37ef684a-ce44-43fc-a803-519ce7d4d225\") " pod="kube-system/global-pull-secret-syncer-952h4"
Sep 09 02:30:20.373004 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.372818    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ddccb44f-2b0d-4cf3-9419-31b5211ab2f5-system-cni-dir\") pod \"multus-phjk6\" (UID: \"ddccb44f-2b0d-4cf3-9419-31b5211ab2f5\") " pod="openshift-multus/multus-phjk6"
Sep 09 02:30:20.373004 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.372821    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/721fa3c1-a7d2-4245-a7bd-76eb7facfc36-serviceca\") pod \"node-ca-zg4dm\" (UID: \"721fa3c1-a7d2-4245-a7bd-76eb7facfc36\") " pod="openshift-image-registry/node-ca-zg4dm"
Sep 09 02:30:20.373004 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.372849    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ddccb44f-2b0d-4cf3-9419-31b5211ab2f5-host-var-lib-cni-bin\") pod \"multus-phjk6\" (UID: \"ddccb44f-2b0d-4cf3-9419-31b5211ab2f5\") " pod="openshift-multus/multus-phjk6"
Sep 09 02:30:20.373004 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.372876    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ddccb44f-2b0d-4cf3-9419-31b5211ab2f5-host-run-multus-certs\") pod \"multus-phjk6\" (UID: \"ddccb44f-2b0d-4cf3-9419-31b5211ab2f5\") " pod="openshift-multus/multus-phjk6"
Sep 09 02:30:20.373004 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.372897    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ddccb44f-2b0d-4cf3-9419-31b5211ab2f5-system-cni-dir\") pod \"multus-phjk6\" (UID: \"ddccb44f-2b0d-4cf3-9419-31b5211ab2f5\") " pod="openshift-multus/multus-phjk6"
Sep 09 02:30:20.373004 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.372900    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcc4f\" (UniqueName: \"kubernetes.io/projected/67afff59-c0d9-43fb-9b1f-581ceab88dbd-kube-api-access-rcc4f\") pod \"iptables-alerter-h2sdl\" (UID: \"67afff59-c0d9-43fb-9b1f-581ceab88dbd\") " pod="openshift-network-operator/iptables-alerter-h2sdl"
Sep 09 02:30:20.373004 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.372923    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c3e9e8ee-29c7-4fe7-8bcd-7f2363992b3c-device-dir\") pod \"aws-ebs-csi-driver-node-2f82x\" (UID: \"c3e9e8ee-29c7-4fe7-8bcd-7f2363992b3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2f82x"
Sep 09 02:30:20.373004 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.372926    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ddccb44f-2b0d-4cf3-9419-31b5211ab2f5-host-var-lib-cni-bin\") pod \"multus-phjk6\" (UID: \"ddccb44f-2b0d-4cf3-9419-31b5211ab2f5\") " pod="openshift-multus/multus-phjk6"
Sep 09 02:30:20.373777 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.372950    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d33ba4b9-3e00-405a-bc2f-9144dbfa5c8f-system-cni-dir\") pod \"multus-additional-cni-plugins-zw227\" (UID: \"d33ba4b9-3e00-405a-bc2f-9144dbfa5c8f\") " pod="openshift-multus/multus-additional-cni-plugins-zw227"
Sep 09 02:30:20.373777 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.372992    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ddccb44f-2b0d-4cf3-9419-31b5211ab2f5-multus-socket-dir-parent\") pod \"multus-phjk6\" (UID: \"ddccb44f-2b0d-4cf3-9419-31b5211ab2f5\") " pod="openshift-multus/multus-phjk6"
Sep 09 02:30:20.373777 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.372997    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d33ba4b9-3e00-405a-bc2f-9144dbfa5c8f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zw227\" (UID: \"d33ba4b9-3e00-405a-bc2f-9144dbfa5c8f\") " pod="openshift-multus/multus-additional-cni-plugins-zw227"
Sep 09 02:30:20.373777 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.373005    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c3e9e8ee-29c7-4fe7-8bcd-7f2363992b3c-device-dir\") pod \"aws-ebs-csi-driver-node-2f82x\" (UID: \"c3e9e8ee-29c7-4fe7-8bcd-7f2363992b3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2f82x"
Sep 09 02:30:20.373777 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.373017    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ddccb44f-2b0d-4cf3-9419-31b5211ab2f5-host-run-netns\") pod \"multus-phjk6\" (UID: \"ddccb44f-2b0d-4cf3-9419-31b5211ab2f5\") " pod="openshift-multus/multus-phjk6"
Sep 09 02:30:20.373777 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.372927    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ddccb44f-2b0d-4cf3-9419-31b5211ab2f5-host-run-multus-certs\") pod \"multus-phjk6\" (UID: \"ddccb44f-2b0d-4cf3-9419-31b5211ab2f5\") " pod="openshift-multus/multus-phjk6"
Sep 09 02:30:20.373777 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.373018    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ddccb44f-2b0d-4cf3-9419-31b5211ab2f5-multus-daemon-config\") pod \"multus-phjk6\" (UID: \"ddccb44f-2b0d-4cf3-9419-31b5211ab2f5\") " pod="openshift-multus/multus-phjk6"
Sep 09 02:30:20.373777 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.373067    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ddccb44f-2b0d-4cf3-9419-31b5211ab2f5-host-run-netns\") pod \"multus-phjk6\" (UID: \"ddccb44f-2b0d-4cf3-9419-31b5211ab2f5\") " pod="openshift-multus/multus-phjk6"
Sep 09 02:30:20.373777 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.373067    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ddccb44f-2b0d-4cf3-9419-31b5211ab2f5-multus-socket-dir-parent\") pod \"multus-phjk6\" (UID: \"ddccb44f-2b0d-4cf3-9419-31b5211ab2f5\") " pod="openshift-multus/multus-phjk6"
Sep 09 02:30:20.373777 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.373087    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d33ba4b9-3e00-405a-bc2f-9144dbfa5c8f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zw227\" (UID: \"d33ba4b9-3e00-405a-bc2f-9144dbfa5c8f\") " pod="openshift-multus/multus-additional-cni-plugins-zw227"
Sep 09 02:30:20.373777 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.373114    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d33ba4b9-3e00-405a-bc2f-9144dbfa5c8f-system-cni-dir\") pod \"multus-additional-cni-plugins-zw227\" (UID: \"d33ba4b9-3e00-405a-bc2f-9144dbfa5c8f\") " pod="openshift-multus/multus-additional-cni-plugins-zw227"
Sep 09 02:30:20.373777 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.373118    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c3e9e8ee-29c7-4fe7-8bcd-7f2363992b3c-registration-dir\") pod \"aws-ebs-csi-driver-node-2f82x\" (UID: \"c3e9e8ee-29c7-4fe7-8bcd-7f2363992b3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2f82x"
Sep 09 02:30:20.373777 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.373145    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c3e9e8ee-29c7-4fe7-8bcd-7f2363992b3c-sys-fs\") pod \"aws-ebs-csi-driver-node-2f82x\" (UID: \"c3e9e8ee-29c7-4fe7-8bcd-7f2363992b3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2f82x"
Sep 09 02:30:20.373777 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.373165    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ddccb44f-2b0d-4cf3-9419-31b5211ab2f5-cnibin\") pod \"multus-phjk6\" (UID: \"ddccb44f-2b0d-4cf3-9419-31b5211ab2f5\") " pod="openshift-multus/multus-phjk6"
Sep 09 02:30:20.373777 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.373190    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ddccb44f-2b0d-4cf3-9419-31b5211ab2f5-multus-conf-dir\") pod \"multus-phjk6\" (UID: \"ddccb44f-2b0d-4cf3-9419-31b5211ab2f5\") " pod="openshift-multus/multus-phjk6"
Sep 09 02:30:20.373777 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.373191    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c3e9e8ee-29c7-4fe7-8bcd-7f2363992b3c-registration-dir\") pod \"aws-ebs-csi-driver-node-2f82x\" (UID: \"c3e9e8ee-29c7-4fe7-8bcd-7f2363992b3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2f82x"
Sep 09 02:30:20.373777 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.373214    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/37ef684a-ce44-43fc-a803-519ce7d4d225-kubelet-config\") pod \"global-pull-secret-syncer-952h4\" (UID: \"37ef684a-ce44-43fc-a803-519ce7d4d225\") " pod="kube-system/global-pull-secret-syncer-952h4"
Sep 09 02:30:20.374480 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.373224    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c3e9e8ee-29c7-4fe7-8bcd-7f2363992b3c-sys-fs\") pod \"aws-ebs-csi-driver-node-2f82x\" (UID: \"c3e9e8ee-29c7-4fe7-8bcd-7f2363992b3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2f82x"
Sep 09 02:30:20.374480 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.373238    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c3e9e8ee-29c7-4fe7-8bcd-7f2363992b3c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2f82x\" (UID: \"c3e9e8ee-29c7-4fe7-8bcd-7f2363992b3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2f82x"
Sep 09 02:30:20.374480 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.373248    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ddccb44f-2b0d-4cf3-9419-31b5211ab2f5-cnibin\") pod \"multus-phjk6\" (UID: \"ddccb44f-2b0d-4cf3-9419-31b5211ab2f5\") " pod="openshift-multus/multus-phjk6"
Sep 09 02:30:20.374480 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.373330    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ddccb44f-2b0d-4cf3-9419-31b5211ab2f5-multus-conf-dir\") pod \"multus-phjk6\" (UID: \"ddccb44f-2b0d-4cf3-9419-31b5211ab2f5\") " pod="openshift-multus/multus-phjk6"
Sep 09 02:30:20.374480 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.373351    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/37ef684a-ce44-43fc-a803-519ce7d4d225-original-pull-secret\") pod \"global-pull-secret-syncer-952h4\" (UID: \"37ef684a-ce44-43fc-a803-519ce7d4d225\") " pod="kube-system/global-pull-secret-syncer-952h4"
Sep 09 02:30:20.374480 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.373367    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c3e9e8ee-29c7-4fe7-8bcd-7f2363992b3c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2f82x\" (UID: \"c3e9e8ee-29c7-4fe7-8bcd-7f2363992b3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2f82x"
Sep 09 02:30:20.374480 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.373372    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/37ef684a-ce44-43fc-a803-519ce7d4d225-kubelet-config\") pod \"global-pull-secret-syncer-952h4\" (UID: \"37ef684a-ce44-43fc-a803-519ce7d4d225\") " pod="kube-system/global-pull-secret-syncer-952h4"
Sep 09 02:30:20.374480 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.373380    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6w59\" (UniqueName: \"kubernetes.io/projected/721fa3c1-a7d2-4245-a7bd-76eb7facfc36-kube-api-access-k6w59\") pod \"node-ca-zg4dm\" (UID: \"721fa3c1-a7d2-4245-a7bd-76eb7facfc36\") " pod="openshift-image-registry/node-ca-zg4dm"
Sep 09 02:30:20.374480 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.373424    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2m477\" (UniqueName: \"kubernetes.io/projected/d33ba4b9-3e00-405a-bc2f-9144dbfa5c8f-kube-api-access-2m477\") pod \"multus-additional-cni-plugins-zw227\" (UID: \"d33ba4b9-3e00-405a-bc2f-9144dbfa5c8f\") " pod="openshift-multus/multus-additional-cni-plugins-zw227"
Sep 09 02:30:20.374480 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.373447    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/721fa3c1-a7d2-4245-a7bd-76eb7facfc36-host\") pod \"node-ca-zg4dm\" (UID: \"721fa3c1-a7d2-4245-a7bd-76eb7facfc36\") " pod="openshift-image-registry/node-ca-zg4dm"
Sep 09 02:30:20.374480 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:20.373453    2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 02:30:20.374480 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.373483    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ddccb44f-2b0d-4cf3-9419-31b5211ab2f5-cni-binary-copy\") pod \"multus-phjk6\" (UID: \"ddccb44f-2b0d-4cf3-9419-31b5211ab2f5\") " pod="openshift-multus/multus-phjk6"
Sep 09 02:30:20.374480 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.373507    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d33ba4b9-3e00-405a-bc2f-9144dbfa5c8f-cni-binary-copy\") pod \"multus-additional-cni-plugins-zw227\" (UID: \"d33ba4b9-3e00-405a-bc2f-9144dbfa5c8f\") " pod="openshift-multus/multus-additional-cni-plugins-zw227"
Sep 09 02:30:20.374480 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:20.373526    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37ef684a-ce44-43fc-a803-519ce7d4d225-original-pull-secret podName:37ef684a-ce44-43fc-a803-519ce7d4d225 nodeName:}" failed. No retries permitted until 2026-09-09 02:30:20.873511591 +0000 UTC m=+3.127815205 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/37ef684a-ce44-43fc-a803-519ce7d4d225-original-pull-secret") pod "global-pull-secret-syncer-952h4" (UID: "37ef684a-ce44-43fc-a803-519ce7d4d225") : object "kube-system"/"original-pull-secret" not registered
Sep 09 02:30:20.374480 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.373557    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mhkj\" (UniqueName: \"kubernetes.io/projected/c3e9e8ee-29c7-4fe7-8bcd-7f2363992b3c-kube-api-access-2mhkj\") pod \"aws-ebs-csi-driver-node-2f82x\" (UID: \"c3e9e8ee-29c7-4fe7-8bcd-7f2363992b3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2f82x"
Sep 09 02:30:20.374480 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.373570    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d33ba4b9-3e00-405a-bc2f-9144dbfa5c8f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zw227\" (UID: \"d33ba4b9-3e00-405a-bc2f-9144dbfa5c8f\") " pod="openshift-multus/multus-additional-cni-plugins-zw227"
Sep 09 02:30:20.374480 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.373585    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ddccb44f-2b0d-4cf3-9419-31b5211ab2f5-hostroot\") pod \"multus-phjk6\" (UID: \"ddccb44f-2b0d-4cf3-9419-31b5211ab2f5\") " pod="openshift-multus/multus-phjk6"
Sep 09 02:30:20.375280 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.373643    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/721fa3c1-a7d2-4245-a7bd-76eb7facfc36-host\") pod \"node-ca-zg4dm\" (UID: \"721fa3c1-a7d2-4245-a7bd-76eb7facfc36\") " pod="openshift-image-registry/node-ca-zg4dm"
Sep 09 02:30:20.375280 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.373647    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ddccb44f-2b0d-4cf3-9419-31b5211ab2f5-hostroot\") pod \"multus-phjk6\" (UID: \"ddccb44f-2b0d-4cf3-9419-31b5211ab2f5\") " pod="openshift-multus/multus-phjk6"
Sep 09 02:30:20.375280 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.374035    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d33ba4b9-3e00-405a-bc2f-9144dbfa5c8f-cni-binary-copy\") pod \"multus-additional-cni-plugins-zw227\" (UID: \"d33ba4b9-3e00-405a-bc2f-9144dbfa5c8f\") " pod="openshift-multus/multus-additional-cni-plugins-zw227"
Sep 09 02:30:20.375280 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.374043    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ddccb44f-2b0d-4cf3-9419-31b5211ab2f5-cni-binary-copy\") pod \"multus-phjk6\" (UID: \"ddccb44f-2b0d-4cf3-9419-31b5211ab2f5\") " pod="openshift-multus/multus-phjk6"
Sep 09 02:30:20.389106 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.389033    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2m477\" (UniqueName: \"kubernetes.io/projected/d33ba4b9-3e00-405a-bc2f-9144dbfa5c8f-kube-api-access-2m477\") pod \"multus-additional-cni-plugins-zw227\" (UID: \"d33ba4b9-3e00-405a-bc2f-9144dbfa5c8f\") " pod="openshift-multus/multus-additional-cni-plugins-zw227"
Sep 09 02:30:20.389502 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.389467    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcc4f\" (UniqueName: \"kubernetes.io/projected/67afff59-c0d9-43fb-9b1f-581ceab88dbd-kube-api-access-rcc4f\") pod \"iptables-alerter-h2sdl\" (UID: \"67afff59-c0d9-43fb-9b1f-581ceab88dbd\") " pod="openshift-network-operator/iptables-alerter-h2sdl"
Sep 09 02:30:20.390613 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.390594    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvz5v\" (UniqueName: \"kubernetes.io/projected/ddccb44f-2b0d-4cf3-9419-31b5211ab2f5-kube-api-access-tvz5v\") pod \"multus-phjk6\" (UID: \"ddccb44f-2b0d-4cf3-9419-31b5211ab2f5\") " pod="openshift-multus/multus-phjk6"
Sep 09 02:30:20.390731 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.390710    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mhkj\" (UniqueName: \"kubernetes.io/projected/c3e9e8ee-29c7-4fe7-8bcd-7f2363992b3c-kube-api-access-2mhkj\") pod \"aws-ebs-csi-driver-node-2f82x\" (UID: \"c3e9e8ee-29c7-4fe7-8bcd-7f2363992b3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2f82x"
Sep 09 02:30:20.391298 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.391273    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6w59\" (UniqueName: \"kubernetes.io/projected/721fa3c1-a7d2-4245-a7bd-76eb7facfc36-kube-api-access-k6w59\") pod \"node-ca-zg4dm\" (UID: \"721fa3c1-a7d2-4245-a7bd-76eb7facfc36\") " pod="openshift-image-registry/node-ca-zg4dm"
Sep 09 02:30:20.458575 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.458544    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-d7t89"
Sep 09 02:30:20.465774 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.465750    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xcqxh"
Sep 09 02:30:20.476842 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.476822    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-kxvb7"
Sep 09 02:30:20.482650 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.482626    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-sv9fq"
Sep 09 02:30:20.489465 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.489446    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2f82x"
Sep 09 02:30:20.496042 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.496026    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zg4dm"
Sep 09 02:30:20.505782 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.505761    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-phjk6"
Sep 09 02:30:20.512460 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.512441    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-h2sdl"
Sep 09 02:30:20.517243 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.517225    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-zw227"
Sep 09 02:30:20.600011 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.599989    2570 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 02:30:20.776955 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.776886    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d786708a-bbb1-4250-a2d3-84dc2299d71a-metrics-certs\") pod \"network-metrics-daemon-zrljs\" (UID: \"d786708a-bbb1-4250-a2d3-84dc2299d71a\") " pod="openshift-multus/network-metrics-daemon-zrljs"
Sep 09 02:30:20.777159 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:20.777060    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 02:30:20.777159 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:20.777128    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d786708a-bbb1-4250-a2d3-84dc2299d71a-metrics-certs podName:d786708a-bbb1-4250-a2d3-84dc2299d71a nodeName:}" failed. No retries permitted until 2026-09-09 02:30:21.777108224 +0000 UTC m=+4.031411839 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d786708a-bbb1-4250-a2d3-84dc2299d71a-metrics-certs") pod "network-metrics-daemon-zrljs" (UID: "d786708a-bbb1-4250-a2d3-84dc2299d71a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 02:30:20.877497 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.877467    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/37ef684a-ce44-43fc-a803-519ce7d4d225-original-pull-secret\") pod \"global-pull-secret-syncer-952h4\" (UID: \"37ef684a-ce44-43fc-a803-519ce7d4d225\") " pod="kube-system/global-pull-secret-syncer-952h4"
Sep 09 02:30:20.877672 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:20.877518    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drrcs\" (UniqueName: \"kubernetes.io/projected/ac0a0b68-d43e-4ca8-8c4d-39bbe5e7519c-kube-api-access-drrcs\") pod \"network-check-target-6gkgn\" (UID: \"ac0a0b68-d43e-4ca8-8c4d-39bbe5e7519c\") " pod="openshift-network-diagnostics/network-check-target-6gkgn"
Sep 09 02:30:20.877672 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:20.877632    2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 02:30:20.877672 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:20.877643    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 02:30:20.877672 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:20.877657    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 02:30:20.877672 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:20.877667    2570 projected.go:196] Error preparing data for projected volume kube-api-access-drrcs for pod openshift-network-diagnostics/network-check-target-6gkgn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 02:30:20.877879 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:20.877700    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37ef684a-ce44-43fc-a803-519ce7d4d225-original-pull-secret podName:37ef684a-ce44-43fc-a803-519ce7d4d225 nodeName:}" failed. No retries permitted until 2026-09-09 02:30:21.87768097 +0000 UTC m=+4.131984604 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/37ef684a-ce44-43fc-a803-519ce7d4d225-original-pull-secret") pod "global-pull-secret-syncer-952h4" (UID: "37ef684a-ce44-43fc-a803-519ce7d4d225") : object "kube-system"/"original-pull-secret" not registered
Sep 09 02:30:20.877879 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:20.877717    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ac0a0b68-d43e-4ca8-8c4d-39bbe5e7519c-kube-api-access-drrcs podName:ac0a0b68-d43e-4ca8-8c4d-39bbe5e7519c nodeName:}" failed. No retries permitted until 2026-09-09 02:30:21.877708475 +0000 UTC m=+4.132012089 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-drrcs" (UniqueName: "kubernetes.io/projected/ac0a0b68-d43e-4ca8-8c4d-39bbe5e7519c-kube-api-access-drrcs") pod "network-check-target-6gkgn" (UID: "ac0a0b68-d43e-4ca8-8c4d-39bbe5e7519c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 02:30:21.103373 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:21.103346    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90413a10_3c11_4127_b7c8_6e8ac09b1fc9.slice/crio-0200e3b4bc89f3128331a488317f8570fd610f8a144b6ec926e885ce4354527d WatchSource:0}: Error finding container 0200e3b4bc89f3128331a488317f8570fd610f8a144b6ec926e885ce4354527d: Status 404 returned error can't find the container with id 0200e3b4bc89f3128331a488317f8570fd610f8a144b6ec926e885ce4354527d
Sep 09 02:30:21.104335 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:21.104312    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddccb44f_2b0d_4cf3_9419_31b5211ab2f5.slice/crio-7d1b8bcb6ce0a7553e5353a4747480ee10631f322077b541b1fb72cdb3738971 WatchSource:0}: Error finding container 7d1b8bcb6ce0a7553e5353a4747480ee10631f322077b541b1fb72cdb3738971: Status 404 returned error can't find the container with id 7d1b8bcb6ce0a7553e5353a4747480ee10631f322077b541b1fb72cdb3738971
Sep 09 02:30:21.105107 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:21.105082    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod679d664f_a0b2_4ad9_94e4_014aef8f2e16.slice/crio-74d700144f86fe2946cb87b409c23b590c4870ad71953c1b105e2b0800e1f85b WatchSource:0}: Error finding container 74d700144f86fe2946cb87b409c23b590c4870ad71953c1b105e2b0800e1f85b: Status 404 returned error can't find the container with id 74d700144f86fe2946cb87b409c23b590c4870ad71953c1b105e2b0800e1f85b
Sep 09 02:30:21.106202 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:21.106141    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0d4f170_d793_4602_9c77_afdf152cd687.slice/crio-d23be7e5f092c416d3c7aed7f186b800b976fbf2258277f3351be71d5cf9ef76 WatchSource:0}: Error finding container d23be7e5f092c416d3c7aed7f186b800b976fbf2258277f3351be71d5cf9ef76: Status 404 returned error can't find the container with id d23be7e5f092c416d3c7aed7f186b800b976fbf2258277f3351be71d5cf9ef76
Sep 09 02:30:21.109409 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:21.109359    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3e9e8ee_29c7_4fe7_8bcd_7f2363992b3c.slice/crio-86a5006b185f05759bfd95ee6450f18bfa9776b0b93e9ed38d939cb224cfd8a5 WatchSource:0}: Error finding container 86a5006b185f05759bfd95ee6450f18bfa9776b0b93e9ed38d939cb224cfd8a5: Status 404 returned error can't find the container with id 86a5006b185f05759bfd95ee6450f18bfa9776b0b93e9ed38d939cb224cfd8a5
Sep 09 02:30:21.110921 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:21.110781    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57852f55_dada_41bc_927d_466452a75957.slice/crio-ff812b11d05f52cc38a489884274bf04d17234f4c184d6eb303f084c0a4f9004 WatchSource:0}: Error finding container ff812b11d05f52cc38a489884274bf04d17234f4c184d6eb303f084c0a4f9004: Status 404 returned error can't find the container with id ff812b11d05f52cc38a489884274bf04d17234f4c184d6eb303f084c0a4f9004
Sep 09 02:30:21.111495 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:21.111475    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod721fa3c1_a7d2_4245_a7bd_76eb7facfc36.slice/crio-8e5621f67e11359e5885b3b3d076ab3078a3765322fba7916c72fdf1d483097a WatchSource:0}: Error finding container 8e5621f67e11359e5885b3b3d076ab3078a3765322fba7916c72fdf1d483097a: Status 404 returned error can't find the container with id 8e5621f67e11359e5885b3b3d076ab3078a3765322fba7916c72fdf1d483097a
Sep 09 02:30:21.113330 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:21.113242    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd33ba4b9_3e00_405a_bc2f_9144dbfa5c8f.slice/crio-38acba9aba613146adb23c2f6dc7d51adf0baa70f15089178aec89400f568398 WatchSource:0}: Error finding container 38acba9aba613146adb23c2f6dc7d51adf0baa70f15089178aec89400f568398: Status 404 returned error can't find the container with id 38acba9aba613146adb23c2f6dc7d51adf0baa70f15089178aec89400f568398
Sep 09 02:30:21.115200 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:21.115178    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67afff59_c0d9_43fb_9b1f_581ceab88dbd.slice/crio-764ce6e1ee9cd83d940bf46273609aa1115484de795d30f9a360c0116c36d160 WatchSource:0}: Error finding container 764ce6e1ee9cd83d940bf46273609aa1115484de795d30f9a360c0116c36d160: Status 404 returned error can't find the container with id 764ce6e1ee9cd83d940bf46273609aa1115484de795d30f9a360c0116c36d160
Sep 09 02:30:21.208937 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:21.208907    2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 02:25:20 +0000 UTC" deadline="2028-03-23 10:57:19.000699063 +0000 UTC"
Sep 09 02:30:21.208937 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:21.208933    2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13472h26m57.791768788s"
Sep 09 02:30:21.217324 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:21.217299    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-952h4"
Sep 09 02:30:21.217443 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:21.217422    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-952h4" podUID="37ef684a-ce44-43fc-a803-519ce7d4d225"
Sep 09 02:30:21.226988 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:21.226935    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-h2sdl" event={"ID":"67afff59-c0d9-43fb-9b1f-581ceab88dbd","Type":"ContainerStarted","Data":"764ce6e1ee9cd83d940bf46273609aa1115484de795d30f9a360c0116c36d160"}
Sep 09 02:30:21.228101 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:21.228077    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zw227" event={"ID":"d33ba4b9-3e00-405a-bc2f-9144dbfa5c8f","Type":"ContainerStarted","Data":"38acba9aba613146adb23c2f6dc7d51adf0baa70f15089178aec89400f568398"}
Sep 09 02:30:21.229283 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:21.229255    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xcqxh" event={"ID":"57852f55-dada-41bc-927d-466452a75957","Type":"ContainerStarted","Data":"ff812b11d05f52cc38a489884274bf04d17234f4c184d6eb303f084c0a4f9004"}
Sep 09 02:30:21.230372 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:21.230346    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-sv9fq" event={"ID":"a0d4f170-d793-4602-9c77-afdf152cd687","Type":"ContainerStarted","Data":"d23be7e5f092c416d3c7aed7f186b800b976fbf2258277f3351be71d5cf9ef76"}
Sep 09 02:30:21.231571 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:21.231547    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-d7t89" event={"ID":"90413a10-3c11-4127-b7c8-6e8ac09b1fc9","Type":"ContainerStarted","Data":"0200e3b4bc89f3128331a488317f8570fd610f8a144b6ec926e885ce4354527d"}
Sep 09 02:30:21.233524 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:21.233498    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-188.ec2.internal" event={"ID":"871e64cd6b8e47e52f74676ffa7cfe0c","Type":"ContainerStarted","Data":"4c5374ab51040a7a5dcd7850dbaea6f3405c97a35ab78e0135f2f4c2dc0fd811"}
Sep 09 02:30:21.234750 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:21.234731    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zg4dm" event={"ID":"721fa3c1-a7d2-4245-a7bd-76eb7facfc36","Type":"ContainerStarted","Data":"8e5621f67e11359e5885b3b3d076ab3078a3765322fba7916c72fdf1d483097a"}
Sep 09 02:30:21.235985 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:21.235944    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2f82x" event={"ID":"c3e9e8ee-29c7-4fe7-8bcd-7f2363992b3c","Type":"ContainerStarted","Data":"86a5006b185f05759bfd95ee6450f18bfa9776b0b93e9ed38d939cb224cfd8a5"}
Sep 09 02:30:21.237225 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:21.237186    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-kxvb7" event={"ID":"679d664f-a0b2-4ad9-94e4-014aef8f2e16","Type":"ContainerStarted","Data":"74d700144f86fe2946cb87b409c23b590c4870ad71953c1b105e2b0800e1f85b"}
Sep 09 02:30:21.238641 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:21.238621    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-phjk6" event={"ID":"ddccb44f-2b0d-4cf3-9419-31b5211ab2f5","Type":"ContainerStarted","Data":"7d1b8bcb6ce0a7553e5353a4747480ee10631f322077b541b1fb72cdb3738971"}
Sep 09 02:30:21.262056 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:21.262016    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-188.ec2.internal" podStartSLOduration=2.26200321 podStartE2EDuration="2.26200321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 02:30:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:30:21.261799443 +0000 UTC m=+3.516103080" watchObservedRunningTime="2026-09-09 02:30:21.26200321 +0000 UTC m=+3.516306845"
Sep 09 02:30:21.783768 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:21.783440    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d786708a-bbb1-4250-a2d3-84dc2299d71a-metrics-certs\") pod \"network-metrics-daemon-zrljs\" (UID: \"d786708a-bbb1-4250-a2d3-84dc2299d71a\") " pod="openshift-multus/network-metrics-daemon-zrljs"
Sep 09 02:30:21.783928 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:21.783814    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 02:30:21.783928 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:21.783884    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d786708a-bbb1-4250-a2d3-84dc2299d71a-metrics-certs podName:d786708a-bbb1-4250-a2d3-84dc2299d71a nodeName:}" failed. No retries permitted until 2026-09-09 02:30:23.783863239 +0000 UTC m=+6.038166869 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d786708a-bbb1-4250-a2d3-84dc2299d71a-metrics-certs") pod "network-metrics-daemon-zrljs" (UID: "d786708a-bbb1-4250-a2d3-84dc2299d71a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 02:30:21.884628 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:21.884593    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drrcs\" (UniqueName: \"kubernetes.io/projected/ac0a0b68-d43e-4ca8-8c4d-39bbe5e7519c-kube-api-access-drrcs\") pod \"network-check-target-6gkgn\" (UID: \"ac0a0b68-d43e-4ca8-8c4d-39bbe5e7519c\") " pod="openshift-network-diagnostics/network-check-target-6gkgn"
Sep 09 02:30:21.884799 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:21.884669    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/37ef684a-ce44-43fc-a803-519ce7d4d225-original-pull-secret\") pod \"global-pull-secret-syncer-952h4\" (UID: \"37ef684a-ce44-43fc-a803-519ce7d4d225\") " pod="kube-system/global-pull-secret-syncer-952h4"
Sep 09 02:30:21.884856 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:21.884804    2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 02:30:21.884911 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:21.884863    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37ef684a-ce44-43fc-a803-519ce7d4d225-original-pull-secret podName:37ef684a-ce44-43fc-a803-519ce7d4d225 nodeName:}" failed. No retries permitted until 2026-09-09 02:30:23.884845458 +0000 UTC m=+6.139149098 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/37ef684a-ce44-43fc-a803-519ce7d4d225-original-pull-secret") pod "global-pull-secret-syncer-952h4" (UID: "37ef684a-ce44-43fc-a803-519ce7d4d225") : object "kube-system"/"original-pull-secret" not registered
Sep 09 02:30:21.884991 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:21.884948    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 02:30:21.884991 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:21.884962    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 02:30:21.884991 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:21.884990    2570 projected.go:196] Error preparing data for projected volume kube-api-access-drrcs for pod openshift-network-diagnostics/network-check-target-6gkgn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 02:30:21.885141 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:21.885029    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ac0a0b68-d43e-4ca8-8c4d-39bbe5e7519c-kube-api-access-drrcs podName:ac0a0b68-d43e-4ca8-8c4d-39bbe5e7519c nodeName:}" failed. No retries permitted until 2026-09-09 02:30:23.885017838 +0000 UTC m=+6.139321467 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-drrcs" (UniqueName: "kubernetes.io/projected/ac0a0b68-d43e-4ca8-8c4d-39bbe5e7519c-kube-api-access-drrcs") pod "network-check-target-6gkgn" (UID: "ac0a0b68-d43e-4ca8-8c4d-39bbe5e7519c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 02:30:22.209795 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:22.209655    2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 02:25:20 +0000 UTC" deadline="2028-05-24 17:50:30.171464418 +0000 UTC"
Sep 09 02:30:22.209795 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:22.209715    2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14967h20m7.961753273s"
Sep 09 02:30:22.218024 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:22.217526    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zrljs"
Sep 09 02:30:22.218024 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:22.217664    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zrljs" podUID="d786708a-bbb1-4250-a2d3-84dc2299d71a"
Sep 09 02:30:22.218479 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:22.218301    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6gkgn"
Sep 09 02:30:22.218479 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:22.218411    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6gkgn" podUID="ac0a0b68-d43e-4ca8-8c4d-39bbe5e7519c"
Sep 09 02:30:22.255453 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:22.255426    2570 generic.go:356] "Generic (PLEG): container finished" podID="7c9f79a4d290b3116ce4e01e83369531" containerID="ed9c4c019795ee093a0f0ae14985cc1452b68b49fbca5ed1ad80ade8c0065685" exitCode=0
Sep 09 02:30:22.255937 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:22.255913    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-188.ec2.internal" event={"ID":"7c9f79a4d290b3116ce4e01e83369531","Type":"ContainerDied","Data":"ed9c4c019795ee093a0f0ae14985cc1452b68b49fbca5ed1ad80ade8c0065685"}
Sep 09 02:30:23.216613 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:23.216526    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-952h4"
Sep 09 02:30:23.217053 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:23.216674    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-952h4" podUID="37ef684a-ce44-43fc-a803-519ce7d4d225"
Sep 09 02:30:23.281007 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:23.280952    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-188.ec2.internal" event={"ID":"7c9f79a4d290b3116ce4e01e83369531","Type":"ContainerStarted","Data":"307a8abdeae018b322b7c08e949da998eebaa829a62bef8e2d5b9dc60c0b5250"}
Sep 09 02:30:23.301801 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:23.301293    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-188.ec2.internal" podStartSLOduration=4.301271826 podStartE2EDuration="4.301271826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 02:30:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:30:23.300102574 +0000 UTC m=+5.554406211" watchObservedRunningTime="2026-09-09 02:30:23.301271826 +0000 UTC m=+5.555575462"
Sep 09 02:30:23.804945 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:23.804909    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d786708a-bbb1-4250-a2d3-84dc2299d71a-metrics-certs\") pod \"network-metrics-daemon-zrljs\" (UID: \"d786708a-bbb1-4250-a2d3-84dc2299d71a\") " pod="openshift-multus/network-metrics-daemon-zrljs"
Sep 09 02:30:23.805120 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:23.805075    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 02:30:23.805184 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:23.805137    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d786708a-bbb1-4250-a2d3-84dc2299d71a-metrics-certs podName:d786708a-bbb1-4250-a2d3-84dc2299d71a nodeName:}" failed. No retries permitted until 2026-09-09 02:30:27.805119649 +0000 UTC m=+10.059423266 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d786708a-bbb1-4250-a2d3-84dc2299d71a-metrics-certs") pod "network-metrics-daemon-zrljs" (UID: "d786708a-bbb1-4250-a2d3-84dc2299d71a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 02:30:23.905544 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:23.905511    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/37ef684a-ce44-43fc-a803-519ce7d4d225-original-pull-secret\") pod \"global-pull-secret-syncer-952h4\" (UID: \"37ef684a-ce44-43fc-a803-519ce7d4d225\") " pod="kube-system/global-pull-secret-syncer-952h4"
Sep 09 02:30:23.905722 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:23.905582    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drrcs\" (UniqueName: \"kubernetes.io/projected/ac0a0b68-d43e-4ca8-8c4d-39bbe5e7519c-kube-api-access-drrcs\") pod \"network-check-target-6gkgn\" (UID: \"ac0a0b68-d43e-4ca8-8c4d-39bbe5e7519c\") " pod="openshift-network-diagnostics/network-check-target-6gkgn"
Sep 09 02:30:23.905722 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:23.905716    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 02:30:23.905828 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:23.905740    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 02:30:23.905828 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:23.905754    2570 projected.go:196] Error preparing data for projected volume kube-api-access-drrcs for pod openshift-network-diagnostics/network-check-target-6gkgn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 02:30:23.905828 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:23.905811    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ac0a0b68-d43e-4ca8-8c4d-39bbe5e7519c-kube-api-access-drrcs podName:ac0a0b68-d43e-4ca8-8c4d-39bbe5e7519c nodeName:}" failed. No retries permitted until 2026-09-09 02:30:27.905794627 +0000 UTC m=+10.160098248 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-drrcs" (UniqueName: "kubernetes.io/projected/ac0a0b68-d43e-4ca8-8c4d-39bbe5e7519c-kube-api-access-drrcs") pod "network-check-target-6gkgn" (UID: "ac0a0b68-d43e-4ca8-8c4d-39bbe5e7519c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 02:30:23.906117 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:23.906096    2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 02:30:23.906201 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:23.906164    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37ef684a-ce44-43fc-a803-519ce7d4d225-original-pull-secret podName:37ef684a-ce44-43fc-a803-519ce7d4d225 nodeName:}" failed. No retries permitted until 2026-09-09 02:30:27.906148644 +0000 UTC m=+10.160452283 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/37ef684a-ce44-43fc-a803-519ce7d4d225-original-pull-secret") pod "global-pull-secret-syncer-952h4" (UID: "37ef684a-ce44-43fc-a803-519ce7d4d225") : object "kube-system"/"original-pull-secret" not registered
Sep 09 02:30:24.217568 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:24.217446    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6gkgn"
Sep 09 02:30:24.217568 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:24.217483    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zrljs"
Sep 09 02:30:24.218074 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:24.217572    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6gkgn" podUID="ac0a0b68-d43e-4ca8-8c4d-39bbe5e7519c"
Sep 09 02:30:24.218074 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:24.217994    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zrljs" podUID="d786708a-bbb1-4250-a2d3-84dc2299d71a"
Sep 09 02:30:25.216982 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:25.216921    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-952h4"
Sep 09 02:30:25.217169 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:25.217072    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-952h4" podUID="37ef684a-ce44-43fc-a803-519ce7d4d225"
Sep 09 02:30:26.216860 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:26.216816    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zrljs"
Sep 09 02:30:26.217357 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:26.216833    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6gkgn"
Sep 09 02:30:26.217357 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:26.216957    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zrljs" podUID="d786708a-bbb1-4250-a2d3-84dc2299d71a"
Sep 09 02:30:26.217357 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:26.217024    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6gkgn" podUID="ac0a0b68-d43e-4ca8-8c4d-39bbe5e7519c"
Sep 09 02:30:27.217271 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:27.217236    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-952h4"
Sep 09 02:30:27.217723 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:27.217336    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-952h4" podUID="37ef684a-ce44-43fc-a803-519ce7d4d225"
Sep 09 02:30:27.835947 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:27.835907    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d786708a-bbb1-4250-a2d3-84dc2299d71a-metrics-certs\") pod \"network-metrics-daemon-zrljs\" (UID: \"d786708a-bbb1-4250-a2d3-84dc2299d71a\") " pod="openshift-multus/network-metrics-daemon-zrljs"
Sep 09 02:30:27.836147 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:27.836102    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 02:30:27.836208 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:27.836177    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d786708a-bbb1-4250-a2d3-84dc2299d71a-metrics-certs podName:d786708a-bbb1-4250-a2d3-84dc2299d71a nodeName:}" failed. No retries permitted until 2026-09-09 02:30:35.836155629 +0000 UTC m=+18.090459244 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d786708a-bbb1-4250-a2d3-84dc2299d71a-metrics-certs") pod "network-metrics-daemon-zrljs" (UID: "d786708a-bbb1-4250-a2d3-84dc2299d71a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 02:30:27.937089 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:27.937051    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/37ef684a-ce44-43fc-a803-519ce7d4d225-original-pull-secret\") pod \"global-pull-secret-syncer-952h4\" (UID: \"37ef684a-ce44-43fc-a803-519ce7d4d225\") " pod="kube-system/global-pull-secret-syncer-952h4"
Sep 09 02:30:27.937310 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:27.937143    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drrcs\" (UniqueName: \"kubernetes.io/projected/ac0a0b68-d43e-4ca8-8c4d-39bbe5e7519c-kube-api-access-drrcs\") pod \"network-check-target-6gkgn\" (UID: \"ac0a0b68-d43e-4ca8-8c4d-39bbe5e7519c\") " pod="openshift-network-diagnostics/network-check-target-6gkgn"
Sep 09 02:30:27.937310 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:27.937218    2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 02:30:27.937310 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:27.937285    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 02:30:27.937310 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:27.937293    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37ef684a-ce44-43fc-a803-519ce7d4d225-original-pull-secret podName:37ef684a-ce44-43fc-a803-519ce7d4d225 nodeName:}" failed. No retries permitted until 2026-09-09 02:30:35.937272602 +0000 UTC m=+18.191576215 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/37ef684a-ce44-43fc-a803-519ce7d4d225-original-pull-secret") pod "global-pull-secret-syncer-952h4" (UID: "37ef684a-ce44-43fc-a803-519ce7d4d225") : object "kube-system"/"original-pull-secret" not registered
Sep 09 02:30:27.937310 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:27.937302    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 02:30:27.937559 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:27.937322    2570 projected.go:196] Error preparing data for projected volume kube-api-access-drrcs for pod openshift-network-diagnostics/network-check-target-6gkgn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 02:30:27.937559 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:27.937376    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ac0a0b68-d43e-4ca8-8c4d-39bbe5e7519c-kube-api-access-drrcs podName:ac0a0b68-d43e-4ca8-8c4d-39bbe5e7519c nodeName:}" failed. No retries permitted until 2026-09-09 02:30:35.937361771 +0000 UTC m=+18.191665401 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-drrcs" (UniqueName: "kubernetes.io/projected/ac0a0b68-d43e-4ca8-8c4d-39bbe5e7519c-kube-api-access-drrcs") pod "network-check-target-6gkgn" (UID: "ac0a0b68-d43e-4ca8-8c4d-39bbe5e7519c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 02:30:28.218862 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:28.217298    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zrljs"
Sep 09 02:30:28.218862 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:28.217431    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zrljs" podUID="d786708a-bbb1-4250-a2d3-84dc2299d71a"
Sep 09 02:30:28.218862 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:28.217458    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6gkgn"
Sep 09 02:30:28.218862 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:28.217563    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6gkgn" podUID="ac0a0b68-d43e-4ca8-8c4d-39bbe5e7519c"
Sep 09 02:30:29.217396 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:29.216874    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-952h4"
Sep 09 02:30:29.217396 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:29.217018    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-952h4" podUID="37ef684a-ce44-43fc-a803-519ce7d4d225"
Sep 09 02:30:30.216642 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:30.216601    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6gkgn"
Sep 09 02:30:30.217149 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:30.216601    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zrljs"
Sep 09 02:30:30.217149 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:30.216771    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6gkgn" podUID="ac0a0b68-d43e-4ca8-8c4d-39bbe5e7519c"
Sep 09 02:30:30.217149 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:30.216845    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zrljs" podUID="d786708a-bbb1-4250-a2d3-84dc2299d71a"
Sep 09 02:30:31.216725 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:31.216695    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-952h4"
Sep 09 02:30:31.217027 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:31.216797    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-952h4" podUID="37ef684a-ce44-43fc-a803-519ce7d4d225"
Sep 09 02:30:32.217002 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:32.216960    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zrljs"
Sep 09 02:30:32.217426 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:32.216957    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6gkgn"
Sep 09 02:30:32.217426 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:32.217103    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zrljs" podUID="d786708a-bbb1-4250-a2d3-84dc2299d71a"
Sep 09 02:30:32.217426 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:32.217169    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6gkgn" podUID="ac0a0b68-d43e-4ca8-8c4d-39bbe5e7519c"
Sep 09 02:30:33.216429 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:33.216390    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-952h4"
Sep 09 02:30:33.216575 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:33.216492    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-952h4" podUID="37ef684a-ce44-43fc-a803-519ce7d4d225"
Sep 09 02:30:34.217113 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:34.217078    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6gkgn"
Sep 09 02:30:34.217565 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:34.217083    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zrljs"
Sep 09 02:30:34.217565 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:34.217212    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6gkgn" podUID="ac0a0b68-d43e-4ca8-8c4d-39bbe5e7519c"
Sep 09 02:30:34.217565 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:34.217288    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zrljs" podUID="d786708a-bbb1-4250-a2d3-84dc2299d71a"
Sep 09 02:30:35.217068 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:35.216993    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-952h4"
Sep 09 02:30:35.217244 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:35.217097    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-952h4" podUID="37ef684a-ce44-43fc-a803-519ce7d4d225"
Sep 09 02:30:35.891350 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:35.891317    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d786708a-bbb1-4250-a2d3-84dc2299d71a-metrics-certs\") pod \"network-metrics-daemon-zrljs\" (UID: \"d786708a-bbb1-4250-a2d3-84dc2299d71a\") " pod="openshift-multus/network-metrics-daemon-zrljs"
Sep 09 02:30:35.891512 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:35.891469    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 02:30:35.891562 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:35.891538    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d786708a-bbb1-4250-a2d3-84dc2299d71a-metrics-certs podName:d786708a-bbb1-4250-a2d3-84dc2299d71a nodeName:}" failed. No retries permitted until 2026-09-09 02:30:51.891519678 +0000 UTC m=+34.145823495 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d786708a-bbb1-4250-a2d3-84dc2299d71a-metrics-certs") pod "network-metrics-daemon-zrljs" (UID: "d786708a-bbb1-4250-a2d3-84dc2299d71a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 02:30:35.992263 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:35.992233    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/37ef684a-ce44-43fc-a803-519ce7d4d225-original-pull-secret\") pod \"global-pull-secret-syncer-952h4\" (UID: \"37ef684a-ce44-43fc-a803-519ce7d4d225\") " pod="kube-system/global-pull-secret-syncer-952h4"
Sep 09 02:30:35.992438 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:35.992280    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drrcs\" (UniqueName: \"kubernetes.io/projected/ac0a0b68-d43e-4ca8-8c4d-39bbe5e7519c-kube-api-access-drrcs\") pod \"network-check-target-6gkgn\" (UID: \"ac0a0b68-d43e-4ca8-8c4d-39bbe5e7519c\") " pod="openshift-network-diagnostics/network-check-target-6gkgn"
Sep 09 02:30:35.992438 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:35.992382    2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 02:30:35.992438 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:35.992411    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 02:30:35.992438 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:35.992425    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 02:30:35.992438 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:35.992436    2570 projected.go:196] Error preparing data for projected volume kube-api-access-drrcs for pod openshift-network-diagnostics/network-check-target-6gkgn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 02:30:35.992683 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:35.992449    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37ef684a-ce44-43fc-a803-519ce7d4d225-original-pull-secret podName:37ef684a-ce44-43fc-a803-519ce7d4d225 nodeName:}" failed. No retries permitted until 2026-09-09 02:30:51.992431339 +0000 UTC m=+34.246734971 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/37ef684a-ce44-43fc-a803-519ce7d4d225-original-pull-secret") pod "global-pull-secret-syncer-952h4" (UID: "37ef684a-ce44-43fc-a803-519ce7d4d225") : object "kube-system"/"original-pull-secret" not registered
Sep 09 02:30:35.992683 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:35.992477    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ac0a0b68-d43e-4ca8-8c4d-39bbe5e7519c-kube-api-access-drrcs podName:ac0a0b68-d43e-4ca8-8c4d-39bbe5e7519c nodeName:}" failed. No retries permitted until 2026-09-09 02:30:51.992462872 +0000 UTC m=+34.246766503 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-drrcs" (UniqueName: "kubernetes.io/projected/ac0a0b68-d43e-4ca8-8c4d-39bbe5e7519c-kube-api-access-drrcs") pod "network-check-target-6gkgn" (UID: "ac0a0b68-d43e-4ca8-8c4d-39bbe5e7519c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 02:30:36.217110 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:36.217033    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6gkgn"
Sep 09 02:30:36.217265 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:36.217043    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zrljs"
Sep 09 02:30:36.217265 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:36.217176    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6gkgn" podUID="ac0a0b68-d43e-4ca8-8c4d-39bbe5e7519c"
Sep 09 02:30:36.217265 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:36.217244    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zrljs" podUID="d786708a-bbb1-4250-a2d3-84dc2299d71a"
Sep 09 02:30:37.216724 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:37.216676    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-952h4"
Sep 09 02:30:37.216948 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:37.216787    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-952h4" podUID="37ef684a-ce44-43fc-a803-519ce7d4d225"
Sep 09 02:30:38.218058 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:38.218025    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6gkgn"
Sep 09 02:30:38.218536 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:38.218115    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6gkgn" podUID="ac0a0b68-d43e-4ca8-8c4d-39bbe5e7519c"
Sep 09 02:30:38.218536 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:38.218186    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zrljs"
Sep 09 02:30:38.218536 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:38.218304    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zrljs" podUID="d786708a-bbb1-4250-a2d3-84dc2299d71a"
Sep 09 02:30:39.216638 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:39.216361    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-952h4"
Sep 09 02:30:39.216737 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:39.216719    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-952h4" podUID="37ef684a-ce44-43fc-a803-519ce7d4d225"
Sep 09 02:30:39.327159 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:39.327071    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-kxvb7" event={"ID":"679d664f-a0b2-4ad9-94e4-014aef8f2e16","Type":"ContainerStarted","Data":"05270c2f92687280e7c14c4bdae8312efc69e48f29f931de9a6b39e19def4bd9"}
Sep 09 02:30:39.328757 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:39.328726    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-phjk6" event={"ID":"ddccb44f-2b0d-4cf3-9419-31b5211ab2f5","Type":"ContainerStarted","Data":"b0479b1cfd31aea1a2bbe42eafb79be1626be8aa70fbb170dab116fba1823003"}
Sep 09 02:30:39.330424 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:39.330397    2570 generic.go:356] "Generic (PLEG): container finished" podID="d33ba4b9-3e00-405a-bc2f-9144dbfa5c8f" containerID="2692edd4da5023bea1f508413de1da77b1ccba09d0b7f9a71860e9729733ffb0" exitCode=0
Sep 09 02:30:39.330521 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:39.330480    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zw227" event={"ID":"d33ba4b9-3e00-405a-bc2f-9144dbfa5c8f","Type":"ContainerDied","Data":"2692edd4da5023bea1f508413de1da77b1ccba09d0b7f9a71860e9729733ffb0"}
Sep 09 02:30:39.334127 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:39.334104    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xcqxh" event={"ID":"57852f55-dada-41bc-927d-466452a75957","Type":"ContainerStarted","Data":"ab709c80bef296cfe86b6f3cb08af5a6d9b4bd81c44959ada65f732e77b3097c"}
Sep 09 02:30:39.334225 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:39.334177    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xcqxh" event={"ID":"57852f55-dada-41bc-927d-466452a75957","Type":"ContainerStarted","Data":"f2257bd8e08c38a147432442b3b3ed38819aba135ef773bc3764b6402d415bdd"}
Sep 09 02:30:39.334286 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:39.334229    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xcqxh" event={"ID":"57852f55-dada-41bc-927d-466452a75957","Type":"ContainerStarted","Data":"92aab141dd3947e0d54cf554a7da6726bb6498d02b8b9dd02914cd8be2e3d537"}
Sep 09 02:30:39.334286 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:39.334276    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xcqxh" event={"ID":"57852f55-dada-41bc-927d-466452a75957","Type":"ContainerStarted","Data":"431523a7e169ea96e50c7c00b9067bf35869b6699964bc37eec9b893e8485f36"}
Sep 09 02:30:39.334395 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:39.334324    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xcqxh" event={"ID":"57852f55-dada-41bc-927d-466452a75957","Type":"ContainerStarted","Data":"5508c2e44cc6970f20870a6d59fc78f1383e5dc57bda08dea3d53587bb763a0e"}
Sep 09 02:30:39.334395 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:39.334383    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xcqxh" event={"ID":"57852f55-dada-41bc-927d-466452a75957","Type":"ContainerStarted","Data":"976d7538c067ec9844bee4659ff50373c1edecde455315d30597676247a9d868"}
Sep 09 02:30:39.336124 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:39.336102    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-sv9fq" event={"ID":"a0d4f170-d793-4602-9c77-afdf152cd687","Type":"ContainerStarted","Data":"9abbbb5c2c9da007459a23af00eb416cfb13d7f4586b734cfe644dda7c76eb9d"}
Sep 09 02:30:39.337591 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:39.337571    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-d7t89" event={"ID":"90413a10-3c11-4127-b7c8-6e8ac09b1fc9","Type":"ContainerStarted","Data":"f03c66842c426c60d1ca2e6f0fe6b31c41467906912aa9a91ddc1842b970a8b6"}
Sep 09 02:30:39.339109 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:39.339089    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zg4dm" event={"ID":"721fa3c1-a7d2-4245-a7bd-76eb7facfc36","Type":"ContainerStarted","Data":"6e1f883926652fcd767ec4a435efd7e29e263720e9141901c9ccdeb880f21018"}
Sep 09 02:30:39.340770 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:39.340748    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2f82x" event={"ID":"c3e9e8ee-29c7-4fe7-8bcd-7f2363992b3c","Type":"ContainerStarted","Data":"bbfae8e297f2953027ebd356df071cd1425e7090046ddc4f89440d7d54074396"}
Sep 09 02:30:39.343780 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:39.343743    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-kxvb7" podStartSLOduration=7.6143779590000005 podStartE2EDuration="21.343731915s" totalImagesPullingTime="13.729353956s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:30:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:30:39.342844693 +0000 UTC m=+21.597148329" watchObservedRunningTime="2026-09-09 02:30:39.343731915 +0000 UTC m=+21.598035552"
Sep 09 02:30:39.359174 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:39.359136    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-d7t89" podStartSLOduration=3.707367282 podStartE2EDuration="21.359124663s" totalImagesPullingTime="17.651757381s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 02:30:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:30:39.358946035 +0000 UTC m=+21.613249672" watchObservedRunningTime="2026-09-09 02:30:39.359124663 +0000 UTC m=+21.613428298"
Sep 09 02:30:39.379370 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:39.379335    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-sv9fq" podStartSLOduration=3.728936468 podStartE2EDuration="21.379322877s" totalImagesPullingTime="17.650386409s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 02:30:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:30:39.379006286 +0000 UTC m=+21.633309918" watchObservedRunningTime="2026-09-09 02:30:39.379322877 +0000 UTC m=+21.633626543"
Sep 09 02:30:39.408064 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:39.408026    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-zg4dm" podStartSLOduration=3.79256964 podStartE2EDuration="21.408013911s" totalImagesPullingTime="17.615444271s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 02:30:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:30:39.407289211 +0000 UTC m=+21.661592848" watchObservedRunningTime="2026-09-09 02:30:39.408013911 +0000 UTC m=+21.662317547"
Sep 09 02:30:39.428387 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:39.428352    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-phjk6" podStartSLOduration=3.7718678089999997 podStartE2EDuration="21.428341323s" totalImagesPullingTime="17.656473514s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 02:30:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:30:39.428182374 +0000 UTC m=+21.682486009" watchObservedRunningTime="2026-09-09 02:30:39.428341323 +0000 UTC m=+21.682644959"
Sep 09 02:30:39.911657 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:39.911624    2570 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 09 02:30:40.217071 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:40.217037    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6gkgn"
Sep 09 02:30:40.217232 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:40.217040    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zrljs"
Sep 09 02:30:40.217232 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:40.217170    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6gkgn" podUID="ac0a0b68-d43e-4ca8-8c4d-39bbe5e7519c"
Sep 09 02:30:40.217351 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:40.217232    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zrljs" podUID="d786708a-bbb1-4250-a2d3-84dc2299d71a"
Sep 09 02:30:40.225224 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:40.225138    2570 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-09T02:30:39.911643331Z","UUID":"78dd7121-f9ea-46e8-a80b-b16b80d2f8c4","Handler":null,"Name":"","Endpoint":""}
Sep 09 02:30:40.226772 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:40.226750    2570 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 09 02:30:40.226867 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:40.226780    2570 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 09 02:30:40.346374 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:40.346327    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2f82x" event={"ID":"c3e9e8ee-29c7-4fe7-8bcd-7f2363992b3c","Type":"ContainerStarted","Data":"ea772700375a491903bee3706c72d63cdf74ae94bb11cfce452ac833d783d9ee"}
Sep 09 02:30:40.348618 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:40.348589    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-h2sdl" event={"ID":"67afff59-c0d9-43fb-9b1f-581ceab88dbd","Type":"ContainerStarted","Data":"2948cc47e417e9d6483caada70bc45e47c3957e1639678db31b857ffc0fc4a29"}
Sep 09 02:30:40.724321 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:40.724287    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-kxvb7"
Sep 09 02:30:40.724923 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:40.724898    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-kxvb7"
Sep 09 02:30:40.744485 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:40.744433    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-h2sdl" podStartSLOduration=9.024585258 podStartE2EDuration="22.744417904s" totalImagesPullingTime="13.719832646s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 02:30:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:30:40.368188962 +0000 UTC m=+22.622492598" watchObservedRunningTime="2026-09-09 02:30:40.744417904 +0000 UTC m=+22.998721535"
Sep 09 02:30:41.216637 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:41.216610    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-952h4"
Sep 09 02:30:41.216829 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:41.216733    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-952h4" podUID="37ef684a-ce44-43fc-a803-519ce7d4d225"
Sep 09 02:30:41.352841 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:41.352805    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2f82x" event={"ID":"c3e9e8ee-29c7-4fe7-8bcd-7f2363992b3c","Type":"ContainerStarted","Data":"e829f604302082ae416c37cf1d40059b2891deb8192a459f0980eaa49c1643ad"}
Sep 09 02:30:41.378805 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:41.378760    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2f82x" podStartSLOduration=4.1031055 podStartE2EDuration="23.378742389s" totalImagesPullingTime="19.275636889s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 02:30:18 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:30:41.378223865 +0000 UTC m=+23.632527502" watchObservedRunningTime="2026-09-09 02:30:41.378742389 +0000 UTC m=+23.633046028"
Sep 09 02:30:42.216693 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:42.216662    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zrljs"
Sep 09 02:30:42.217004 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:42.216671    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6gkgn"
Sep 09 02:30:42.217004 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:42.216800    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zrljs" podUID="d786708a-bbb1-4250-a2d3-84dc2299d71a"
Sep 09 02:30:42.217004 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:42.216869    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6gkgn" podUID="ac0a0b68-d43e-4ca8-8c4d-39bbe5e7519c"
Sep 09 02:30:42.359073 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:42.359044    2570 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 09 02:30:42.359438 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:42.359037    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xcqxh" event={"ID":"57852f55-dada-41bc-927d-466452a75957","Type":"ContainerStarted","Data":"ef657190cdea729df570ee6c3fc98849a061b1fe21272706923517316828a221"}
Sep 09 02:30:43.216937 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:43.216908    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-952h4"
Sep 09 02:30:43.217106 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:43.217046    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-952h4" podUID="37ef684a-ce44-43fc-a803-519ce7d4d225"
Sep 09 02:30:44.219961 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:44.219763    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6gkgn"
Sep 09 02:30:44.220624 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:44.219776    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zrljs"
Sep 09 02:30:44.220624 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:44.219994    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6gkgn" podUID="ac0a0b68-d43e-4ca8-8c4d-39bbe5e7519c"
Sep 09 02:30:44.220624 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:44.220096    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zrljs" podUID="d786708a-bbb1-4250-a2d3-84dc2299d71a"
Sep 09 02:30:44.364687 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:44.364656    2570 generic.go:356] "Generic (PLEG): container finished" podID="d33ba4b9-3e00-405a-bc2f-9144dbfa5c8f" containerID="54b000c6d0fa67a29bb03593f1e82ecb81038dfca2acb21a9342c1580e36540c" exitCode=0
Sep 09 02:30:44.364839 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:44.364742    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zw227" event={"ID":"d33ba4b9-3e00-405a-bc2f-9144dbfa5c8f","Type":"ContainerDied","Data":"54b000c6d0fa67a29bb03593f1e82ecb81038dfca2acb21a9342c1580e36540c"}
Sep 09 02:30:44.368538 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:44.368516    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xcqxh" event={"ID":"57852f55-dada-41bc-927d-466452a75957","Type":"ContainerStarted","Data":"77d44a41295badb94e14a374c5b3cf181dc7f009478e2c92754860e25ea36d56"}
Sep 09 02:30:44.368764 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:44.368748    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xcqxh"
Sep 09 02:30:44.368837 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:44.368817    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xcqxh"
Sep 09 02:30:44.383327 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:44.383290    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xcqxh"
Sep 09 02:30:44.427065 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:44.427025    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-xcqxh" podStartSLOduration=8.644512944 podStartE2EDuration="26.427012516s" totalImagesPullingTime="17.782499572s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 02:30:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:30:44.424814463 +0000 UTC m=+26.679118100" watchObservedRunningTime="2026-09-09 02:30:44.427012516 +0000 UTC m=+26.681316178"
Sep 09 02:30:45.217118 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:45.216875    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-952h4"
Sep 09 02:30:45.217271 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:45.217229    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-952h4" podUID="37ef684a-ce44-43fc-a803-519ce7d4d225"
Sep 09 02:30:45.372080 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:45.372053    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xcqxh"
Sep 09 02:30:45.391333 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:45.391305    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xcqxh"
Sep 09 02:30:45.656925 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:45.656892    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-952h4"]
Sep 09 02:30:45.657125 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:45.657045    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-952h4"
Sep 09 02:30:45.657193 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:45.657156    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-952h4" podUID="37ef684a-ce44-43fc-a803-519ce7d4d225"
Sep 09 02:30:45.661009 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:45.659928    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-6gkgn"]
Sep 09 02:30:45.661009 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:45.660045    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zrljs"]
Sep 09 02:30:45.661009 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:45.660182    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zrljs"
Sep 09 02:30:45.661009 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:45.660353    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zrljs" podUID="d786708a-bbb1-4250-a2d3-84dc2299d71a"
Sep 09 02:30:45.661009 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:45.660467    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6gkgn"
Sep 09 02:30:45.661009 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:45.660599    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6gkgn" podUID="ac0a0b68-d43e-4ca8-8c4d-39bbe5e7519c"
Sep 09 02:30:46.374899 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:46.374862    2570 generic.go:356] "Generic (PLEG): container finished" podID="d33ba4b9-3e00-405a-bc2f-9144dbfa5c8f" containerID="b293bc7625b78dbc24a75ed85cfeb8a3dfe5cb66bfaf1bc9212418133a1620d1" exitCode=0
Sep 09 02:30:46.375402 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:46.374943    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zw227" event={"ID":"d33ba4b9-3e00-405a-bc2f-9144dbfa5c8f","Type":"ContainerDied","Data":"b293bc7625b78dbc24a75ed85cfeb8a3dfe5cb66bfaf1bc9212418133a1620d1"}
Sep 09 02:30:47.216713 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:47.216683    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6gkgn"
Sep 09 02:30:47.216713 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:47.216717    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-952h4"
Sep 09 02:30:47.216905 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:47.216683    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zrljs"
Sep 09 02:30:47.216905 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:47.216839    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6gkgn" podUID="ac0a0b68-d43e-4ca8-8c4d-39bbe5e7519c"
Sep 09 02:30:47.216999 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:47.216946    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-952h4" podUID="37ef684a-ce44-43fc-a803-519ce7d4d225"
Sep 09 02:30:47.217048 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:47.217032    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zrljs" podUID="d786708a-bbb1-4250-a2d3-84dc2299d71a"
Sep 09 02:30:47.379174 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:47.379144    2570 generic.go:356] "Generic (PLEG): container finished" podID="d33ba4b9-3e00-405a-bc2f-9144dbfa5c8f" containerID="1e23b6b27214f01cc77c311f73e5e809ce45a1761318f221bad62e33121cf878" exitCode=0
Sep 09 02:30:47.379556 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:47.379223    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zw227" event={"ID":"d33ba4b9-3e00-405a-bc2f-9144dbfa5c8f","Type":"ContainerDied","Data":"1e23b6b27214f01cc77c311f73e5e809ce45a1761318f221bad62e33121cf878"}
Sep 09 02:30:48.833516 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:48.833328    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-kxvb7"
Sep 09 02:30:48.833969 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:48.833667    2570 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 09 02:30:48.834242 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:48.834222    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-kxvb7"
Sep 09 02:30:49.216868 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:49.216787    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zrljs"
Sep 09 02:30:49.216868 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:49.216837    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6gkgn"
Sep 09 02:30:49.217094 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:49.216786    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-952h4"
Sep 09 02:30:49.217094 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:49.216917    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zrljs" podUID="d786708a-bbb1-4250-a2d3-84dc2299d71a"
Sep 09 02:30:49.217094 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:49.217008    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-952h4" podUID="37ef684a-ce44-43fc-a803-519ce7d4d225"
Sep 09 02:30:49.217094 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:49.217062    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6gkgn" podUID="ac0a0b68-d43e-4ca8-8c4d-39bbe5e7519c"
Sep 09 02:30:51.217279 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:51.217237    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6gkgn"
Sep 09 02:30:51.217279 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:51.217269    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-952h4"
Sep 09 02:30:51.217852 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:51.217243    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zrljs"
Sep 09 02:30:51.217852 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:51.217374    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6gkgn" podUID="ac0a0b68-d43e-4ca8-8c4d-39bbe5e7519c"
Sep 09 02:30:51.217852 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:51.217438    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zrljs" podUID="d786708a-bbb1-4250-a2d3-84dc2299d71a"
Sep 09 02:30:51.217852 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:51.217545    2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-952h4" podUID="37ef684a-ce44-43fc-a803-519ce7d4d225"
Sep 09 02:30:51.524960 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:51.524896    2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-188.ec2.internal" event="NodeReady"
Sep 09 02:30:51.525104 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:51.525052    2570 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 09 02:30:51.576519 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:51.576491    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-ttpxf"]
Sep 09 02:30:51.590717 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:51.590691    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-wjvs5"]
Sep 09 02:30:51.590851 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:51.590798    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ttpxf"
Sep 09 02:30:51.594888 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:51.594862    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 09 02:30:51.595037 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:51.594863    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 09 02:30:51.595037 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:51.594925    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-99mnl\""
Sep 09 02:30:51.621431 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:51.621409    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-ttpxf"]
Sep 09 02:30:51.621527 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:51.621442    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-wjvs5"]
Sep 09 02:30:51.621527 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:51.621515    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-wjvs5"
Sep 09 02:30:51.624736 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:51.624722    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 09 02:30:51.624834 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:51.624773    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 09 02:30:51.625165 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:51.625152    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 09 02:30:51.625387 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:51.625375    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-r6wq8\""
Sep 09 02:30:51.720057 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:51.720025    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ec031e2f-88e3-4608-a1b7-de11b4d537a0-cert\") pod \"ingress-canary-wjvs5\" (UID: \"ec031e2f-88e3-4608-a1b7-de11b4d537a0\") " pod="openshift-ingress-canary/ingress-canary-wjvs5"
Sep 09 02:30:51.720196 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:51.720074    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v82j7\" (UniqueName: \"kubernetes.io/projected/47cb49af-ac6a-4f18-ad58-5b40818d5827-kube-api-access-v82j7\") pod \"dns-default-ttpxf\" (UID: \"47cb49af-ac6a-4f18-ad58-5b40818d5827\") " pod="openshift-dns/dns-default-ttpxf"
Sep 09 02:30:51.720196 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:51.720133    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/47cb49af-ac6a-4f18-ad58-5b40818d5827-metrics-tls\") pod \"dns-default-ttpxf\" (UID: \"47cb49af-ac6a-4f18-ad58-5b40818d5827\") " pod="openshift-dns/dns-default-ttpxf"
Sep 09 02:30:51.720269 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:51.720197    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/47cb49af-ac6a-4f18-ad58-5b40818d5827-config-volume\") pod \"dns-default-ttpxf\" (UID: \"47cb49af-ac6a-4f18-ad58-5b40818d5827\") " pod="openshift-dns/dns-default-ttpxf"
Sep 09 02:30:51.720269 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:51.720214    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2g4b9\" (UniqueName: \"kubernetes.io/projected/ec031e2f-88e3-4608-a1b7-de11b4d537a0-kube-api-access-2g4b9\") pod \"ingress-canary-wjvs5\" (UID: \"ec031e2f-88e3-4608-a1b7-de11b4d537a0\") " pod="openshift-ingress-canary/ingress-canary-wjvs5"
Sep 09 02:30:51.720269 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:51.720228    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47cb49af-ac6a-4f18-ad58-5b40818d5827-tmp-dir\") pod \"dns-default-ttpxf\" (UID: \"47cb49af-ac6a-4f18-ad58-5b40818d5827\") " pod="openshift-dns/dns-default-ttpxf"
Sep 09 02:30:51.821044 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:51.821020    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v82j7\" (UniqueName: \"kubernetes.io/projected/47cb49af-ac6a-4f18-ad58-5b40818d5827-kube-api-access-v82j7\") pod \"dns-default-ttpxf\" (UID: \"47cb49af-ac6a-4f18-ad58-5b40818d5827\") " pod="openshift-dns/dns-default-ttpxf"
Sep 09 02:30:51.821139 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:51.821068    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/47cb49af-ac6a-4f18-ad58-5b40818d5827-metrics-tls\") pod \"dns-default-ttpxf\" (UID: \"47cb49af-ac6a-4f18-ad58-5b40818d5827\") " pod="openshift-dns/dns-default-ttpxf"
Sep 09 02:30:51.821139 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:51.821115    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/47cb49af-ac6a-4f18-ad58-5b40818d5827-config-volume\") pod \"dns-default-ttpxf\" (UID: \"47cb49af-ac6a-4f18-ad58-5b40818d5827\") " pod="openshift-dns/dns-default-ttpxf"
Sep 09 02:30:51.821228 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:51.821137    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2g4b9\" (UniqueName: \"kubernetes.io/projected/ec031e2f-88e3-4608-a1b7-de11b4d537a0-kube-api-access-2g4b9\") pod \"ingress-canary-wjvs5\" (UID: \"ec031e2f-88e3-4608-a1b7-de11b4d537a0\") " pod="openshift-ingress-canary/ingress-canary-wjvs5"
Sep 09 02:30:51.821228 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:51.821157    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47cb49af-ac6a-4f18-ad58-5b40818d5827-tmp-dir\") pod \"dns-default-ttpxf\" (UID: \"47cb49af-ac6a-4f18-ad58-5b40818d5827\") " pod="openshift-dns/dns-default-ttpxf"
Sep 09 02:30:51.821228 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:51.821175    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ec031e2f-88e3-4608-a1b7-de11b4d537a0-cert\") pod \"ingress-canary-wjvs5\" (UID: \"ec031e2f-88e3-4608-a1b7-de11b4d537a0\") " pod="openshift-ingress-canary/ingress-canary-wjvs5"
Sep 09 02:30:51.821375 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:51.821246    2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 02:30:51.821375 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:51.821324    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47cb49af-ac6a-4f18-ad58-5b40818d5827-metrics-tls podName:47cb49af-ac6a-4f18-ad58-5b40818d5827 nodeName:}" failed. No retries permitted until 2026-09-09 02:30:52.321302446 +0000 UTC m=+34.575606061 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/47cb49af-ac6a-4f18-ad58-5b40818d5827-metrics-tls") pod "dns-default-ttpxf" (UID: "47cb49af-ac6a-4f18-ad58-5b40818d5827") : secret "dns-default-metrics-tls" not found
Sep 09 02:30:51.821565 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:51.821539    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47cb49af-ac6a-4f18-ad58-5b40818d5827-tmp-dir\") pod \"dns-default-ttpxf\" (UID: \"47cb49af-ac6a-4f18-ad58-5b40818d5827\") " pod="openshift-dns/dns-default-ttpxf"
Sep 09 02:30:51.821729 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:51.821708    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/47cb49af-ac6a-4f18-ad58-5b40818d5827-config-volume\") pod \"dns-default-ttpxf\" (UID: \"47cb49af-ac6a-4f18-ad58-5b40818d5827\") " pod="openshift-dns/dns-default-ttpxf"
Sep 09 02:30:51.825256 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:51.825235    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ec031e2f-88e3-4608-a1b7-de11b4d537a0-cert\") pod \"ingress-canary-wjvs5\" (UID: \"ec031e2f-88e3-4608-a1b7-de11b4d537a0\") " pod="openshift-ingress-canary/ingress-canary-wjvs5"
Sep 09 02:30:51.831605 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:51.831588    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v82j7\" (UniqueName: \"kubernetes.io/projected/47cb49af-ac6a-4f18-ad58-5b40818d5827-kube-api-access-v82j7\") pod \"dns-default-ttpxf\" (UID: \"47cb49af-ac6a-4f18-ad58-5b40818d5827\") " pod="openshift-dns/dns-default-ttpxf"
Sep 09 02:30:51.833146 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:51.833127    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2g4b9\" (UniqueName: \"kubernetes.io/projected/ec031e2f-88e3-4608-a1b7-de11b4d537a0-kube-api-access-2g4b9\") pod \"ingress-canary-wjvs5\" (UID: \"ec031e2f-88e3-4608-a1b7-de11b4d537a0\") " pod="openshift-ingress-canary/ingress-canary-wjvs5"
Sep 09 02:30:51.921860 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:51.921825    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d786708a-bbb1-4250-a2d3-84dc2299d71a-metrics-certs\") pod \"network-metrics-daemon-zrljs\" (UID: \"d786708a-bbb1-4250-a2d3-84dc2299d71a\") " pod="openshift-multus/network-metrics-daemon-zrljs"
Sep 09 02:30:51.922030 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:51.921967    2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 02:30:51.922077 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:51.922048    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d786708a-bbb1-4250-a2d3-84dc2299d71a-metrics-certs podName:d786708a-bbb1-4250-a2d3-84dc2299d71a nodeName:}" failed. No retries permitted until 2026-09-09 02:31:23.92203348 +0000 UTC m=+66.176337093 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d786708a-bbb1-4250-a2d3-84dc2299d71a-metrics-certs") pod "network-metrics-daemon-zrljs" (UID: "d786708a-bbb1-4250-a2d3-84dc2299d71a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 02:30:51.929847 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:51.929823    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-wjvs5"
Sep 09 02:30:52.022300 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:52.022265    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/37ef684a-ce44-43fc-a803-519ce7d4d225-original-pull-secret\") pod \"global-pull-secret-syncer-952h4\" (UID: \"37ef684a-ce44-43fc-a803-519ce7d4d225\") " pod="kube-system/global-pull-secret-syncer-952h4"
Sep 09 02:30:52.022478 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:52.022346    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drrcs\" (UniqueName: \"kubernetes.io/projected/ac0a0b68-d43e-4ca8-8c4d-39bbe5e7519c-kube-api-access-drrcs\") pod \"network-check-target-6gkgn\" (UID: \"ac0a0b68-d43e-4ca8-8c4d-39bbe5e7519c\") " pod="openshift-network-diagnostics/network-check-target-6gkgn"
Sep 09 02:30:52.022478 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:52.022464    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 02:30:52.022478 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:52.022466    2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 02:30:52.022634 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:52.022564    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37ef684a-ce44-43fc-a803-519ce7d4d225-original-pull-secret podName:37ef684a-ce44-43fc-a803-519ce7d4d225 nodeName:}" failed. No retries permitted until 2026-09-09 02:31:24.022541992 +0000 UTC m=+66.276845622 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/37ef684a-ce44-43fc-a803-519ce7d4d225-original-pull-secret") pod "global-pull-secret-syncer-952h4" (UID: "37ef684a-ce44-43fc-a803-519ce7d4d225") : object "kube-system"/"original-pull-secret" not registered
Sep 09 02:30:52.022634 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:52.022478    2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 02:30:52.022738 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:52.022652    2570 projected.go:196] Error preparing data for projected volume kube-api-access-drrcs for pod openshift-network-diagnostics/network-check-target-6gkgn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 02:30:52.022738 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:52.022711    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ac0a0b68-d43e-4ca8-8c4d-39bbe5e7519c-kube-api-access-drrcs podName:ac0a0b68-d43e-4ca8-8c4d-39bbe5e7519c nodeName:}" failed. No retries permitted until 2026-09-09 02:31:24.022697321 +0000 UTC m=+66.277000948 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-drrcs" (UniqueName: "kubernetes.io/projected/ac0a0b68-d43e-4ca8-8c4d-39bbe5e7519c-kube-api-access-drrcs") pod "network-check-target-6gkgn" (UID: "ac0a0b68-d43e-4ca8-8c4d-39bbe5e7519c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 02:30:52.253215 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:52.253185    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-wjvs5"]
Sep 09 02:30:52.256409 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:30:52.256248    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec031e2f_88e3_4608_a1b7_de11b4d537a0.slice/crio-93696cbb1944435aeb40a688f29218abf2cdf446d76fd7e08037489556c3faed WatchSource:0}: Error finding container 93696cbb1944435aeb40a688f29218abf2cdf446d76fd7e08037489556c3faed: Status 404 returned error can't find the container with id 93696cbb1944435aeb40a688f29218abf2cdf446d76fd7e08037489556c3faed
Sep 09 02:30:52.324538 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:52.324511    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/47cb49af-ac6a-4f18-ad58-5b40818d5827-metrics-tls\") pod \"dns-default-ttpxf\" (UID: \"47cb49af-ac6a-4f18-ad58-5b40818d5827\") " pod="openshift-dns/dns-default-ttpxf"
Sep 09 02:30:52.324656 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:52.324627    2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 02:30:52.324692 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:52.324673    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47cb49af-ac6a-4f18-ad58-5b40818d5827-metrics-tls podName:47cb49af-ac6a-4f18-ad58-5b40818d5827 nodeName:}" failed. No retries permitted until 2026-09-09 02:30:53.324661813 +0000 UTC m=+35.578965428 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/47cb49af-ac6a-4f18-ad58-5b40818d5827-metrics-tls") pod "dns-default-ttpxf" (UID: "47cb49af-ac6a-4f18-ad58-5b40818d5827") : secret "dns-default-metrics-tls" not found
Sep 09 02:30:52.394144 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:52.394118    2570 generic.go:356] "Generic (PLEG): container finished" podID="d33ba4b9-3e00-405a-bc2f-9144dbfa5c8f" containerID="32f5852c2dbda3ee35118be13ac8673364988243d2af656ae62f0ccfa1dddce4" exitCode=0
Sep 09 02:30:52.394277 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:52.394205    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zw227" event={"ID":"d33ba4b9-3e00-405a-bc2f-9144dbfa5c8f","Type":"ContainerDied","Data":"32f5852c2dbda3ee35118be13ac8673364988243d2af656ae62f0ccfa1dddce4"}
Sep 09 02:30:52.395456 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:52.395438    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-wjvs5" event={"ID":"ec031e2f-88e3-4608-a1b7-de11b4d537a0","Type":"ContainerStarted","Data":"93696cbb1944435aeb40a688f29218abf2cdf446d76fd7e08037489556c3faed"}
Sep 09 02:30:53.217326 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:53.217296    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6gkgn"
Sep 09 02:30:53.217326 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:53.217331    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-952h4"
Sep 09 02:30:53.217583 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:53.217406    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zrljs"
Sep 09 02:30:53.220284 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:53.220254    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 09 02:30:53.221501 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:53.221480    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-rd54v\""
Sep 09 02:30:53.221761 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:53.221742    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-shl9x\""
Sep 09 02:30:53.221866 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:53.221833    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 09 02:30:53.221931 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:53.221890    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 09 02:30:53.222120 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:53.222105    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 09 02:30:53.334007 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:53.333958    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/47cb49af-ac6a-4f18-ad58-5b40818d5827-metrics-tls\") pod \"dns-default-ttpxf\" (UID: \"47cb49af-ac6a-4f18-ad58-5b40818d5827\") " pod="openshift-dns/dns-default-ttpxf"
Sep 09 02:30:53.334424 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:53.334167    2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 02:30:53.334424 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:53.334236    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47cb49af-ac6a-4f18-ad58-5b40818d5827-metrics-tls podName:47cb49af-ac6a-4f18-ad58-5b40818d5827 nodeName:}" failed. No retries permitted until 2026-09-09 02:30:55.334217073 +0000 UTC m=+37.588520701 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/47cb49af-ac6a-4f18-ad58-5b40818d5827-metrics-tls") pod "dns-default-ttpxf" (UID: "47cb49af-ac6a-4f18-ad58-5b40818d5827") : secret "dns-default-metrics-tls" not found
Sep 09 02:30:53.401935 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:53.401907    2570 generic.go:356] "Generic (PLEG): container finished" podID="d33ba4b9-3e00-405a-bc2f-9144dbfa5c8f" containerID="265c4ef0f4282bd4933790cd930f76c71ce21ef827c2240a24f63f4349f1d937" exitCode=0
Sep 09 02:30:53.402108 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:53.401960    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zw227" event={"ID":"d33ba4b9-3e00-405a-bc2f-9144dbfa5c8f","Type":"ContainerDied","Data":"265c4ef0f4282bd4933790cd930f76c71ce21ef827c2240a24f63f4349f1d937"}
Sep 09 02:30:54.409378 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:54.409338    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zw227" event={"ID":"d33ba4b9-3e00-405a-bc2f-9144dbfa5c8f","Type":"ContainerStarted","Data":"1d6adae412d3c4a9d7c78e89ee8179e5ad88924bf1e64eb53cc007ece35eb7e4"}
Sep 09 02:30:54.436221 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:54.436002    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-zw227" podStartSLOduration=8.254418555 podStartE2EDuration="36.435967088s" totalImagesPullingTime="28.181548533s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 02:30:18 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:30:54.434555329 +0000 UTC m=+36.688858976" watchObservedRunningTime="2026-09-09 02:30:54.435967088 +0000 UTC m=+36.690270726"
Sep 09 02:30:55.348660 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:55.348624    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/47cb49af-ac6a-4f18-ad58-5b40818d5827-metrics-tls\") pod \"dns-default-ttpxf\" (UID: \"47cb49af-ac6a-4f18-ad58-5b40818d5827\") " pod="openshift-dns/dns-default-ttpxf"
Sep 09 02:30:55.348810 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:55.348771    2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 02:30:55.348852 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:55.348840    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47cb49af-ac6a-4f18-ad58-5b40818d5827-metrics-tls podName:47cb49af-ac6a-4f18-ad58-5b40818d5827 nodeName:}" failed. No retries permitted until 2026-09-09 02:30:59.348821779 +0000 UTC m=+41.603125413 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/47cb49af-ac6a-4f18-ad58-5b40818d5827-metrics-tls") pod "dns-default-ttpxf" (UID: "47cb49af-ac6a-4f18-ad58-5b40818d5827") : secret "dns-default-metrics-tls" not found
Sep 09 02:30:55.413318 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:55.413278    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-wjvs5" event={"ID":"ec031e2f-88e3-4608-a1b7-de11b4d537a0","Type":"ContainerStarted","Data":"adc497396f96747eabed601b9c724f70add4161ce52c6c8b6683dff692623fca"}
Sep 09 02:30:55.434816 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:55.434772    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-wjvs5" podStartSLOduration=1.725221669 podStartE2EDuration="4.434758017s" totalImagesPullingTime="2.709536348s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:30:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:30:55.434478947 +0000 UTC m=+37.688782583" watchObservedRunningTime="2026-09-09 02:30:55.434758017 +0000 UTC m=+37.689061687"
Sep 09 02:30:59.374278 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:30:59.374081    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/47cb49af-ac6a-4f18-ad58-5b40818d5827-metrics-tls\") pod \"dns-default-ttpxf\" (UID: \"47cb49af-ac6a-4f18-ad58-5b40818d5827\") " pod="openshift-dns/dns-default-ttpxf"
Sep 09 02:30:59.374673 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:59.374230    2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 02:30:59.374673 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:30:59.374365    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47cb49af-ac6a-4f18-ad58-5b40818d5827-metrics-tls podName:47cb49af-ac6a-4f18-ad58-5b40818d5827 nodeName:}" failed. No retries permitted until 2026-09-09 02:31:07.374350218 +0000 UTC m=+49.628653846 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/47cb49af-ac6a-4f18-ad58-5b40818d5827-metrics-tls") pod "dns-default-ttpxf" (UID: "47cb49af-ac6a-4f18-ad58-5b40818d5827") : secret "dns-default-metrics-tls" not found
Sep 09 02:31:04.119439 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:04.119407    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-5g4th"]
Sep 09 02:31:04.162919 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:04.162891    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-85dqd"]
Sep 09 02:31:04.163093 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:04.163055    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-5g4th"
Sep 09 02:31:04.169654 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:04.169629    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 09 02:31:04.169803 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:04.169683    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 09 02:31:04.172832 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:04.172815    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-5dd9m\""
Sep 09 02:31:04.172934 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:04.172837    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 09 02:31:04.172934 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:04.172897    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 09 02:31:04.191594 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:04.191555    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-5g4th"]
Sep 09 02:31:04.191594 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:04.191594    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-85dqd"]
Sep 09 02:31:04.191757 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:04.191672    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-85dqd"
Sep 09 02:31:04.196707 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:04.196668    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 09 02:31:04.196840 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:04.196724    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 09 02:31:04.196840 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:04.196776    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 09 02:31:04.197098 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:04.197010    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 09 02:31:04.197098 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:04.197049    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-pgbpx\""
Sep 09 02:31:04.204507 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:04.204489    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 09 02:31:04.219572 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:04.219545    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-lp6p9"]
Sep 09 02:31:04.242896 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:04.242871    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-69c66d4b4-pl2k4"]
Sep 09 02:31:04.243069 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:04.243017    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-lp6p9"
Sep 09 02:31:04.250350 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:04.250331    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 09 02:31:04.250713 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:04.250699    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 09 02:31:04.250912 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:04.250896    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 09 02:31:04.250912 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:04.250906    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 09 02:31:04.251032 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:04.250990    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-v7g6p\""
Sep 09 02:31:04.266965 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:04.266945    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-lp6p9"]
Sep 09 02:31:04.267054 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:04.266987    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-69c66d4b4-pl2k4"]
Sep 09 02:31:04.267091 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:04.267059    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-69c66d4b4-pl2k4"
Sep 09 02:31:04.270401 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:04.270386    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 09 02:31:04.270690 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:04.270673    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 09 02:31:04.270770 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:04.270759    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 09 02:31:04.271047 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:04.271014    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 09 02:31:04.271144 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:04.271077    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-42tmd\""
Sep 09 02:31:04.271144 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:04.271095    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 09 02:31:04.271233 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:04.271185    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 09 02:31:04.306622 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:04.306595    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/c5fd6885-5b56-4f0c-b1dd-3cff5153682b-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-5g4th\" (UID: \"c5fd6885-5b56-4f0c-b1dd-3cff5153682b\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-5g4th"
Sep 09 02:31:04.306789 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:04.306631    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/a00f5e8a-987c-4c31-a8a9-b6a9d713d064-snapshots\") pod \"insights-operator-76bcd6ccf5-85dqd\" (UID: \"a00f5e8a-987c-4c31-a8a9-b6a9d713d064\") " pod="openshift-insights/insights-operator-76bcd6ccf5-85dqd"
Sep 09 02:31:04.306789 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:04.306705    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a00f5e8a-987c-4c31-a8a9-b6a9d713d064-serving-cert\") pod \"insights-operator-76bcd6ccf5-85dqd\" (UID: \"a00f5e8a-987c-4c31-a8a9-b6a9d713d064\") " pod="openshift-insights/insights-operator-76bcd6ccf5-85dqd"
Sep 09 02:31:04.306789 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:04.306745    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a00f5e8a-987c-4c31-a8a9-b6a9d713d064-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-85dqd\" (UID: \"a00f5e8a-987c-4c31-a8a9-b6a9d713d064\") " pod="openshift-insights/insights-operator-76bcd6ccf5-85dqd"
Sep 09 02:31:04.306904 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:04.306790    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhx9x\" (UniqueName: \"kubernetes.io/projected/c5fd6885-5b56-4f0c-b1dd-3cff5153682b-kube-api-access-mhx9x\") pod \"cluster-monitoring-operator-6cdcc96965-5g4th\" (UID: \"c5fd6885-5b56-4f0c-b1dd-3cff5153682b\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-5g4th"
Sep 09 02:31:04.306904 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:04.306814    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a00f5e8a-987c-4c31-a8a9-b6a9d713d064-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-85dqd\" (UID: \"a00f5e8a-987c-4c31-a8a9-b6a9d713d064\") " pod="openshift-insights/insights-operator-76bcd6ccf5-85dqd"
Sep 09 02:31:04.306904 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:04.306847    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c5fd6885-5b56-4f0c-b1dd-3cff5153682b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-5g4th\" (UID: \"c5fd6885-5b56-4f0c-b1dd-3cff5153682b\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-5g4th"
Sep 09 02:31:04.306904 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:04.306880    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsmcd\" (UniqueName: \"kubernetes.io/projected/a00f5e8a-987c-4c31-a8a9-b6a9d713d064-kube-api-access-wsmcd\") pod \"insights-operator-76bcd6ccf5-85dqd\" (UID: \"a00f5e8a-987c-4c31-a8a9-b6a9d713d064\") " pod="openshift-insights/insights-operator-76bcd6ccf5-85dqd"
Sep 09 02:31:04.306904 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:04.306900    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a00f5e8a-987c-4c31-a8a9-b6a9d713d064-tmp\") pod \"insights-operator-76bcd6ccf5-85dqd\" (UID: \"a00f5e8a-987c-4c31-a8a9-b6a9d713d064\") " pod="openshift-insights/insights-operator-76bcd6ccf5-85dqd"
Sep 09 02:31:04.407798 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:04.407735    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhx9x\" (UniqueName: \"kubernetes.io/projected/c5fd6885-5b56-4f0c-b1dd-3cff5153682b-kube-api-access-mhx9x\") pod \"cluster-monitoring-operator-6cdcc96965-5g4th\" (UID: \"c5fd6885-5b56-4f0c-b1dd-3cff5153682b\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-5g4th"
Sep 09 02:31:04.407798 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:04.407767    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a00f5e8a-987c-4c31-a8a9-b6a9d713d064-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-85dqd\" (UID: \"a00f5e8a-987c-4c31-a8a9-b6a9d713d064\") " pod="openshift-insights/insights-operator-76bcd6ccf5-85dqd"
Sep 09 02:31:04.407798 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:04.407791    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/d5dad8e0-d4e8-4325-a274-a134fc6959a6-stats-auth\") pod \"router-default-69c66d4b4-pl2k4\" (UID: \"d5dad8e0-d4e8-4325-a274-a134fc6959a6\") " pod="openshift-ingress/router-default-69c66d4b4-pl2k4"
Sep 09 02:31:04.408060 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:04.407823    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c5fd6885-5b56-4f0c-b1dd-3cff5153682b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-5g4th\" (UID: \"c5fd6885-5b56-4f0c-b1dd-3cff5153682b\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-5g4th"
Sep 09 02:31:04.408060 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:04.407849    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qgs7\" (UniqueName: \"kubernetes.io/projected/d5dad8e0-d4e8-4325-a274-a134fc6959a6-kube-api-access-7qgs7\") pod \"router-default-69c66d4b4-pl2k4\" (UID: \"d5dad8e0-d4e8-4325-a274-a134fc6959a6\") " pod="openshift-ingress/router-default-69c66d4b4-pl2k4"
Sep 09 02:31:04.408060 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:04.407880    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsmcd\" (UniqueName: \"kubernetes.io/projected/a00f5e8a-987c-4c31-a8a9-b6a9d713d064-kube-api-access-wsmcd\") pod \"insights-operator-76bcd6ccf5-85dqd\" (UID: \"a00f5e8a-987c-4c31-a8a9-b6a9d713d064\") " pod="openshift-insights/insights-operator-76bcd6ccf5-85dqd"
Sep 09 02:31:04.408060 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:04.407907    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/d5dad8e0-d4e8-4325-a274-a134fc6959a6-default-certificate\") pod \"router-default-69c66d4b4-pl2k4\" (UID: \"d5dad8e0-d4e8-4325-a274-a134fc6959a6\") " pod="openshift-ingress/router-default-69c66d4b4-pl2k4"
Sep 09 02:31:04.408060 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:04.407969    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5dad8e0-d4e8-4325-a274-a134fc6959a6-service-ca-bundle\") pod \"router-default-69c66d4b4-pl2k4\" (UID: \"d5dad8e0-d4e8-4325-a274-a134fc6959a6\") " pod="openshift-ingress/router-default-69c66d4b4-pl2k4"
Sep 09 02:31:04.408060 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:31:04.408004    2570 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 02:31:04.408304 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:04.408090    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a00f5e8a-987c-4c31-a8a9-b6a9d713d064-tmp\") pod \"insights-operator-76bcd6ccf5-85dqd\" (UID: \"a00f5e8a-987c-4c31-a8a9-b6a9d713d064\") " pod="openshift-insights/insights-operator-76bcd6ccf5-85dqd"
Sep 09 02:31:04.408304 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:31:04.408114    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5fd6885-5b56-4f0c-b1dd-3cff5153682b-cluster-monitoring-operator-tls podName:c5fd6885-5b56-4f0c-b1dd-3cff5153682b nodeName:}" failed. No retries permitted until 2026-09-09 02:31:04.908091218 +0000 UTC m=+47.162394848 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/c5fd6885-5b56-4f0c-b1dd-3cff5153682b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-5g4th" (UID: "c5fd6885-5b56-4f0c-b1dd-3cff5153682b") : secret "cluster-monitoring-operator-tls" not found
Sep 09 02:31:04.408304 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:04.408146    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5dad8e0-d4e8-4325-a274-a134fc6959a6-metrics-certs\") pod \"router-default-69c66d4b4-pl2k4\" (UID: \"d5dad8e0-d4e8-4325-a274-a134fc6959a6\") " pod="openshift-ingress/router-default-69c66d4b4-pl2k4"
Sep 09 02:31:04.408304 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:04.408213    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5a719fd4-b0a2-4fd8-9c0c-ac89d3f340dc-config\") pod \"service-ca-operator-859f94c4b-lp6p9\" (UID: \"5a719fd4-b0a2-4fd8-9c0c-ac89d3f340dc\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-lp6p9"
Sep 09 02:31:04.408304 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:04.408244    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xs7th\" (UniqueName: \"kubernetes.io/projected/5a719fd4-b0a2-4fd8-9c0c-ac89d3f340dc-kube-api-access-xs7th\") pod \"service-ca-operator-859f94c4b-lp6p9\" (UID: \"5a719fd4-b0a2-4fd8-9c0c-ac89d3f340dc\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-lp6p9"
Sep 09 02:31:04.408304 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:04.408286    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/c5fd6885-5b56-4f0c-b1dd-3cff5153682b-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-5g4th\" (UID: \"c5fd6885-5b56-4f0c-b1dd-3cff5153682b\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-5g4th"
Sep 09 02:31:04.408529 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:04.408305    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/a00f5e8a-987c-4c31-a8a9-b6a9d713d064-snapshots\") pod \"insights-operator-76bcd6ccf5-85dqd\" (UID: \"a00f5e8a-987c-4c31-a8a9-b6a9d713d064\") " pod="openshift-insights/insights-operator-76bcd6ccf5-85dqd"
Sep 09 02:31:04.408529 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:04.408362    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5a719fd4-b0a2-4fd8-9c0c-ac89d3f340dc-serving-cert\") pod \"service-ca-operator-859f94c4b-lp6p9\" (UID: \"5a719fd4-b0a2-4fd8-9c0c-ac89d3f340dc\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-lp6p9"
Sep 09 02:31:04.408529 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:04.408391    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a00f5e8a-987c-4c31-a8a9-b6a9d713d064-tmp\") pod \"insights-operator-76bcd6ccf5-85dqd\" (UID: \"a00f5e8a-987c-4c31-a8a9-b6a9d713d064\") " pod="openshift-insights/insights-operator-76bcd6ccf5-85dqd"
Sep 09 02:31:04.408529 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:04.408430    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a00f5e8a-987c-4c31-a8a9-b6a9d713d064-serving-cert\") pod \"insights-operator-76bcd6ccf5-85dqd\" (UID: \"a00f5e8a-987c-4c31-a8a9-b6a9d713d064\") " pod="openshift-insights/insights-operator-76bcd6ccf5-85dqd"
Sep 09 02:31:04.408529 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:04.408468    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a00f5e8a-987c-4c31-a8a9-b6a9d713d064-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-85dqd\" (UID: \"a00f5e8a-987c-4c31-a8a9-b6a9d713d064\") " pod="openshift-insights/insights-operator-76bcd6ccf5-85dqd"
Sep 09 02:31:04.408762 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:04.408676    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a00f5e8a-987c-4c31-a8a9-b6a9d713d064-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-85dqd\" (UID: \"a00f5e8a-987c-4c31-a8a9-b6a9d713d064\") " pod="openshift-insights/insights-operator-76bcd6ccf5-85dqd"
Sep 09 02:31:04.408860 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:04.408838    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/a00f5e8a-987c-4c31-a8a9-b6a9d713d064-snapshots\") pod \"insights-operator-76bcd6ccf5-85dqd\" (UID: \"a00f5e8a-987c-4c31-a8a9-b6a9d713d064\") " pod="openshift-insights/insights-operator-76bcd6ccf5-85dqd"
Sep 09 02:31:04.408917 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:04.408869    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/c5fd6885-5b56-4f0c-b1dd-3cff5153682b-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-5g4th\" (UID: \"c5fd6885-5b56-4f0c-b1dd-3cff5153682b\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-5g4th"
Sep 09 02:31:04.408917 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:04.408911    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a00f5e8a-987c-4c31-a8a9-b6a9d713d064-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-85dqd\" (UID: \"a00f5e8a-987c-4c31-a8a9-b6a9d713d064\") " pod="openshift-insights/insights-operator-76bcd6ccf5-85dqd"
Sep 09 02:31:04.412064 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:04.412041    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a00f5e8a-987c-4c31-a8a9-b6a9d713d064-serving-cert\") pod \"insights-operator-76bcd6ccf5-85dqd\" (UID: \"a00f5e8a-987c-4c31-a8a9-b6a9d713d064\") " pod="openshift-insights/insights-operator-76bcd6ccf5-85dqd"
Sep 09 02:31:04.447470 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:04.447446    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhx9x\" (UniqueName: \"kubernetes.io/projected/c5fd6885-5b56-4f0c-b1dd-3cff5153682b-kube-api-access-mhx9x\") pod \"cluster-monitoring-operator-6cdcc96965-5g4th\" (UID: \"c5fd6885-5b56-4f0c-b1dd-3cff5153682b\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-5g4th"
Sep 09 02:31:04.471729 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:04.471707    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsmcd\" (UniqueName: \"kubernetes.io/projected/a00f5e8a-987c-4c31-a8a9-b6a9d713d064-kube-api-access-wsmcd\") pod \"insights-operator-76bcd6ccf5-85dqd\" (UID: \"a00f5e8a-987c-4c31-a8a9-b6a9d713d064\") " pod="openshift-insights/insights-operator-76bcd6ccf5-85dqd"
Sep 09 02:31:04.499695 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:04.499676    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-85dqd"
Sep 09 02:31:04.509102 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:04.509081    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5dad8e0-d4e8-4325-a274-a134fc6959a6-service-ca-bundle\") pod \"router-default-69c66d4b4-pl2k4\" (UID: \"d5dad8e0-d4e8-4325-a274-a134fc6959a6\") " pod="openshift-ingress/router-default-69c66d4b4-pl2k4"
Sep 09 02:31:04.509171 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:04.509109    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5dad8e0-d4e8-4325-a274-a134fc6959a6-metrics-certs\") pod \"router-default-69c66d4b4-pl2k4\" (UID: \"d5dad8e0-d4e8-4325-a274-a134fc6959a6\") " pod="openshift-ingress/router-default-69c66d4b4-pl2k4"
Sep 09 02:31:04.509171 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:04.509129    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5a719fd4-b0a2-4fd8-9c0c-ac89d3f340dc-config\") pod \"service-ca-operator-859f94c4b-lp6p9\" (UID: \"5a719fd4-b0a2-4fd8-9c0c-ac89d3f340dc\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-lp6p9"
Sep 09 02:31:04.509171 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:04.509145    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xs7th\" (UniqueName: \"kubernetes.io/projected/5a719fd4-b0a2-4fd8-9c0c-ac89d3f340dc-kube-api-access-xs7th\") pod \"service-ca-operator-859f94c4b-lp6p9\" (UID: \"5a719fd4-b0a2-4fd8-9c0c-ac89d3f340dc\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-lp6p9"
Sep 09 02:31:04.509310 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:04.509194    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5a719fd4-b0a2-4fd8-9c0c-ac89d3f340dc-serving-cert\") pod \"service-ca-operator-859f94c4b-lp6p9\" (UID: \"5a719fd4-b0a2-4fd8-9c0c-ac89d3f340dc\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-lp6p9"
Sep 09 02:31:04.509310 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:31:04.509231    2570 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 09 02:31:04.509310 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:31:04.509250    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d5dad8e0-d4e8-4325-a274-a134fc6959a6-service-ca-bundle podName:d5dad8e0-d4e8-4325-a274-a134fc6959a6 nodeName:}" failed. No retries permitted until 2026-09-09 02:31:05.009236043 +0000 UTC m=+47.263539679 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d5dad8e0-d4e8-4325-a274-a134fc6959a6-service-ca-bundle") pod "router-default-69c66d4b4-pl2k4" (UID: "d5dad8e0-d4e8-4325-a274-a134fc6959a6") : configmap references non-existent config key: service-ca.crt
Sep 09 02:31:04.509310 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:31:04.509278    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5dad8e0-d4e8-4325-a274-a134fc6959a6-metrics-certs podName:d5dad8e0-d4e8-4325-a274-a134fc6959a6 nodeName:}" failed. No retries permitted until 2026-09-09 02:31:05.009263569 +0000 UTC m=+47.263567202 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d5dad8e0-d4e8-4325-a274-a134fc6959a6-metrics-certs") pod "router-default-69c66d4b4-pl2k4" (UID: "d5dad8e0-d4e8-4325-a274-a134fc6959a6") : secret "router-metrics-certs-default" not found
Sep 09 02:31:04.509477 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:04.509332    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/d5dad8e0-d4e8-4325-a274-a134fc6959a6-stats-auth\") pod \"router-default-69c66d4b4-pl2k4\" (UID: \"d5dad8e0-d4e8-4325-a274-a134fc6959a6\") " pod="openshift-ingress/router-default-69c66d4b4-pl2k4"
Sep 09 02:31:04.509477 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:04.509369    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qgs7\" (UniqueName: \"kubernetes.io/projected/d5dad8e0-d4e8-4325-a274-a134fc6959a6-kube-api-access-7qgs7\") pod \"router-default-69c66d4b4-pl2k4\" (UID: \"d5dad8e0-d4e8-4325-a274-a134fc6959a6\") " pod="openshift-ingress/router-default-69c66d4b4-pl2k4"
Sep 09 02:31:04.509477 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:04.509400    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/d5dad8e0-d4e8-4325-a274-a134fc6959a6-default-certificate\") pod \"router-default-69c66d4b4-pl2k4\" (UID: \"d5dad8e0-d4e8-4325-a274-a134fc6959a6\") " pod="openshift-ingress/router-default-69c66d4b4-pl2k4"
Sep 09 02:31:04.509718 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:04.509687    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5a719fd4-b0a2-4fd8-9c0c-ac89d3f340dc-config\") pod \"service-ca-operator-859f94c4b-lp6p9\" (UID: \"5a719fd4-b0a2-4fd8-9c0c-ac89d3f340dc\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-lp6p9"
Sep 09 02:31:04.511254 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:04.511232    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5a719fd4-b0a2-4fd8-9c0c-ac89d3f340dc-serving-cert\") pod \"service-ca-operator-859f94c4b-lp6p9\" (UID: \"5a719fd4-b0a2-4fd8-9c0c-ac89d3f340dc\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-lp6p9"
Sep 09 02:31:04.511526 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:04.511508    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/d5dad8e0-d4e8-4325-a274-a134fc6959a6-default-certificate\") pod \"router-default-69c66d4b4-pl2k4\" (UID: \"d5dad8e0-d4e8-4325-a274-a134fc6959a6\") " pod="openshift-ingress/router-default-69c66d4b4-pl2k4"
Sep 09 02:31:04.511683 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:04.511666    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/d5dad8e0-d4e8-4325-a274-a134fc6959a6-stats-auth\") pod \"router-default-69c66d4b4-pl2k4\" (UID: \"d5dad8e0-d4e8-4325-a274-a134fc6959a6\") " pod="openshift-ingress/router-default-69c66d4b4-pl2k4"
Sep 09 02:31:04.535419 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:04.535391    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xs7th\" (UniqueName: \"kubernetes.io/projected/5a719fd4-b0a2-4fd8-9c0c-ac89d3f340dc-kube-api-access-xs7th\") pod \"service-ca-operator-859f94c4b-lp6p9\" (UID: \"5a719fd4-b0a2-4fd8-9c0c-ac89d3f340dc\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-lp6p9"
Sep 09 02:31:04.535556 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:04.535540    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qgs7\" (UniqueName: \"kubernetes.io/projected/d5dad8e0-d4e8-4325-a274-a134fc6959a6-kube-api-access-7qgs7\") pod \"router-default-69c66d4b4-pl2k4\" (UID: \"d5dad8e0-d4e8-4325-a274-a134fc6959a6\") " pod="openshift-ingress/router-default-69c66d4b4-pl2k4"
Sep 09 02:31:04.551577 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:04.551549    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-lp6p9"
Sep 09 02:31:04.814231 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:04.814153    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-85dqd"]
Sep 09 02:31:04.818399 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:31:04.818375    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda00f5e8a_987c_4c31_a8a9_b6a9d713d064.slice/crio-8212f8eb5b8a676d8b851b54ceb49ea327f2db3ab48fd5f2f69fb33e4ebd41e0 WatchSource:0}: Error finding container 8212f8eb5b8a676d8b851b54ceb49ea327f2db3ab48fd5f2f69fb33e4ebd41e0: Status 404 returned error can't find the container with id 8212f8eb5b8a676d8b851b54ceb49ea327f2db3ab48fd5f2f69fb33e4ebd41e0
Sep 09 02:31:04.861482 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:04.861460    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-lp6p9"]
Sep 09 02:31:04.865078 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:31:04.865056    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a719fd4_b0a2_4fd8_9c0c_ac89d3f340dc.slice/crio-9ad3dc4004dda0d51e7c4a5ad4a3bb887b1562f72c0b997f7854100d9aa57bcf WatchSource:0}: Error finding container 9ad3dc4004dda0d51e7c4a5ad4a3bb887b1562f72c0b997f7854100d9aa57bcf: Status 404 returned error can't find the container with id 9ad3dc4004dda0d51e7c4a5ad4a3bb887b1562f72c0b997f7854100d9aa57bcf
Sep 09 02:31:04.912012 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:04.911965    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c5fd6885-5b56-4f0c-b1dd-3cff5153682b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-5g4th\" (UID: \"c5fd6885-5b56-4f0c-b1dd-3cff5153682b\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-5g4th"
Sep 09 02:31:04.912110 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:31:04.912095    2570 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 02:31:04.912169 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:31:04.912159    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5fd6885-5b56-4f0c-b1dd-3cff5153682b-cluster-monitoring-operator-tls podName:c5fd6885-5b56-4f0c-b1dd-3cff5153682b nodeName:}" failed. No retries permitted until 2026-09-09 02:31:05.91213816 +0000 UTC m=+48.166441792 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/c5fd6885-5b56-4f0c-b1dd-3cff5153682b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-5g4th" (UID: "c5fd6885-5b56-4f0c-b1dd-3cff5153682b") : secret "cluster-monitoring-operator-tls" not found
Sep 09 02:31:05.013199 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:05.013176    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5dad8e0-d4e8-4325-a274-a134fc6959a6-service-ca-bundle\") pod \"router-default-69c66d4b4-pl2k4\" (UID: \"d5dad8e0-d4e8-4325-a274-a134fc6959a6\") " pod="openshift-ingress/router-default-69c66d4b4-pl2k4"
Sep 09 02:31:05.013286 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:05.013208    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5dad8e0-d4e8-4325-a274-a134fc6959a6-metrics-certs\") pod \"router-default-69c66d4b4-pl2k4\" (UID: \"d5dad8e0-d4e8-4325-a274-a134fc6959a6\") " pod="openshift-ingress/router-default-69c66d4b4-pl2k4"
Sep 09 02:31:05.013332 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:31:05.013319    2570 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 09 02:31:05.013371 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:31:05.013323    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d5dad8e0-d4e8-4325-a274-a134fc6959a6-service-ca-bundle podName:d5dad8e0-d4e8-4325-a274-a134fc6959a6 nodeName:}" failed. No retries permitted until 2026-09-09 02:31:06.013306433 +0000 UTC m=+48.267610062 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d5dad8e0-d4e8-4325-a274-a134fc6959a6-service-ca-bundle") pod "router-default-69c66d4b4-pl2k4" (UID: "d5dad8e0-d4e8-4325-a274-a134fc6959a6") : configmap references non-existent config key: service-ca.crt
Sep 09 02:31:05.013371 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:31:05.013363    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5dad8e0-d4e8-4325-a274-a134fc6959a6-metrics-certs podName:d5dad8e0-d4e8-4325-a274-a134fc6959a6 nodeName:}" failed. No retries permitted until 2026-09-09 02:31:06.01335272 +0000 UTC m=+48.267656335 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d5dad8e0-d4e8-4325-a274-a134fc6959a6-metrics-certs") pod "router-default-69c66d4b4-pl2k4" (UID: "d5dad8e0-d4e8-4325-a274-a134fc6959a6") : secret "router-metrics-certs-default" not found
Sep 09 02:31:05.436101 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:05.436062    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-lp6p9" event={"ID":"5a719fd4-b0a2-4fd8-9c0c-ac89d3f340dc","Type":"ContainerStarted","Data":"9ad3dc4004dda0d51e7c4a5ad4a3bb887b1562f72c0b997f7854100d9aa57bcf"}
Sep 09 02:31:05.437185 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:05.437163    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-85dqd" event={"ID":"a00f5e8a-987c-4c31-a8a9-b6a9d713d064","Type":"ContainerStarted","Data":"8212f8eb5b8a676d8b851b54ceb49ea327f2db3ab48fd5f2f69fb33e4ebd41e0"}
Sep 09 02:31:05.920617 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:05.920578    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c5fd6885-5b56-4f0c-b1dd-3cff5153682b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-5g4th\" (UID: \"c5fd6885-5b56-4f0c-b1dd-3cff5153682b\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-5g4th"
Sep 09 02:31:05.920797 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:31:05.920758    2570 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 02:31:05.920870 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:31:05.920858    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5fd6885-5b56-4f0c-b1dd-3cff5153682b-cluster-monitoring-operator-tls podName:c5fd6885-5b56-4f0c-b1dd-3cff5153682b nodeName:}" failed. No retries permitted until 2026-09-09 02:31:07.920836338 +0000 UTC m=+50.175139974 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/c5fd6885-5b56-4f0c-b1dd-3cff5153682b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-5g4th" (UID: "c5fd6885-5b56-4f0c-b1dd-3cff5153682b") : secret "cluster-monitoring-operator-tls" not found
Sep 09 02:31:06.021709 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:06.021679    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5dad8e0-d4e8-4325-a274-a134fc6959a6-service-ca-bundle\") pod \"router-default-69c66d4b4-pl2k4\" (UID: \"d5dad8e0-d4e8-4325-a274-a134fc6959a6\") " pod="openshift-ingress/router-default-69c66d4b4-pl2k4"
Sep 09 02:31:06.021876 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:06.021717    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5dad8e0-d4e8-4325-a274-a134fc6959a6-metrics-certs\") pod \"router-default-69c66d4b4-pl2k4\" (UID: \"d5dad8e0-d4e8-4325-a274-a134fc6959a6\") " pod="openshift-ingress/router-default-69c66d4b4-pl2k4"
Sep 09 02:31:06.021876 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:31:06.021865    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d5dad8e0-d4e8-4325-a274-a134fc6959a6-service-ca-bundle podName:d5dad8e0-d4e8-4325-a274-a134fc6959a6 nodeName:}" failed. No retries permitted until 2026-09-09 02:31:08.021843273 +0000 UTC m=+50.276146899 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d5dad8e0-d4e8-4325-a274-a134fc6959a6-service-ca-bundle") pod "router-default-69c66d4b4-pl2k4" (UID: "d5dad8e0-d4e8-4325-a274-a134fc6959a6") : configmap references non-existent config key: service-ca.crt
Sep 09 02:31:06.022029 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:31:06.021913    2570 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 09 02:31:06.022029 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:31:06.021962    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5dad8e0-d4e8-4325-a274-a134fc6959a6-metrics-certs podName:d5dad8e0-d4e8-4325-a274-a134fc6959a6 nodeName:}" failed. No retries permitted until 2026-09-09 02:31:08.021948325 +0000 UTC m=+50.276251953 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d5dad8e0-d4e8-4325-a274-a134fc6959a6-metrics-certs") pod "router-default-69c66d4b4-pl2k4" (UID: "d5dad8e0-d4e8-4325-a274-a134fc6959a6") : secret "router-metrics-certs-default" not found
Sep 09 02:31:07.433269 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:07.433233    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/47cb49af-ac6a-4f18-ad58-5b40818d5827-metrics-tls\") pod \"dns-default-ttpxf\" (UID: \"47cb49af-ac6a-4f18-ad58-5b40818d5827\") " pod="openshift-dns/dns-default-ttpxf"
Sep 09 02:31:07.433599 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:31:07.433342    2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 02:31:07.433599 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:31:07.433392    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47cb49af-ac6a-4f18-ad58-5b40818d5827-metrics-tls podName:47cb49af-ac6a-4f18-ad58-5b40818d5827 nodeName:}" failed. No retries permitted until 2026-09-09 02:31:23.433379705 +0000 UTC m=+65.687683331 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/47cb49af-ac6a-4f18-ad58-5b40818d5827-metrics-tls") pod "dns-default-ttpxf" (UID: "47cb49af-ac6a-4f18-ad58-5b40818d5827") : secret "dns-default-metrics-tls" not found
Sep 09 02:31:07.938012 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:07.937963    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c5fd6885-5b56-4f0c-b1dd-3cff5153682b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-5g4th\" (UID: \"c5fd6885-5b56-4f0c-b1dd-3cff5153682b\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-5g4th"
Sep 09 02:31:07.938171 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:31:07.938107    2570 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 02:31:07.938224 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:31:07.938184    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5fd6885-5b56-4f0c-b1dd-3cff5153682b-cluster-monitoring-operator-tls podName:c5fd6885-5b56-4f0c-b1dd-3cff5153682b nodeName:}" failed. No retries permitted until 2026-09-09 02:31:11.938164152 +0000 UTC m=+54.192467789 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/c5fd6885-5b56-4f0c-b1dd-3cff5153682b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-5g4th" (UID: "c5fd6885-5b56-4f0c-b1dd-3cff5153682b") : secret "cluster-monitoring-operator-tls" not found
Sep 09 02:31:08.039357 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:08.039269    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5dad8e0-d4e8-4325-a274-a134fc6959a6-service-ca-bundle\") pod \"router-default-69c66d4b4-pl2k4\" (UID: \"d5dad8e0-d4e8-4325-a274-a134fc6959a6\") " pod="openshift-ingress/router-default-69c66d4b4-pl2k4"
Sep 09 02:31:08.039357 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:08.039321    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5dad8e0-d4e8-4325-a274-a134fc6959a6-metrics-certs\") pod \"router-default-69c66d4b4-pl2k4\" (UID: \"d5dad8e0-d4e8-4325-a274-a134fc6959a6\") " pod="openshift-ingress/router-default-69c66d4b4-pl2k4"
Sep 09 02:31:08.039533 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:31:08.039428    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d5dad8e0-d4e8-4325-a274-a134fc6959a6-service-ca-bundle podName:d5dad8e0-d4e8-4325-a274-a134fc6959a6 nodeName:}" failed. No retries permitted until 2026-09-09 02:31:12.039411605 +0000 UTC m=+54.293715233 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d5dad8e0-d4e8-4325-a274-a134fc6959a6-service-ca-bundle") pod "router-default-69c66d4b4-pl2k4" (UID: "d5dad8e0-d4e8-4325-a274-a134fc6959a6") : configmap references non-existent config key: service-ca.crt
Sep 09 02:31:08.039533 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:31:08.039472    2570 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 09 02:31:08.039533 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:31:08.039526    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5dad8e0-d4e8-4325-a274-a134fc6959a6-metrics-certs podName:d5dad8e0-d4e8-4325-a274-a134fc6959a6 nodeName:}" failed. No retries permitted until 2026-09-09 02:31:12.03951178 +0000 UTC m=+54.293815400 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d5dad8e0-d4e8-4325-a274-a134fc6959a6-metrics-certs") pod "router-default-69c66d4b4-pl2k4" (UID: "d5dad8e0-d4e8-4325-a274-a134fc6959a6") : secret "router-metrics-certs-default" not found
Sep 09 02:31:08.445508 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:08.445473    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-lp6p9" event={"ID":"5a719fd4-b0a2-4fd8-9c0c-ac89d3f340dc","Type":"ContainerStarted","Data":"2e35c5f282af737f39d62b57ade0154823e79bd6f4bfc18c084f8214f8939407"}
Sep 09 02:31:08.447064 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:08.447040    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-85dqd" event={"ID":"a00f5e8a-987c-4c31-a8a9-b6a9d713d064","Type":"ContainerStarted","Data":"0e145cb0d80b764b4ba8370225882eacd153809cfbf6e321ea59a732d2bc0402"}
Sep 09 02:31:08.469709 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:08.469662    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-lp6p9" podStartSLOduration=1.561296631 podStartE2EDuration="4.469646858s" totalImagesPullingTime="2.908350227s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:31:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:31:08.465215905 +0000 UTC m=+50.719519545" watchObservedRunningTime="2026-09-09 02:31:08.469646858 +0000 UTC m=+50.723950495"
Sep 09 02:31:08.937021 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:08.936950    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-76bcd6ccf5-85dqd" podStartSLOduration=1.987890895 podStartE2EDuration="4.936931369s" totalImagesPullingTime="2.949040474s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:31:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:31:08.488809519 +0000 UTC m=+50.743113155" watchObservedRunningTime="2026-09-09 02:31:08.936931369 +0000 UTC m=+51.191235005"
Sep 09 02:31:08.937176 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:08.937141    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-jqjd8"]
Sep 09 02:31:08.955141 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:08.955109    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jqjd8"
Sep 09 02:31:08.964005 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:08.963969    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 09 02:31:08.964106 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:08.964026    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 09 02:31:08.964106 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:08.964048    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-68fdc\""
Sep 09 02:31:08.977970 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:08.977949    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jqjd8"]
Sep 09 02:31:09.046769 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:09.046741    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6jkk\" (UniqueName: \"kubernetes.io/projected/175f5821-f3e7-44b4-8f45-945ab415b1c3-kube-api-access-w6jkk\") pod \"insights-runtime-extractor-jqjd8\" (UID: \"175f5821-f3e7-44b4-8f45-945ab415b1c3\") " pod="openshift-insights/insights-runtime-extractor-jqjd8"
Sep 09 02:31:09.046899 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:09.046805    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/175f5821-f3e7-44b4-8f45-945ab415b1c3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jqjd8\" (UID: \"175f5821-f3e7-44b4-8f45-945ab415b1c3\") " pod="openshift-insights/insights-runtime-extractor-jqjd8"
Sep 09 02:31:09.046899 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:09.046863    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/175f5821-f3e7-44b4-8f45-945ab415b1c3-crio-socket\") pod \"insights-runtime-extractor-jqjd8\" (UID: \"175f5821-f3e7-44b4-8f45-945ab415b1c3\") " pod="openshift-insights/insights-runtime-extractor-jqjd8"
Sep 09 02:31:09.046991 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:09.046913    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/175f5821-f3e7-44b4-8f45-945ab415b1c3-data-volume\") pod \"insights-runtime-extractor-jqjd8\" (UID: \"175f5821-f3e7-44b4-8f45-945ab415b1c3\") " pod="openshift-insights/insights-runtime-extractor-jqjd8"
Sep 09 02:31:09.046991 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:09.046954    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/175f5821-f3e7-44b4-8f45-945ab415b1c3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jqjd8\" (UID: \"175f5821-f3e7-44b4-8f45-945ab415b1c3\") " pod="openshift-insights/insights-runtime-extractor-jqjd8"
Sep 09 02:31:09.147506 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:09.147472    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/175f5821-f3e7-44b4-8f45-945ab415b1c3-data-volume\") pod \"insights-runtime-extractor-jqjd8\" (UID: \"175f5821-f3e7-44b4-8f45-945ab415b1c3\") " pod="openshift-insights/insights-runtime-extractor-jqjd8"
Sep 09 02:31:09.147616 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:09.147528    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/175f5821-f3e7-44b4-8f45-945ab415b1c3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jqjd8\" (UID: \"175f5821-f3e7-44b4-8f45-945ab415b1c3\") " pod="openshift-insights/insights-runtime-extractor-jqjd8"
Sep 09 02:31:09.147616 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:31:09.147599    2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 02:31:09.147708 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:09.147649    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6jkk\" (UniqueName: \"kubernetes.io/projected/175f5821-f3e7-44b4-8f45-945ab415b1c3-kube-api-access-w6jkk\") pod \"insights-runtime-extractor-jqjd8\" (UID: \"175f5821-f3e7-44b4-8f45-945ab415b1c3\") " pod="openshift-insights/insights-runtime-extractor-jqjd8"
Sep 09 02:31:09.147708 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:31:09.147671    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/175f5821-f3e7-44b4-8f45-945ab415b1c3-kube-rbac-proxy-cm podName:175f5821-f3e7-44b4-8f45-945ab415b1c3 nodeName:}" failed. No retries permitted until 2026-09-09 02:31:09.647651056 +0000 UTC m=+51.901954685 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/175f5821-f3e7-44b4-8f45-945ab415b1c3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jqjd8" (UID: "175f5821-f3e7-44b4-8f45-945ab415b1c3") : configmap "kube-rbac-proxy" not found
Sep 09 02:31:09.147795 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:09.147735    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/175f5821-f3e7-44b4-8f45-945ab415b1c3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jqjd8\" (UID: \"175f5821-f3e7-44b4-8f45-945ab415b1c3\") " pod="openshift-insights/insights-runtime-extractor-jqjd8"
Sep 09 02:31:09.147795 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:09.147770    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/175f5821-f3e7-44b4-8f45-945ab415b1c3-crio-socket\") pod \"insights-runtime-extractor-jqjd8\" (UID: \"175f5821-f3e7-44b4-8f45-945ab415b1c3\") " pod="openshift-insights/insights-runtime-extractor-jqjd8"
Sep 09 02:31:09.147910 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:31:09.147878    2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 02:31:09.147991 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:09.147921    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/175f5821-f3e7-44b4-8f45-945ab415b1c3-data-volume\") pod \"insights-runtime-extractor-jqjd8\" (UID: \"175f5821-f3e7-44b4-8f45-945ab415b1c3\") " pod="openshift-insights/insights-runtime-extractor-jqjd8"
Sep 09 02:31:09.147991 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:31:09.147959    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/175f5821-f3e7-44b4-8f45-945ab415b1c3-insights-runtime-extractor-tls podName:175f5821-f3e7-44b4-8f45-945ab415b1c3 nodeName:}" failed. No retries permitted until 2026-09-09 02:31:09.647939873 +0000 UTC m=+51.902243507 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/175f5821-f3e7-44b4-8f45-945ab415b1c3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jqjd8" (UID: "175f5821-f3e7-44b4-8f45-945ab415b1c3") : secret "insights-runtime-extractor-tls" not found
Sep 09 02:31:09.148061 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:09.147968    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/175f5821-f3e7-44b4-8f45-945ab415b1c3-crio-socket\") pod \"insights-runtime-extractor-jqjd8\" (UID: \"175f5821-f3e7-44b4-8f45-945ab415b1c3\") " pod="openshift-insights/insights-runtime-extractor-jqjd8"
Sep 09 02:31:09.157688 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:09.157660    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6jkk\" (UniqueName: \"kubernetes.io/projected/175f5821-f3e7-44b4-8f45-945ab415b1c3-kube-api-access-w6jkk\") pod \"insights-runtime-extractor-jqjd8\" (UID: \"175f5821-f3e7-44b4-8f45-945ab415b1c3\") " pod="openshift-insights/insights-runtime-extractor-jqjd8"
Sep 09 02:31:09.650835 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:09.650802    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/175f5821-f3e7-44b4-8f45-945ab415b1c3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jqjd8\" (UID: \"175f5821-f3e7-44b4-8f45-945ab415b1c3\") " pod="openshift-insights/insights-runtime-extractor-jqjd8"
Sep 09 02:31:09.651207 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:09.650879    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/175f5821-f3e7-44b4-8f45-945ab415b1c3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jqjd8\" (UID: \"175f5821-f3e7-44b4-8f45-945ab415b1c3\") " pod="openshift-insights/insights-runtime-extractor-jqjd8"
Sep 09 02:31:09.651207 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:31:09.650935    2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 02:31:09.651207 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:31:09.651008    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/175f5821-f3e7-44b4-8f45-945ab415b1c3-insights-runtime-extractor-tls podName:175f5821-f3e7-44b4-8f45-945ab415b1c3 nodeName:}" failed. No retries permitted until 2026-09-09 02:31:10.65099471 +0000 UTC m=+52.905298349 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/175f5821-f3e7-44b4-8f45-945ab415b1c3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jqjd8" (UID: "175f5821-f3e7-44b4-8f45-945ab415b1c3") : secret "insights-runtime-extractor-tls" not found
Sep 09 02:31:09.651207 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:31:09.650942    2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 02:31:09.651207 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:31:09.651058    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/175f5821-f3e7-44b4-8f45-945ab415b1c3-kube-rbac-proxy-cm podName:175f5821-f3e7-44b4-8f45-945ab415b1c3 nodeName:}" failed. No retries permitted until 2026-09-09 02:31:10.651047358 +0000 UTC m=+52.905350986 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/175f5821-f3e7-44b4-8f45-945ab415b1c3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jqjd8" (UID: "175f5821-f3e7-44b4-8f45-945ab415b1c3") : configmap "kube-rbac-proxy" not found
Sep 09 02:31:10.657816 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:10.657777    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/175f5821-f3e7-44b4-8f45-945ab415b1c3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jqjd8\" (UID: \"175f5821-f3e7-44b4-8f45-945ab415b1c3\") " pod="openshift-insights/insights-runtime-extractor-jqjd8"
Sep 09 02:31:10.658288 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:10.657901    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/175f5821-f3e7-44b4-8f45-945ab415b1c3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jqjd8\" (UID: \"175f5821-f3e7-44b4-8f45-945ab415b1c3\") " pod="openshift-insights/insights-runtime-extractor-jqjd8"
Sep 09 02:31:10.658288 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:31:10.657933    2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 02:31:10.658288 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:31:10.657970    2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 02:31:10.658288 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:31:10.658030    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/175f5821-f3e7-44b4-8f45-945ab415b1c3-insights-runtime-extractor-tls podName:175f5821-f3e7-44b4-8f45-945ab415b1c3 nodeName:}" failed. No retries permitted until 2026-09-09 02:31:12.658010155 +0000 UTC m=+54.912313768 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/175f5821-f3e7-44b4-8f45-945ab415b1c3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jqjd8" (UID: "175f5821-f3e7-44b4-8f45-945ab415b1c3") : secret "insights-runtime-extractor-tls" not found
Sep 09 02:31:10.658288 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:31:10.658050    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/175f5821-f3e7-44b4-8f45-945ab415b1c3-kube-rbac-proxy-cm podName:175f5821-f3e7-44b4-8f45-945ab415b1c3 nodeName:}" failed. No retries permitted until 2026-09-09 02:31:12.6580413 +0000 UTC m=+54.912344917 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/175f5821-f3e7-44b4-8f45-945ab415b1c3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jqjd8" (UID: "175f5821-f3e7-44b4-8f45-945ab415b1c3") : configmap "kube-rbac-proxy" not found
Sep 09 02:31:10.658288 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:10.658177    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-d7t89_90413a10-3c11-4127-b7c8-6e8ac09b1fc9/dns-node-resolver/0.log"
Sep 09 02:31:11.967021 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:11.966983    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c5fd6885-5b56-4f0c-b1dd-3cff5153682b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-5g4th\" (UID: \"c5fd6885-5b56-4f0c-b1dd-3cff5153682b\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-5g4th"
Sep 09 02:31:11.967403 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:31:11.967103    2570 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 02:31:11.967403 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:31:11.967160    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5fd6885-5b56-4f0c-b1dd-3cff5153682b-cluster-monitoring-operator-tls podName:c5fd6885-5b56-4f0c-b1dd-3cff5153682b nodeName:}" failed. No retries permitted until 2026-09-09 02:31:19.967145698 +0000 UTC m=+62.221449325 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/c5fd6885-5b56-4f0c-b1dd-3cff5153682b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-5g4th" (UID: "c5fd6885-5b56-4f0c-b1dd-3cff5153682b") : secret "cluster-monitoring-operator-tls" not found
Sep 09 02:31:12.056142 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:12.056114    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zg4dm_721fa3c1-a7d2-4245-a7bd-76eb7facfc36/node-ca/0.log"
Sep 09 02:31:12.067275 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:12.067250    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5dad8e0-d4e8-4325-a274-a134fc6959a6-service-ca-bundle\") pod \"router-default-69c66d4b4-pl2k4\" (UID: \"d5dad8e0-d4e8-4325-a274-a134fc6959a6\") " pod="openshift-ingress/router-default-69c66d4b4-pl2k4"
Sep 09 02:31:12.067362 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:12.067286    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5dad8e0-d4e8-4325-a274-a134fc6959a6-metrics-certs\") pod \"router-default-69c66d4b4-pl2k4\" (UID: \"d5dad8e0-d4e8-4325-a274-a134fc6959a6\") " pod="openshift-ingress/router-default-69c66d4b4-pl2k4"
Sep 09 02:31:12.067414 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:31:12.067402    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d5dad8e0-d4e8-4325-a274-a134fc6959a6-service-ca-bundle podName:d5dad8e0-d4e8-4325-a274-a134fc6959a6 nodeName:}" failed. No retries permitted until 2026-09-09 02:31:20.067386845 +0000 UTC m=+62.321690488 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d5dad8e0-d4e8-4325-a274-a134fc6959a6-service-ca-bundle") pod "router-default-69c66d4b4-pl2k4" (UID: "d5dad8e0-d4e8-4325-a274-a134fc6959a6") : configmap references non-existent config key: service-ca.crt
Sep 09 02:31:12.067414 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:31:12.067409    2570 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 09 02:31:12.067499 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:31:12.067444    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5dad8e0-d4e8-4325-a274-a134fc6959a6-metrics-certs podName:d5dad8e0-d4e8-4325-a274-a134fc6959a6 nodeName:}" failed. No retries permitted until 2026-09-09 02:31:20.067434446 +0000 UTC m=+62.321738060 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d5dad8e0-d4e8-4325-a274-a134fc6959a6-metrics-certs") pod "router-default-69c66d4b4-pl2k4" (UID: "d5dad8e0-d4e8-4325-a274-a134fc6959a6") : secret "router-metrics-certs-default" not found
Sep 09 02:31:12.659572 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:12.659520    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-wjvs5_ec031e2f-88e3-4608-a1b7-de11b4d537a0/serve-healthcheck-canary/0.log"
Sep 09 02:31:12.672319 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:12.672296    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/175f5821-f3e7-44b4-8f45-945ab415b1c3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jqjd8\" (UID: \"175f5821-f3e7-44b4-8f45-945ab415b1c3\") " pod="openshift-insights/insights-runtime-extractor-jqjd8"
Sep 09 02:31:12.672384 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:12.672361    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/175f5821-f3e7-44b4-8f45-945ab415b1c3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jqjd8\" (UID: \"175f5821-f3e7-44b4-8f45-945ab415b1c3\") " pod="openshift-insights/insights-runtime-extractor-jqjd8"
Sep 09 02:31:12.672433 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:31:12.672414    2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 02:31:12.672472 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:31:12.672440    2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 02:31:12.672507 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:31:12.672489    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/175f5821-f3e7-44b4-8f45-945ab415b1c3-insights-runtime-extractor-tls podName:175f5821-f3e7-44b4-8f45-945ab415b1c3 nodeName:}" failed. No retries permitted until 2026-09-09 02:31:16.672477165 +0000 UTC m=+58.926780779 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/175f5821-f3e7-44b4-8f45-945ab415b1c3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jqjd8" (UID: "175f5821-f3e7-44b4-8f45-945ab415b1c3") : secret "insights-runtime-extractor-tls" not found
Sep 09 02:31:12.672548 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:31:12.672507    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/175f5821-f3e7-44b4-8f45-945ab415b1c3-kube-rbac-proxy-cm podName:175f5821-f3e7-44b4-8f45-945ab415b1c3 nodeName:}" failed. No retries permitted until 2026-09-09 02:31:16.672497477 +0000 UTC m=+58.926801100 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/175f5821-f3e7-44b4-8f45-945ab415b1c3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jqjd8" (UID: "175f5821-f3e7-44b4-8f45-945ab415b1c3") : configmap "kube-rbac-proxy" not found
Sep 09 02:31:16.700947 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:16.700915    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/175f5821-f3e7-44b4-8f45-945ab415b1c3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jqjd8\" (UID: \"175f5821-f3e7-44b4-8f45-945ab415b1c3\") " pod="openshift-insights/insights-runtime-extractor-jqjd8"
Sep 09 02:31:16.701491 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:16.700986    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/175f5821-f3e7-44b4-8f45-945ab415b1c3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jqjd8\" (UID: \"175f5821-f3e7-44b4-8f45-945ab415b1c3\") " pod="openshift-insights/insights-runtime-extractor-jqjd8"
Sep 09 02:31:16.701491 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:31:16.701060    2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 02:31:16.701491 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:31:16.701072    2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 02:31:16.701491 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:31:16.701123    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/175f5821-f3e7-44b4-8f45-945ab415b1c3-insights-runtime-extractor-tls podName:175f5821-f3e7-44b4-8f45-945ab415b1c3 nodeName:}" failed. No retries permitted until 2026-09-09 02:31:24.70111073 +0000 UTC m=+66.955414357 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/175f5821-f3e7-44b4-8f45-945ab415b1c3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jqjd8" (UID: "175f5821-f3e7-44b4-8f45-945ab415b1c3") : secret "insights-runtime-extractor-tls" not found
Sep 09 02:31:16.701491 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:31:16.701157    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/175f5821-f3e7-44b4-8f45-945ab415b1c3-kube-rbac-proxy-cm podName:175f5821-f3e7-44b4-8f45-945ab415b1c3 nodeName:}" failed. No retries permitted until 2026-09-09 02:31:24.701135105 +0000 UTC m=+66.955438731 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/175f5821-f3e7-44b4-8f45-945ab415b1c3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jqjd8" (UID: "175f5821-f3e7-44b4-8f45-945ab415b1c3") : configmap "kube-rbac-proxy" not found
Sep 09 02:31:17.389791 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:17.389763    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xcqxh"
Sep 09 02:31:20.027773 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:20.027737    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c5fd6885-5b56-4f0c-b1dd-3cff5153682b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-5g4th\" (UID: \"c5fd6885-5b56-4f0c-b1dd-3cff5153682b\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-5g4th"
Sep 09 02:31:20.028154 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:31:20.027874    2570 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 02:31:20.028154 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:31:20.027930    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5fd6885-5b56-4f0c-b1dd-3cff5153682b-cluster-monitoring-operator-tls podName:c5fd6885-5b56-4f0c-b1dd-3cff5153682b nodeName:}" failed. No retries permitted until 2026-09-09 02:31:36.027915768 +0000 UTC m=+78.282219387 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/c5fd6885-5b56-4f0c-b1dd-3cff5153682b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-5g4th" (UID: "c5fd6885-5b56-4f0c-b1dd-3cff5153682b") : secret "cluster-monitoring-operator-tls" not found
Sep 09 02:31:20.128711 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:20.128660    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5dad8e0-d4e8-4325-a274-a134fc6959a6-service-ca-bundle\") pod \"router-default-69c66d4b4-pl2k4\" (UID: \"d5dad8e0-d4e8-4325-a274-a134fc6959a6\") " pod="openshift-ingress/router-default-69c66d4b4-pl2k4"
Sep 09 02:31:20.128711 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:20.128721    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5dad8e0-d4e8-4325-a274-a134fc6959a6-metrics-certs\") pod \"router-default-69c66d4b4-pl2k4\" (UID: \"d5dad8e0-d4e8-4325-a274-a134fc6959a6\") " pod="openshift-ingress/router-default-69c66d4b4-pl2k4"
Sep 09 02:31:20.128906 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:31:20.128845    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d5dad8e0-d4e8-4325-a274-a134fc6959a6-service-ca-bundle podName:d5dad8e0-d4e8-4325-a274-a134fc6959a6 nodeName:}" failed. No retries permitted until 2026-09-09 02:31:36.128822542 +0000 UTC m=+78.383126179 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d5dad8e0-d4e8-4325-a274-a134fc6959a6-service-ca-bundle") pod "router-default-69c66d4b4-pl2k4" (UID: "d5dad8e0-d4e8-4325-a274-a134fc6959a6") : configmap references non-existent config key: service-ca.crt
Sep 09 02:31:20.128906 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:31:20.128890    2570 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 09 02:31:20.128998 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:31:20.128943    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5dad8e0-d4e8-4325-a274-a134fc6959a6-metrics-certs podName:d5dad8e0-d4e8-4325-a274-a134fc6959a6 nodeName:}" failed. No retries permitted until 2026-09-09 02:31:36.128932239 +0000 UTC m=+78.383235872 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d5dad8e0-d4e8-4325-a274-a134fc6959a6-metrics-certs") pod "router-default-69c66d4b4-pl2k4" (UID: "d5dad8e0-d4e8-4325-a274-a134fc6959a6") : secret "router-metrics-certs-default" not found
Sep 09 02:31:23.454015 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:23.453956    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/47cb49af-ac6a-4f18-ad58-5b40818d5827-metrics-tls\") pod \"dns-default-ttpxf\" (UID: \"47cb49af-ac6a-4f18-ad58-5b40818d5827\") " pod="openshift-dns/dns-default-ttpxf"
Sep 09 02:31:23.456153 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:23.456132    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/47cb49af-ac6a-4f18-ad58-5b40818d5827-metrics-tls\") pod \"dns-default-ttpxf\" (UID: \"47cb49af-ac6a-4f18-ad58-5b40818d5827\") " pod="openshift-dns/dns-default-ttpxf"
Sep 09 02:31:23.703839 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:23.703811    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-99mnl\""
Sep 09 02:31:23.711720 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:23.711675    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ttpxf"
Sep 09 02:31:23.956193 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:23.956163    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d786708a-bbb1-4250-a2d3-84dc2299d71a-metrics-certs\") pod \"network-metrics-daemon-zrljs\" (UID: \"d786708a-bbb1-4250-a2d3-84dc2299d71a\") " pod="openshift-multus/network-metrics-daemon-zrljs"
Sep 09 02:31:23.959449 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:23.959426    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 09 02:31:23.969039 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:23.968989    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d786708a-bbb1-4250-a2d3-84dc2299d71a-metrics-certs\") pod \"network-metrics-daemon-zrljs\" (UID: \"d786708a-bbb1-4250-a2d3-84dc2299d71a\") " pod="openshift-multus/network-metrics-daemon-zrljs"
Sep 09 02:31:23.985208 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:23.985188    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-ttpxf"]
Sep 09 02:31:23.988166 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:31:23.988140    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47cb49af_ac6a_4f18_ad58_5b40818d5827.slice/crio-9d5cda2e27b21cd15953e3b79c90772e64e257c6505d502507bd0346e4b5a76e WatchSource:0}: Error finding container 9d5cda2e27b21cd15953e3b79c90772e64e257c6505d502507bd0346e4b5a76e: Status 404 returned error can't find the container with id 9d5cda2e27b21cd15953e3b79c90772e64e257c6505d502507bd0346e4b5a76e
Sep 09 02:31:24.057438 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:24.057412    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/37ef684a-ce44-43fc-a803-519ce7d4d225-original-pull-secret\") pod \"global-pull-secret-syncer-952h4\" (UID: \"37ef684a-ce44-43fc-a803-519ce7d4d225\") " pod="kube-system/global-pull-secret-syncer-952h4"
Sep 09 02:31:24.057521 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:24.057460    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drrcs\" (UniqueName: \"kubernetes.io/projected/ac0a0b68-d43e-4ca8-8c4d-39bbe5e7519c-kube-api-access-drrcs\") pod \"network-check-target-6gkgn\" (UID: \"ac0a0b68-d43e-4ca8-8c4d-39bbe5e7519c\") " pod="openshift-network-diagnostics/network-check-target-6gkgn"
Sep 09 02:31:24.060849 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:24.060829    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 09 02:31:24.060935 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:24.060866    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 09 02:31:24.070221 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:24.070200    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/37ef684a-ce44-43fc-a803-519ce7d4d225-original-pull-secret\") pod \"global-pull-secret-syncer-952h4\" (UID: \"37ef684a-ce44-43fc-a803-519ce7d4d225\") " pod="kube-system/global-pull-secret-syncer-952h4"
Sep 09 02:31:24.084317 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:24.084298    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 09 02:31:24.091109 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:24.091093    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-drrcs\" (UniqueName: \"kubernetes.io/projected/ac0a0b68-d43e-4ca8-8c4d-39bbe5e7519c-kube-api-access-drrcs\") pod \"network-check-target-6gkgn\" (UID: \"ac0a0b68-d43e-4ca8-8c4d-39bbe5e7519c\") " pod="openshift-network-diagnostics/network-check-target-6gkgn"
Sep 09 02:31:24.133167 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:24.133146    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-rd54v\""
Sep 09 02:31:24.139432 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:24.139415    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6gkgn"
Sep 09 02:31:24.139747 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:24.139734    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-shl9x\""
Sep 09 02:31:24.142343 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:24.142322    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-952h4"
Sep 09 02:31:24.146915 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:24.146899    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zrljs"
Sep 09 02:31:24.420302 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:24.420256    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-952h4"]
Sep 09 02:31:24.422020 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:24.421995    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-6gkgn"]
Sep 09 02:31:24.423834 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:31:24.423810    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37ef684a_ce44_43fc_a803_519ce7d4d225.slice/crio-a47d0cbffecbeeae277e3ed33646791b1107d45a6ba42b61a79fd201126f9fec WatchSource:0}: Error finding container a47d0cbffecbeeae277e3ed33646791b1107d45a6ba42b61a79fd201126f9fec: Status 404 returned error can't find the container with id a47d0cbffecbeeae277e3ed33646791b1107d45a6ba42b61a79fd201126f9fec
Sep 09 02:31:24.427089 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:24.427020    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zrljs"]
Sep 09 02:31:24.430512 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:31:24.430489    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd786708a_bbb1_4250_a2d3_84dc2299d71a.slice/crio-da0a0fe9ff5431df82a668b1fdd7ffa44c147c7ca685f36e25e187746ee963c7 WatchSource:0}: Error finding container da0a0fe9ff5431df82a668b1fdd7ffa44c147c7ca685f36e25e187746ee963c7: Status 404 returned error can't find the container with id da0a0fe9ff5431df82a668b1fdd7ffa44c147c7ca685f36e25e187746ee963c7
Sep 09 02:31:24.486489 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:24.486429    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-952h4" event={"ID":"37ef684a-ce44-43fc-a803-519ce7d4d225","Type":"ContainerStarted","Data":"a47d0cbffecbeeae277e3ed33646791b1107d45a6ba42b61a79fd201126f9fec"}
Sep 09 02:31:24.487848 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:24.487825    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ttpxf" event={"ID":"47cb49af-ac6a-4f18-ad58-5b40818d5827","Type":"ContainerStarted","Data":"9d5cda2e27b21cd15953e3b79c90772e64e257c6505d502507bd0346e4b5a76e"}
Sep 09 02:31:24.489249 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:24.489228    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zrljs" event={"ID":"d786708a-bbb1-4250-a2d3-84dc2299d71a","Type":"ContainerStarted","Data":"da0a0fe9ff5431df82a668b1fdd7ffa44c147c7ca685f36e25e187746ee963c7"}
Sep 09 02:31:24.490594 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:24.490567    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-6gkgn" event={"ID":"ac0a0b68-d43e-4ca8-8c4d-39bbe5e7519c","Type":"ContainerStarted","Data":"83757ddb252cff19a6c8ae16fc76a7b0d69316ee98756f129d7ba23ef51fa4e2"}
Sep 09 02:31:24.764248 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:24.764171    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/175f5821-f3e7-44b4-8f45-945ab415b1c3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jqjd8\" (UID: \"175f5821-f3e7-44b4-8f45-945ab415b1c3\") " pod="openshift-insights/insights-runtime-extractor-jqjd8"
Sep 09 02:31:24.764493 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:24.764463    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/175f5821-f3e7-44b4-8f45-945ab415b1c3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jqjd8\" (UID: \"175f5821-f3e7-44b4-8f45-945ab415b1c3\") " pod="openshift-insights/insights-runtime-extractor-jqjd8"
Sep 09 02:31:24.764608 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:31:24.764589    2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 02:31:24.764677 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:31:24.764666    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/175f5821-f3e7-44b4-8f45-945ab415b1c3-insights-runtime-extractor-tls podName:175f5821-f3e7-44b4-8f45-945ab415b1c3 nodeName:}" failed. No retries permitted until 2026-09-09 02:31:40.764653109 +0000 UTC m=+83.018956736 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/175f5821-f3e7-44b4-8f45-945ab415b1c3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jqjd8" (UID: "175f5821-f3e7-44b4-8f45-945ab415b1c3") : secret "insights-runtime-extractor-tls" not found
Sep 09 02:31:24.764738 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:24.764718    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/175f5821-f3e7-44b4-8f45-945ab415b1c3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jqjd8\" (UID: \"175f5821-f3e7-44b4-8f45-945ab415b1c3\") " pod="openshift-insights/insights-runtime-extractor-jqjd8"
Sep 09 02:31:26.505142 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:26.505100    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ttpxf" event={"ID":"47cb49af-ac6a-4f18-ad58-5b40818d5827","Type":"ContainerStarted","Data":"9afb5724f35276a2ce362f1794c25900ef03d367e63201ba8fad233a8c3ffe3f"}
Sep 09 02:31:26.505577 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:26.505220    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ttpxf" event={"ID":"47cb49af-ac6a-4f18-ad58-5b40818d5827","Type":"ContainerStarted","Data":"49b604ee78b7defe7340c990f554d3ba0efe5195ba7936216f2fdab0d47cad70"}
Sep 09 02:31:26.505577 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:26.505401    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-ttpxf"
Sep 09 02:31:26.507283 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:26.507256    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zrljs" event={"ID":"d786708a-bbb1-4250-a2d3-84dc2299d71a","Type":"ContainerStarted","Data":"cd113007bf21b8b6c267c805d828358b046fd9ee1ef0d1abd49071b41cbbb902"}
Sep 09 02:31:26.507411 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:26.507302    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zrljs" event={"ID":"d786708a-bbb1-4250-a2d3-84dc2299d71a","Type":"ContainerStarted","Data":"7ec3663ba6d6b9872c051d17dc8543d0fee05d7eaf3f4221640b68ec1b6154cf"}
Sep 09 02:31:26.526248 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:26.526197    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-ttpxf" podStartSLOduration=33.517778699 podStartE2EDuration="35.526179705s" totalImagesPullingTime="2.008401006s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:30:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:31:26.525285968 +0000 UTC m=+68.779589606" watchObservedRunningTime="2026-09-09 02:31:26.526179705 +0000 UTC m=+68.780483343"
Sep 09 02:31:26.546583 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:26.546531    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-zrljs" podStartSLOduration=66.981516859 podStartE2EDuration="1m8.546513622s" totalImagesPullingTime="1.564996763s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:30:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:31:26.545570284 +0000 UTC m=+68.799873920" watchObservedRunningTime="2026-09-09 02:31:26.546513622 +0000 UTC m=+68.800817261"
Sep 09 02:31:29.519396 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:29.519362    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-952h4" event={"ID":"37ef684a-ce44-43fc-a803-519ce7d4d225","Type":"ContainerStarted","Data":"034524993e6b5e93b6e189b6828b6d639d1c047b3cf3d86eb7c6b0ab67c921c9"}
Sep 09 02:31:29.520925 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:29.520895    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-6gkgn" event={"ID":"ac0a0b68-d43e-4ca8-8c4d-39bbe5e7519c","Type":"ContainerStarted","Data":"2348c2324ad8713995e87f802fad7801e9c8fda2743eb551671eab4c41673c26"}
Sep 09 02:31:29.521049 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:29.521024    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-6gkgn"
Sep 09 02:31:29.538104 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:29.538058    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-952h4" podStartSLOduration=66.878946363 podStartE2EDuration="1m11.538043091s" totalImagesPullingTime="4.659096728s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 02:30:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:31:29.537268358 +0000 UTC m=+71.791571997" watchObservedRunningTime="2026-09-09 02:31:29.538043091 +0000 UTC m=+71.792346726"
Sep 09 02:31:29.559110 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:29.559073    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-6gkgn" podStartSLOduration=66.907409916 podStartE2EDuration="1m11.559060677s" totalImagesPullingTime="4.651650761s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:30:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:31:29.558036216 +0000 UTC m=+71.812339852" watchObservedRunningTime="2026-09-09 02:31:29.559060677 +0000 UTC m=+71.813364312"
Sep 09 02:31:34.099264 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:34.099233    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-947dd5f4d-j5cwc"]
Sep 09 02:31:34.102210 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:34.102192    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-947dd5f4d-j5cwc"
Sep 09 02:31:34.107208 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:34.107185    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 09 02:31:34.107315 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:34.107185    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 09 02:31:34.107315 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:34.107303    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 09 02:31:34.111614 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:34.111598    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 09 02:31:34.121077 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:34.121055    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-947dd5f4d-j5cwc"]
Sep 09 02:31:34.143516 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:34.143490    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/682234e8-5efb-4ccd-9381-9c3e31da1ef5-klusterlet-config\") pod \"klusterlet-addon-workmgr-947dd5f4d-j5cwc\" (UID: \"682234e8-5efb-4ccd-9381-9c3e31da1ef5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-947dd5f4d-j5cwc"
Sep 09 02:31:34.143602 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:34.143526    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/682234e8-5efb-4ccd-9381-9c3e31da1ef5-tmp\") pod \"klusterlet-addon-workmgr-947dd5f4d-j5cwc\" (UID: \"682234e8-5efb-4ccd-9381-9c3e31da1ef5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-947dd5f4d-j5cwc"
Sep 09 02:31:34.143642 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:34.143610    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whj5p\" (UniqueName: \"kubernetes.io/projected/682234e8-5efb-4ccd-9381-9c3e31da1ef5-kube-api-access-whj5p\") pod \"klusterlet-addon-workmgr-947dd5f4d-j5cwc\" (UID: \"682234e8-5efb-4ccd-9381-9c3e31da1ef5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-947dd5f4d-j5cwc"
Sep 09 02:31:34.204213 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:34.204185    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-fb8975b97-h6747"]
Sep 09 02:31:34.207104 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:34.207087    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-fb8975b97-h6747"
Sep 09 02:31:34.222518 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:34.222497    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-dckqx\""
Sep 09 02:31:34.222610 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:34.222587    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 09 02:31:34.222676 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:34.222662    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 09 02:31:34.227442 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:34.227425    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 09 02:31:34.233926 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:34.233903    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 09 02:31:34.234827 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:34.234806    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-fb8975b97-h6747"]
Sep 09 02:31:34.244934 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:34.244913    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/87f74e41-b9f9-4e9b-81fa-6d1599a0234b-installation-pull-secrets\") pod \"image-registry-fb8975b97-h6747\" (UID: \"87f74e41-b9f9-4e9b-81fa-6d1599a0234b\") " pod="openshift-image-registry/image-registry-fb8975b97-h6747"
Sep 09 02:31:34.245031 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:34.244951    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/87f74e41-b9f9-4e9b-81fa-6d1599a0234b-registry-tls\") pod \"image-registry-fb8975b97-h6747\" (UID: \"87f74e41-b9f9-4e9b-81fa-6d1599a0234b\") " pod="openshift-image-registry/image-registry-fb8975b97-h6747"
Sep 09 02:31:34.245031 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:34.244983    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/87f74e41-b9f9-4e9b-81fa-6d1599a0234b-image-registry-private-configuration\") pod \"image-registry-fb8975b97-h6747\" (UID: \"87f74e41-b9f9-4e9b-81fa-6d1599a0234b\") " pod="openshift-image-registry/image-registry-fb8975b97-h6747"
Sep 09 02:31:34.245119 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:34.245096    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whj5p\" (UniqueName: \"kubernetes.io/projected/682234e8-5efb-4ccd-9381-9c3e31da1ef5-kube-api-access-whj5p\") pod \"klusterlet-addon-workmgr-947dd5f4d-j5cwc\" (UID: \"682234e8-5efb-4ccd-9381-9c3e31da1ef5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-947dd5f4d-j5cwc"
Sep 09 02:31:34.245167 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:34.245129    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/87f74e41-b9f9-4e9b-81fa-6d1599a0234b-trusted-ca\") pod \"image-registry-fb8975b97-h6747\" (UID: \"87f74e41-b9f9-4e9b-81fa-6d1599a0234b\") " pod="openshift-image-registry/image-registry-fb8975b97-h6747"
Sep 09 02:31:34.245167 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:34.245159    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/87f74e41-b9f9-4e9b-81fa-6d1599a0234b-bound-sa-token\") pod \"image-registry-fb8975b97-h6747\" (UID: \"87f74e41-b9f9-4e9b-81fa-6d1599a0234b\") " pod="openshift-image-registry/image-registry-fb8975b97-h6747"
Sep 09 02:31:34.245260 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:34.245240    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/87f74e41-b9f9-4e9b-81fa-6d1599a0234b-registry-certificates\") pod \"image-registry-fb8975b97-h6747\" (UID: \"87f74e41-b9f9-4e9b-81fa-6d1599a0234b\") " pod="openshift-image-registry/image-registry-fb8975b97-h6747"
Sep 09 02:31:34.245308 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:34.245291    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/87f74e41-b9f9-4e9b-81fa-6d1599a0234b-ca-trust-extracted\") pod \"image-registry-fb8975b97-h6747\" (UID: \"87f74e41-b9f9-4e9b-81fa-6d1599a0234b\") " pod="openshift-image-registry/image-registry-fb8975b97-h6747"
Sep 09 02:31:34.245348 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:34.245335    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/682234e8-5efb-4ccd-9381-9c3e31da1ef5-klusterlet-config\") pod \"klusterlet-addon-workmgr-947dd5f4d-j5cwc\" (UID: \"682234e8-5efb-4ccd-9381-9c3e31da1ef5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-947dd5f4d-j5cwc"
Sep 09 02:31:34.245393 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:34.245364    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/682234e8-5efb-4ccd-9381-9c3e31da1ef5-tmp\") pod \"klusterlet-addon-workmgr-947dd5f4d-j5cwc\" (UID: \"682234e8-5efb-4ccd-9381-9c3e31da1ef5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-947dd5f4d-j5cwc"
Sep 09 02:31:34.245393 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:34.245387    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrh5c\" (UniqueName: \"kubernetes.io/projected/87f74e41-b9f9-4e9b-81fa-6d1599a0234b-kube-api-access-vrh5c\") pod \"image-registry-fb8975b97-h6747\" (UID: \"87f74e41-b9f9-4e9b-81fa-6d1599a0234b\") " pod="openshift-image-registry/image-registry-fb8975b97-h6747"
Sep 09 02:31:34.245713 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:34.245692    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/682234e8-5efb-4ccd-9381-9c3e31da1ef5-tmp\") pod \"klusterlet-addon-workmgr-947dd5f4d-j5cwc\" (UID: \"682234e8-5efb-4ccd-9381-9c3e31da1ef5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-947dd5f4d-j5cwc"
Sep 09 02:31:34.247689 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:34.247671    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/682234e8-5efb-4ccd-9381-9c3e31da1ef5-klusterlet-config\") pod \"klusterlet-addon-workmgr-947dd5f4d-j5cwc\" (UID: \"682234e8-5efb-4ccd-9381-9c3e31da1ef5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-947dd5f4d-j5cwc"
Sep 09 02:31:34.266752 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:34.266731    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-whj5p\" (UniqueName: \"kubernetes.io/projected/682234e8-5efb-4ccd-9381-9c3e31da1ef5-kube-api-access-whj5p\") pod \"klusterlet-addon-workmgr-947dd5f4d-j5cwc\" (UID: \"682234e8-5efb-4ccd-9381-9c3e31da1ef5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-947dd5f4d-j5cwc"
Sep 09 02:31:34.345810 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:34.345784    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrh5c\" (UniqueName: \"kubernetes.io/projected/87f74e41-b9f9-4e9b-81fa-6d1599a0234b-kube-api-access-vrh5c\") pod \"image-registry-fb8975b97-h6747\" (UID: \"87f74e41-b9f9-4e9b-81fa-6d1599a0234b\") " pod="openshift-image-registry/image-registry-fb8975b97-h6747"
Sep 09 02:31:34.345902 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:34.345817    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/87f74e41-b9f9-4e9b-81fa-6d1599a0234b-installation-pull-secrets\") pod \"image-registry-fb8975b97-h6747\" (UID: \"87f74e41-b9f9-4e9b-81fa-6d1599a0234b\") " pod="openshift-image-registry/image-registry-fb8975b97-h6747"
Sep 09 02:31:34.345902 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:34.345836    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/87f74e41-b9f9-4e9b-81fa-6d1599a0234b-registry-tls\") pod \"image-registry-fb8975b97-h6747\" (UID: \"87f74e41-b9f9-4e9b-81fa-6d1599a0234b\") " pod="openshift-image-registry/image-registry-fb8975b97-h6747"
Sep 09 02:31:34.346051 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:34.346029    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/87f74e41-b9f9-4e9b-81fa-6d1599a0234b-image-registry-private-configuration\") pod \"image-registry-fb8975b97-h6747\" (UID: \"87f74e41-b9f9-4e9b-81fa-6d1599a0234b\") " pod="openshift-image-registry/image-registry-fb8975b97-h6747"
Sep 09 02:31:34.346123 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:34.346113    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/87f74e41-b9f9-4e9b-81fa-6d1599a0234b-trusted-ca\") pod \"image-registry-fb8975b97-h6747\" (UID: \"87f74e41-b9f9-4e9b-81fa-6d1599a0234b\") " pod="openshift-image-registry/image-registry-fb8975b97-h6747"
Sep 09 02:31:34.346175 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:34.346149    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/87f74e41-b9f9-4e9b-81fa-6d1599a0234b-bound-sa-token\") pod \"image-registry-fb8975b97-h6747\" (UID: \"87f74e41-b9f9-4e9b-81fa-6d1599a0234b\") " pod="openshift-image-registry/image-registry-fb8975b97-h6747"
Sep 09 02:31:34.346228 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:34.346202    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/87f74e41-b9f9-4e9b-81fa-6d1599a0234b-registry-certificates\") pod \"image-registry-fb8975b97-h6747\" (UID: \"87f74e41-b9f9-4e9b-81fa-6d1599a0234b\") " pod="openshift-image-registry/image-registry-fb8975b97-h6747"
Sep 09 02:31:34.346281 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:34.346258    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/87f74e41-b9f9-4e9b-81fa-6d1599a0234b-ca-trust-extracted\") pod \"image-registry-fb8975b97-h6747\" (UID: \"87f74e41-b9f9-4e9b-81fa-6d1599a0234b\") " pod="openshift-image-registry/image-registry-fb8975b97-h6747"
Sep 09 02:31:34.346580 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:34.346560    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/87f74e41-b9f9-4e9b-81fa-6d1599a0234b-ca-trust-extracted\") pod \"image-registry-fb8975b97-h6747\" (UID: \"87f74e41-b9f9-4e9b-81fa-6d1599a0234b\") " pod="openshift-image-registry/image-registry-fb8975b97-h6747"
Sep 09 02:31:34.347271 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:34.347250    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/87f74e41-b9f9-4e9b-81fa-6d1599a0234b-trusted-ca\") pod \"image-registry-fb8975b97-h6747\" (UID: \"87f74e41-b9f9-4e9b-81fa-6d1599a0234b\") " pod="openshift-image-registry/image-registry-fb8975b97-h6747"
Sep 09 02:31:34.347577 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:34.347555    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/87f74e41-b9f9-4e9b-81fa-6d1599a0234b-registry-certificates\") pod \"image-registry-fb8975b97-h6747\" (UID: \"87f74e41-b9f9-4e9b-81fa-6d1599a0234b\") " pod="openshift-image-registry/image-registry-fb8975b97-h6747"
Sep 09 02:31:34.348413 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:34.348383    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/87f74e41-b9f9-4e9b-81fa-6d1599a0234b-image-registry-private-configuration\") pod \"image-registry-fb8975b97-h6747\" (UID: \"87f74e41-b9f9-4e9b-81fa-6d1599a0234b\") " pod="openshift-image-registry/image-registry-fb8975b97-h6747"
Sep 09 02:31:34.348413 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:34.348403    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/87f74e41-b9f9-4e9b-81fa-6d1599a0234b-registry-tls\") pod \"image-registry-fb8975b97-h6747\" (UID: \"87f74e41-b9f9-4e9b-81fa-6d1599a0234b\") " pod="openshift-image-registry/image-registry-fb8975b97-h6747"
Sep 09 02:31:34.348525 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:34.348508    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/87f74e41-b9f9-4e9b-81fa-6d1599a0234b-installation-pull-secrets\") pod \"image-registry-fb8975b97-h6747\" (UID: \"87f74e41-b9f9-4e9b-81fa-6d1599a0234b\") " pod="openshift-image-registry/image-registry-fb8975b97-h6747"
Sep 09 02:31:34.364892 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:34.364846    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrh5c\" (UniqueName: \"kubernetes.io/projected/87f74e41-b9f9-4e9b-81fa-6d1599a0234b-kube-api-access-vrh5c\") pod \"image-registry-fb8975b97-h6747\" (UID: \"87f74e41-b9f9-4e9b-81fa-6d1599a0234b\") " pod="openshift-image-registry/image-registry-fb8975b97-h6747"
Sep 09 02:31:34.369961 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:34.369936    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/87f74e41-b9f9-4e9b-81fa-6d1599a0234b-bound-sa-token\") pod \"image-registry-fb8975b97-h6747\" (UID: \"87f74e41-b9f9-4e9b-81fa-6d1599a0234b\") " pod="openshift-image-registry/image-registry-fb8975b97-h6747"
Sep 09 02:31:34.410831 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:34.410810    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-947dd5f4d-j5cwc"
Sep 09 02:31:34.515834 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:34.515802    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-fb8975b97-h6747"
Sep 09 02:31:34.688441 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:34.688416    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-947dd5f4d-j5cwc"]
Sep 09 02:31:34.690846 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:31:34.690820    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod682234e8_5efb_4ccd_9381_9c3e31da1ef5.slice/crio-6bc736fdaf8cc3415bbdbb14039466c33ce249ea15ab88c9630f1ec884e95fdd WatchSource:0}: Error finding container 6bc736fdaf8cc3415bbdbb14039466c33ce249ea15ab88c9630f1ec884e95fdd: Status 404 returned error can't find the container with id 6bc736fdaf8cc3415bbdbb14039466c33ce249ea15ab88c9630f1ec884e95fdd
Sep 09 02:31:34.804079 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:34.804056    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-fb8975b97-h6747"]
Sep 09 02:31:34.805833 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:31:34.805800    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87f74e41_b9f9_4e9b_81fa_6d1599a0234b.slice/crio-fb235c3e0e8bae552d8c90d9d38735e1f11a50af570734efbc02b7809c6ac044 WatchSource:0}: Error finding container fb235c3e0e8bae552d8c90d9d38735e1f11a50af570734efbc02b7809c6ac044: Status 404 returned error can't find the container with id fb235c3e0e8bae552d8c90d9d38735e1f11a50af570734efbc02b7809c6ac044
Sep 09 02:31:35.540891 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:35.540846    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-947dd5f4d-j5cwc" event={"ID":"682234e8-5efb-4ccd-9381-9c3e31da1ef5","Type":"ContainerStarted","Data":"6bc736fdaf8cc3415bbdbb14039466c33ce249ea15ab88c9630f1ec884e95fdd"}
Sep 09 02:31:35.542267 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:35.542241    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-fb8975b97-h6747" event={"ID":"87f74e41-b9f9-4e9b-81fa-6d1599a0234b","Type":"ContainerStarted","Data":"96e505fa3143b6edc044e17921c7e1afdbb5efce75cd50552a733ef19067168d"}
Sep 09 02:31:35.542386 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:35.542285    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-fb8975b97-h6747" event={"ID":"87f74e41-b9f9-4e9b-81fa-6d1599a0234b","Type":"ContainerStarted","Data":"fb235c3e0e8bae552d8c90d9d38735e1f11a50af570734efbc02b7809c6ac044"}
Sep 09 02:31:35.542386 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:35.542370    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-fb8975b97-h6747"
Sep 09 02:31:35.567577 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:35.567534    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-fb8975b97-h6747" podStartSLOduration=1.567519623 podStartE2EDuration="1.567519623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:31:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:31:35.566570778 +0000 UTC m=+77.820874431" watchObservedRunningTime="2026-09-09 02:31:35.567519623 +0000 UTC m=+77.821823250"
Sep 09 02:31:36.061345 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:36.061296    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c5fd6885-5b56-4f0c-b1dd-3cff5153682b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-5g4th\" (UID: \"c5fd6885-5b56-4f0c-b1dd-3cff5153682b\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-5g4th"
Sep 09 02:31:36.064773 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:36.064750    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c5fd6885-5b56-4f0c-b1dd-3cff5153682b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-5g4th\" (UID: \"c5fd6885-5b56-4f0c-b1dd-3cff5153682b\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-5g4th"
Sep 09 02:31:36.162441 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:36.162407    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5dad8e0-d4e8-4325-a274-a134fc6959a6-metrics-certs\") pod \"router-default-69c66d4b4-pl2k4\" (UID: \"d5dad8e0-d4e8-4325-a274-a134fc6959a6\") " pod="openshift-ingress/router-default-69c66d4b4-pl2k4"
Sep 09 02:31:36.162573 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:36.162460    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5dad8e0-d4e8-4325-a274-a134fc6959a6-service-ca-bundle\") pod \"router-default-69c66d4b4-pl2k4\" (UID: \"d5dad8e0-d4e8-4325-a274-a134fc6959a6\") " pod="openshift-ingress/router-default-69c66d4b4-pl2k4"
Sep 09 02:31:36.163058 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:36.163028    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5dad8e0-d4e8-4325-a274-a134fc6959a6-service-ca-bundle\") pod \"router-default-69c66d4b4-pl2k4\" (UID: \"d5dad8e0-d4e8-4325-a274-a134fc6959a6\") " pod="openshift-ingress/router-default-69c66d4b4-pl2k4"
Sep 09 02:31:36.165221 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:36.165192    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5dad8e0-d4e8-4325-a274-a134fc6959a6-metrics-certs\") pod \"router-default-69c66d4b4-pl2k4\" (UID: \"d5dad8e0-d4e8-4325-a274-a134fc6959a6\") " pod="openshift-ingress/router-default-69c66d4b4-pl2k4"
Sep 09 02:31:36.275601 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:36.275567    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-5dd9m\""
Sep 09 02:31:36.283313 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:36.283239    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-5g4th"
Sep 09 02:31:36.377934 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:36.377907    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-42tmd\""
Sep 09 02:31:36.385579 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:36.385552    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-69c66d4b4-pl2k4"
Sep 09 02:31:36.513898 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:36.513868    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-ttpxf"
Sep 09 02:31:36.584959 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:36.584927    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-5g4th"]
Sep 09 02:31:36.588560 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:31:36.588534    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5fd6885_5b56_4f0c_b1dd_3cff5153682b.slice/crio-6512eaf006b0a52e3a17e99179886858305a77eb711b3fef0eb3ffd2e51cd2e8 WatchSource:0}: Error finding container 6512eaf006b0a52e3a17e99179886858305a77eb711b3fef0eb3ffd2e51cd2e8: Status 404 returned error can't find the container with id 6512eaf006b0a52e3a17e99179886858305a77eb711b3fef0eb3ffd2e51cd2e8
Sep 09 02:31:36.675802 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:36.675770    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-69c66d4b4-pl2k4"]
Sep 09 02:31:37.340594 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:31:37.340563    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5dad8e0_d4e8_4325_a274_a134fc6959a6.slice/crio-40744faa089c488fbeef56938289eae1a013f5c66b6628fed28e88720bf58571 WatchSource:0}: Error finding container 40744faa089c488fbeef56938289eae1a013f5c66b6628fed28e88720bf58571: Status 404 returned error can't find the container with id 40744faa089c488fbeef56938289eae1a013f5c66b6628fed28e88720bf58571
Sep 09 02:31:37.551738 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:37.551688    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-69c66d4b4-pl2k4" event={"ID":"d5dad8e0-d4e8-4325-a274-a134fc6959a6","Type":"ContainerStarted","Data":"3ad7b0d53385ccfbd1a36aa3ff22cb56886767a7c98c85ba2da1b51ccd65c7d6"}
Sep 09 02:31:37.551918 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:37.551763    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-69c66d4b4-pl2k4" event={"ID":"d5dad8e0-d4e8-4325-a274-a134fc6959a6","Type":"ContainerStarted","Data":"40744faa089c488fbeef56938289eae1a013f5c66b6628fed28e88720bf58571"}
Sep 09 02:31:37.553494 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:37.553463    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-5g4th" event={"ID":"c5fd6885-5b56-4f0c-b1dd-3cff5153682b","Type":"ContainerStarted","Data":"6512eaf006b0a52e3a17e99179886858305a77eb711b3fef0eb3ffd2e51cd2e8"}
Sep 09 02:31:37.593323 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:37.593221    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-69c66d4b4-pl2k4" podStartSLOduration=33.593205277 podStartE2EDuration="33.593205277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:31:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:31:37.591398967 +0000 UTC m=+79.845702604" watchObservedRunningTime="2026-09-09 02:31:37.593205277 +0000 UTC m=+79.847508913"
Sep 09 02:31:38.385777 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:38.385744    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-69c66d4b4-pl2k4"
Sep 09 02:31:38.388530 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:38.388505    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-69c66d4b4-pl2k4"
Sep 09 02:31:38.557869 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:38.557804    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-69c66d4b4-pl2k4"
Sep 09 02:31:38.559347 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:38.559319    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-69c66d4b4-pl2k4"
Sep 09 02:31:39.563625 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:39.563587    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-5g4th" event={"ID":"c5fd6885-5b56-4f0c-b1dd-3cff5153682b","Type":"ContainerStarted","Data":"6b848dfa9599443d4deffd5cc2a154892c573f2092da731c30126b3ae924426a"}
Sep 09 02:31:39.586257 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:39.586201    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-5g4th" podStartSLOduration=33.441535795 podStartE2EDuration="35.586179s" totalImagesPullingTime="2.144643205s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:31:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:31:39.585088434 +0000 UTC m=+81.839392071" watchObservedRunningTime="2026-09-09 02:31:39.586179 +0000 UTC m=+81.840482637"
Sep 09 02:31:40.568726 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:40.568693    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-947dd5f4d-j5cwc" event={"ID":"682234e8-5efb-4ccd-9381-9c3e31da1ef5","Type":"ContainerStarted","Data":"871af7b869ce4615b69532a9a02a57138bee0603fc39af3e1dc0b814dae91394"}
Sep 09 02:31:40.589988 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:40.589926    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-947dd5f4d-j5cwc" podStartSLOduration=0.793057294 podStartE2EDuration="6.589904404s" totalImagesPullingTime="5.79684711s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:31:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:31:40.588216002 +0000 UTC m=+82.842519638" watchObservedRunningTime="2026-09-09 02:31:40.589904404 +0000 UTC m=+82.844208041"
Sep 09 02:31:40.806619 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:40.806473    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/175f5821-f3e7-44b4-8f45-945ab415b1c3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jqjd8\" (UID: \"175f5821-f3e7-44b4-8f45-945ab415b1c3\") " pod="openshift-insights/insights-runtime-extractor-jqjd8"
Sep 09 02:31:40.808851 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:40.808819    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/175f5821-f3e7-44b4-8f45-945ab415b1c3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jqjd8\" (UID: \"175f5821-f3e7-44b4-8f45-945ab415b1c3\") " pod="openshift-insights/insights-runtime-extractor-jqjd8"
Sep 09 02:31:41.066809 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:41.066729    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-68fdc\""
Sep 09 02:31:41.074053 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:41.074031    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jqjd8"
Sep 09 02:31:41.352474 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:41.352440    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jqjd8"]
Sep 09 02:31:41.355178 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:31:41.355150    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod175f5821_f3e7_44b4_8f45_945ab415b1c3.slice/crio-43e5915ec3a28a786cfe8f793ea0071853bf992309ea325f5567edc592243337 WatchSource:0}: Error finding container 43e5915ec3a28a786cfe8f793ea0071853bf992309ea325f5567edc592243337: Status 404 returned error can't find the container with id 43e5915ec3a28a786cfe8f793ea0071853bf992309ea325f5567edc592243337
Sep 09 02:31:41.573596 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:41.573563    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jqjd8" event={"ID":"175f5821-f3e7-44b4-8f45-945ab415b1c3","Type":"ContainerStarted","Data":"4b6e684c0718413219c352e6be22fd2adec5b67ff2449153b1e324402c2532ce"}
Sep 09 02:31:41.573954 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:41.573620    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jqjd8" event={"ID":"175f5821-f3e7-44b4-8f45-945ab415b1c3","Type":"ContainerStarted","Data":"43e5915ec3a28a786cfe8f793ea0071853bf992309ea325f5567edc592243337"}
Sep 09 02:31:41.573954 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:41.573719    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-947dd5f4d-j5cwc"
Sep 09 02:31:41.575517 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:41.575497    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-947dd5f4d-j5cwc"
Sep 09 02:31:42.342167 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:42.342097    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-zxcbk"]
Sep 09 02:31:42.345210 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:42.345193    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-zxcbk"
Sep 09 02:31:42.350358 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:42.350337    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 09 02:31:42.350358 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:42.350354    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 09 02:31:42.351648 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:42.351627    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-8pfjq\""
Sep 09 02:31:42.351731 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:42.351669    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 09 02:31:42.357621 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:42.357598    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-zxcbk"]
Sep 09 02:31:42.418182 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:42.418153    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d19499c4-d8c2-4865-bf9b-51bcc6f46905-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-zxcbk\" (UID: \"d19499c4-d8c2-4865-bf9b-51bcc6f46905\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-zxcbk"
Sep 09 02:31:42.418294 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:42.418196    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnl29\" (UniqueName: \"kubernetes.io/projected/d19499c4-d8c2-4865-bf9b-51bcc6f46905-kube-api-access-rnl29\") pod \"prometheus-operator-6f4946db56-zxcbk\" (UID: \"d19499c4-d8c2-4865-bf9b-51bcc6f46905\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-zxcbk"
Sep 09 02:31:42.418294 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:42.418230    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d19499c4-d8c2-4865-bf9b-51bcc6f46905-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-zxcbk\" (UID: \"d19499c4-d8c2-4865-bf9b-51bcc6f46905\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-zxcbk"
Sep 09 02:31:42.418294 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:42.418277    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/d19499c4-d8c2-4865-bf9b-51bcc6f46905-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-zxcbk\" (UID: \"d19499c4-d8c2-4865-bf9b-51bcc6f46905\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-zxcbk"
Sep 09 02:31:42.518916 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:42.518892    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnl29\" (UniqueName: \"kubernetes.io/projected/d19499c4-d8c2-4865-bf9b-51bcc6f46905-kube-api-access-rnl29\") pod \"prometheus-operator-6f4946db56-zxcbk\" (UID: \"d19499c4-d8c2-4865-bf9b-51bcc6f46905\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-zxcbk"
Sep 09 02:31:42.519039 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:42.518944    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d19499c4-d8c2-4865-bf9b-51bcc6f46905-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-zxcbk\" (UID: \"d19499c4-d8c2-4865-bf9b-51bcc6f46905\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-zxcbk"
Sep 09 02:31:42.519107 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:42.519087    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/d19499c4-d8c2-4865-bf9b-51bcc6f46905-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-zxcbk\" (UID: \"d19499c4-d8c2-4865-bf9b-51bcc6f46905\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-zxcbk"
Sep 09 02:31:42.519173 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:42.519142    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d19499c4-d8c2-4865-bf9b-51bcc6f46905-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-zxcbk\" (UID: \"d19499c4-d8c2-4865-bf9b-51bcc6f46905\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-zxcbk"
Sep 09 02:31:42.519247 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:31:42.519232    2570 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Sep 09 02:31:42.519326 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:31:42.519315    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d19499c4-d8c2-4865-bf9b-51bcc6f46905-prometheus-operator-tls podName:d19499c4-d8c2-4865-bf9b-51bcc6f46905 nodeName:}" failed. No retries permitted until 2026-09-09 02:31:43.019297149 +0000 UTC m=+85.273600766 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/d19499c4-d8c2-4865-bf9b-51bcc6f46905-prometheus-operator-tls") pod "prometheus-operator-6f4946db56-zxcbk" (UID: "d19499c4-d8c2-4865-bf9b-51bcc6f46905") : secret "prometheus-operator-tls" not found
Sep 09 02:31:42.519757 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:42.519737    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d19499c4-d8c2-4865-bf9b-51bcc6f46905-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-zxcbk\" (UID: \"d19499c4-d8c2-4865-bf9b-51bcc6f46905\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-zxcbk"
Sep 09 02:31:42.521286 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:42.521266    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d19499c4-d8c2-4865-bf9b-51bcc6f46905-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-zxcbk\" (UID: \"d19499c4-d8c2-4865-bf9b-51bcc6f46905\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-zxcbk"
Sep 09 02:31:42.528588 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:42.528569    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnl29\" (UniqueName: \"kubernetes.io/projected/d19499c4-d8c2-4865-bf9b-51bcc6f46905-kube-api-access-rnl29\") pod \"prometheus-operator-6f4946db56-zxcbk\" (UID: \"d19499c4-d8c2-4865-bf9b-51bcc6f46905\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-zxcbk"
Sep 09 02:31:42.582773 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:42.582015    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jqjd8" event={"ID":"175f5821-f3e7-44b4-8f45-945ab415b1c3","Type":"ContainerStarted","Data":"b3973ccd81aa1d995dee940798bb38a5def91f2c1aa8d8540e8546470b825882"}
Sep 09 02:31:43.022382 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:43.022331    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/d19499c4-d8c2-4865-bf9b-51bcc6f46905-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-zxcbk\" (UID: \"d19499c4-d8c2-4865-bf9b-51bcc6f46905\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-zxcbk"
Sep 09 02:31:43.025239 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:43.025214    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/d19499c4-d8c2-4865-bf9b-51bcc6f46905-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-zxcbk\" (UID: \"d19499c4-d8c2-4865-bf9b-51bcc6f46905\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-zxcbk"
Sep 09 02:31:43.253060 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:43.253024    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-zxcbk"
Sep 09 02:31:43.586529 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:43.586501    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jqjd8" event={"ID":"175f5821-f3e7-44b4-8f45-945ab415b1c3","Type":"ContainerStarted","Data":"e14626907053f2dfabfcbfc1aa62af38e354b9a50a0f12e0c31743c672cbc909"}
Sep 09 02:31:43.610796 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:43.610737    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-jqjd8" podStartSLOduration=33.55309405 podStartE2EDuration="35.610721825s" totalImagesPullingTime="2.057627775s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 02:31:08 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:31:43.608832411 +0000 UTC m=+85.863136048" watchObservedRunningTime="2026-09-09 02:31:43.610721825 +0000 UTC m=+85.865025460"
Sep 09 02:31:43.738512 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:43.738450    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-zxcbk"]
Sep 09 02:31:43.741100 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:31:43.741079    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd19499c4_d8c2_4865_bf9b_51bcc6f46905.slice/crio-8238fa017839a64d18ee389232ee76fca1a3103d7803466edeaf772590bf679d WatchSource:0}: Error finding container 8238fa017839a64d18ee389232ee76fca1a3103d7803466edeaf772590bf679d: Status 404 returned error can't find the container with id 8238fa017839a64d18ee389232ee76fca1a3103d7803466edeaf772590bf679d
Sep 09 02:31:44.591431 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:44.591392    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-zxcbk" event={"ID":"d19499c4-d8c2-4865-bf9b-51bcc6f46905","Type":"ContainerStarted","Data":"8238fa017839a64d18ee389232ee76fca1a3103d7803466edeaf772590bf679d"}
Sep 09 02:31:45.596215 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:45.596180    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-zxcbk" event={"ID":"d19499c4-d8c2-4865-bf9b-51bcc6f46905","Type":"ContainerStarted","Data":"c6f2318b834d6e17a39abd9a1eb3b30f35b9b40aafdc5e4b0a1277dc8029d438"}
Sep 09 02:31:45.596565 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:45.596233    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-zxcbk" event={"ID":"d19499c4-d8c2-4865-bf9b-51bcc6f46905","Type":"ContainerStarted","Data":"1d8892310568fedce425fc20ee01a580ced1ad30d3960ec78150fb9385a70fd9"}
Sep 09 02:31:45.627843 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:45.627789    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-6f4946db56-zxcbk" podStartSLOduration=2.090270088 podStartE2EDuration="3.627770209s" totalImagesPullingTime="1.537500121s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:31:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:31:45.626179838 +0000 UTC m=+87.880483474" watchObservedRunningTime="2026-09-09 02:31:45.627770209 +0000 UTC m=+87.882073845"
Sep 09 02:31:47.787335 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:47.787301    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-p2sz7"]
Sep 09 02:31:47.790684 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:47.790669    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-p2sz7"
Sep 09 02:31:47.795327 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:47.795309    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 09 02:31:47.795590 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:47.795575    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-tj6kt\""
Sep 09 02:31:47.795884 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:47.795867    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 09 02:31:47.796139 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:47.796126    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 09 02:31:47.860288 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:47.860262    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/32562e97-9ba2-4110-83e6-6013cc7ab2c0-root\") pod \"node-exporter-p2sz7\" (UID: \"32562e97-9ba2-4110-83e6-6013cc7ab2c0\") " pod="openshift-monitoring/node-exporter-p2sz7"
Sep 09 02:31:47.860382 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:47.860301    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/32562e97-9ba2-4110-83e6-6013cc7ab2c0-node-exporter-textfile\") pod \"node-exporter-p2sz7\" (UID: \"32562e97-9ba2-4110-83e6-6013cc7ab2c0\") " pod="openshift-monitoring/node-exporter-p2sz7"
Sep 09 02:31:47.860382 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:47.860322    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/32562e97-9ba2-4110-83e6-6013cc7ab2c0-metrics-client-ca\") pod \"node-exporter-p2sz7\" (UID: \"32562e97-9ba2-4110-83e6-6013cc7ab2c0\") " pod="openshift-monitoring/node-exporter-p2sz7"
Sep 09 02:31:47.860472 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:47.860400    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/32562e97-9ba2-4110-83e6-6013cc7ab2c0-node-exporter-accelerators-collector-config\") pod \"node-exporter-p2sz7\" (UID: \"32562e97-9ba2-4110-83e6-6013cc7ab2c0\") " pod="openshift-monitoring/node-exporter-p2sz7"
Sep 09 02:31:47.860472 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:47.860444    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/32562e97-9ba2-4110-83e6-6013cc7ab2c0-sys\") pod \"node-exporter-p2sz7\" (UID: \"32562e97-9ba2-4110-83e6-6013cc7ab2c0\") " pod="openshift-monitoring/node-exporter-p2sz7"
Sep 09 02:31:47.860542 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:47.860476    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/32562e97-9ba2-4110-83e6-6013cc7ab2c0-node-exporter-wtmp\") pod \"node-exporter-p2sz7\" (UID: \"32562e97-9ba2-4110-83e6-6013cc7ab2c0\") " pod="openshift-monitoring/node-exporter-p2sz7"
Sep 09 02:31:47.860542 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:47.860495    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/32562e97-9ba2-4110-83e6-6013cc7ab2c0-node-exporter-tls\") pod \"node-exporter-p2sz7\" (UID: \"32562e97-9ba2-4110-83e6-6013cc7ab2c0\") " pod="openshift-monitoring/node-exporter-p2sz7"
Sep 09 02:31:47.860542 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:47.860511    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/32562e97-9ba2-4110-83e6-6013cc7ab2c0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-p2sz7\" (UID: \"32562e97-9ba2-4110-83e6-6013cc7ab2c0\") " pod="openshift-monitoring/node-exporter-p2sz7"
Sep 09 02:31:47.860542 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:47.860525    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7wtv\" (UniqueName: \"kubernetes.io/projected/32562e97-9ba2-4110-83e6-6013cc7ab2c0-kube-api-access-j7wtv\") pod \"node-exporter-p2sz7\" (UID: \"32562e97-9ba2-4110-83e6-6013cc7ab2c0\") " pod="openshift-monitoring/node-exporter-p2sz7"
Sep 09 02:31:47.961565 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:47.961525    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/32562e97-9ba2-4110-83e6-6013cc7ab2c0-node-exporter-accelerators-collector-config\") pod \"node-exporter-p2sz7\" (UID: \"32562e97-9ba2-4110-83e6-6013cc7ab2c0\") " pod="openshift-monitoring/node-exporter-p2sz7"
Sep 09 02:31:47.961752 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:47.961573    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/32562e97-9ba2-4110-83e6-6013cc7ab2c0-sys\") pod \"node-exporter-p2sz7\" (UID: \"32562e97-9ba2-4110-83e6-6013cc7ab2c0\") " pod="openshift-monitoring/node-exporter-p2sz7"
Sep 09 02:31:47.961752 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:47.961595    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/32562e97-9ba2-4110-83e6-6013cc7ab2c0-node-exporter-wtmp\") pod \"node-exporter-p2sz7\" (UID: \"32562e97-9ba2-4110-83e6-6013cc7ab2c0\") " pod="openshift-monitoring/node-exporter-p2sz7"
Sep 09 02:31:47.961752 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:47.961616    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/32562e97-9ba2-4110-83e6-6013cc7ab2c0-node-exporter-tls\") pod \"node-exporter-p2sz7\" (UID: \"32562e97-9ba2-4110-83e6-6013cc7ab2c0\") " pod="openshift-monitoring/node-exporter-p2sz7"
Sep 09 02:31:47.961752 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:47.961634    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/32562e97-9ba2-4110-83e6-6013cc7ab2c0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-p2sz7\" (UID: \"32562e97-9ba2-4110-83e6-6013cc7ab2c0\") " pod="openshift-monitoring/node-exporter-p2sz7"
Sep 09 02:31:47.961752 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:47.961651    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7wtv\" (UniqueName: \"kubernetes.io/projected/32562e97-9ba2-4110-83e6-6013cc7ab2c0-kube-api-access-j7wtv\") pod \"node-exporter-p2sz7\" (UID: \"32562e97-9ba2-4110-83e6-6013cc7ab2c0\") " pod="openshift-monitoring/node-exporter-p2sz7"
Sep 09 02:31:47.961752 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:47.961668    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/32562e97-9ba2-4110-83e6-6013cc7ab2c0-sys\") pod \"node-exporter-p2sz7\" (UID: \"32562e97-9ba2-4110-83e6-6013cc7ab2c0\") " pod="openshift-monitoring/node-exporter-p2sz7"
Sep 09 02:31:47.961752 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:47.961676    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/32562e97-9ba2-4110-83e6-6013cc7ab2c0-root\") pod \"node-exporter-p2sz7\" (UID: \"32562e97-9ba2-4110-83e6-6013cc7ab2c0\") " pod="openshift-monitoring/node-exporter-p2sz7"
Sep 09 02:31:47.961752 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:47.961712    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/32562e97-9ba2-4110-83e6-6013cc7ab2c0-root\") pod \"node-exporter-p2sz7\" (UID: \"32562e97-9ba2-4110-83e6-6013cc7ab2c0\") " pod="openshift-monitoring/node-exporter-p2sz7"
Sep 09 02:31:47.961752 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:31:47.961735    2570 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 09 02:31:47.961752 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:47.961755    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/32562e97-9ba2-4110-83e6-6013cc7ab2c0-node-exporter-textfile\") pod \"node-exporter-p2sz7\" (UID: \"32562e97-9ba2-4110-83e6-6013cc7ab2c0\") " pod="openshift-monitoring/node-exporter-p2sz7"
Sep 09 02:31:47.962227 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:47.961757    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/32562e97-9ba2-4110-83e6-6013cc7ab2c0-node-exporter-wtmp\") pod \"node-exporter-p2sz7\" (UID: \"32562e97-9ba2-4110-83e6-6013cc7ab2c0\") " pod="openshift-monitoring/node-exporter-p2sz7"
Sep 09 02:31:47.962227 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:47.961786    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/32562e97-9ba2-4110-83e6-6013cc7ab2c0-metrics-client-ca\") pod \"node-exporter-p2sz7\" (UID: \"32562e97-9ba2-4110-83e6-6013cc7ab2c0\") " pod="openshift-monitoring/node-exporter-p2sz7"
Sep 09 02:31:47.962227 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:31:47.961829    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32562e97-9ba2-4110-83e6-6013cc7ab2c0-node-exporter-tls podName:32562e97-9ba2-4110-83e6-6013cc7ab2c0 nodeName:}" failed. No retries permitted until 2026-09-09 02:31:48.46180981 +0000 UTC m=+90.716113441 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/32562e97-9ba2-4110-83e6-6013cc7ab2c0-node-exporter-tls") pod "node-exporter-p2sz7" (UID: "32562e97-9ba2-4110-83e6-6013cc7ab2c0") : secret "node-exporter-tls" not found
Sep 09 02:31:47.962227 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:47.962159    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/32562e97-9ba2-4110-83e6-6013cc7ab2c0-node-exporter-textfile\") pod \"node-exporter-p2sz7\" (UID: \"32562e97-9ba2-4110-83e6-6013cc7ab2c0\") " pod="openshift-monitoring/node-exporter-p2sz7"
Sep 09 02:31:47.962431 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:47.962342    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/32562e97-9ba2-4110-83e6-6013cc7ab2c0-metrics-client-ca\") pod \"node-exporter-p2sz7\" (UID: \"32562e97-9ba2-4110-83e6-6013cc7ab2c0\") " pod="openshift-monitoring/node-exporter-p2sz7"
Sep 09 02:31:47.962431 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:47.962422    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/32562e97-9ba2-4110-83e6-6013cc7ab2c0-node-exporter-accelerators-collector-config\") pod \"node-exporter-p2sz7\" (UID: \"32562e97-9ba2-4110-83e6-6013cc7ab2c0\") " pod="openshift-monitoring/node-exporter-p2sz7"
Sep 09 02:31:47.964088 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:47.964065    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/32562e97-9ba2-4110-83e6-6013cc7ab2c0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-p2sz7\" (UID: \"32562e97-9ba2-4110-83e6-6013cc7ab2c0\") " pod="openshift-monitoring/node-exporter-p2sz7"
Sep 09 02:31:47.972999 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:47.972965    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7wtv\" (UniqueName: \"kubernetes.io/projected/32562e97-9ba2-4110-83e6-6013cc7ab2c0-kube-api-access-j7wtv\") pod \"node-exporter-p2sz7\" (UID: \"32562e97-9ba2-4110-83e6-6013cc7ab2c0\") " pod="openshift-monitoring/node-exporter-p2sz7"
Sep 09 02:31:48.465836 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:48.465798    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/32562e97-9ba2-4110-83e6-6013cc7ab2c0-node-exporter-tls\") pod \"node-exporter-p2sz7\" (UID: \"32562e97-9ba2-4110-83e6-6013cc7ab2c0\") " pod="openshift-monitoring/node-exporter-p2sz7"
Sep 09 02:31:48.468108 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:48.468083    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/32562e97-9ba2-4110-83e6-6013cc7ab2c0-node-exporter-tls\") pod \"node-exporter-p2sz7\" (UID: \"32562e97-9ba2-4110-83e6-6013cc7ab2c0\") " pod="openshift-monitoring/node-exporter-p2sz7"
Sep 09 02:31:48.699664 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:48.699629    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-p2sz7"
Sep 09 02:31:48.714256 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:31:48.714224    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32562e97_9ba2_4110_83e6_6013cc7ab2c0.slice/crio-e7e99220869bc9faf3d3e3fff8a31df83abe2654c0e1170e1cb3bc422a9be52e WatchSource:0}: Error finding container e7e99220869bc9faf3d3e3fff8a31df83abe2654c0e1170e1cb3bc422a9be52e: Status 404 returned error can't find the container with id e7e99220869bc9faf3d3e3fff8a31df83abe2654c0e1170e1cb3bc422a9be52e
Sep 09 02:31:49.610431 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:49.610393    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-p2sz7" event={"ID":"32562e97-9ba2-4110-83e6-6013cc7ab2c0","Type":"ContainerStarted","Data":"e7e99220869bc9faf3d3e3fff8a31df83abe2654c0e1170e1cb3bc422a9be52e"}
Sep 09 02:31:50.615782 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:50.615742    2570 generic.go:356] "Generic (PLEG): container finished" podID="32562e97-9ba2-4110-83e6-6013cc7ab2c0" containerID="d98a37192305db857cfbcde7d16a24c0103350e10a7a401ae9db5444176ab00a" exitCode=0
Sep 09 02:31:50.616167 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:50.615821    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-p2sz7" event={"ID":"32562e97-9ba2-4110-83e6-6013cc7ab2c0","Type":"ContainerDied","Data":"d98a37192305db857cfbcde7d16a24c0103350e10a7a401ae9db5444176ab00a"}
Sep 09 02:31:51.621770 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:51.621735    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-p2sz7" event={"ID":"32562e97-9ba2-4110-83e6-6013cc7ab2c0","Type":"ContainerStarted","Data":"e3b928e56284c9cd365a516151661b74cc632311811247c91965d5bb65611433"}
Sep 09 02:31:51.622121 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:51.621788    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-p2sz7" event={"ID":"32562e97-9ba2-4110-83e6-6013cc7ab2c0","Type":"ContainerStarted","Data":"4c4b397df907068a68d09dcd1c91b5fd0aa785dbae7de399101f42b0cff97418"}
Sep 09 02:31:51.648994 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:51.646716    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-p2sz7" podStartSLOduration=3.554017704 podStartE2EDuration="4.646696844s" totalImagesPullingTime="1.030257457s" totalInitContainerRuntime="62.421683ms" isStatefulPod=true podCreationTimestamp="2026-09-09 02:31:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:31:51.645137101 +0000 UTC m=+93.899440739" watchObservedRunningTime="2026-09-09 02:31:51.646696844 +0000 UTC m=+93.901000483"
Sep 09 02:31:52.163740 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:52.163702    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-786f55c85c-6dh59"]
Sep 09 02:31:52.166772 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:52.166756    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-786f55c85c-6dh59"
Sep 09 02:31:52.170606 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:52.170577    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 09 02:31:52.171014 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:52.170996    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 09 02:31:52.171075 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:52.171040    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-vkxnr\""
Sep 09 02:31:52.171398 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:52.171381    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 09 02:31:52.172003 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:52.171969    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 09 02:31:52.172307 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:52.172288    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-53t0q1jiekk88\""
Sep 09 02:31:52.180384 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:52.180361    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-786f55c85c-6dh59"]
Sep 09 02:31:52.297234 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:52.297202    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60fe9c3e-cc9c-4832-b1fb-be95737a6596-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-786f55c85c-6dh59\" (UID: \"60fe9c3e-cc9c-4832-b1fb-be95737a6596\") " pod="openshift-monitoring/metrics-server-786f55c85c-6dh59"
Sep 09 02:31:52.297410 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:52.297266    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/60fe9c3e-cc9c-4832-b1fb-be95737a6596-metrics-server-audit-profiles\") pod \"metrics-server-786f55c85c-6dh59\" (UID: \"60fe9c3e-cc9c-4832-b1fb-be95737a6596\") " pod="openshift-monitoring/metrics-server-786f55c85c-6dh59"
Sep 09 02:31:52.297410 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:52.297286    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/60fe9c3e-cc9c-4832-b1fb-be95737a6596-client-ca-bundle\") pod \"metrics-server-786f55c85c-6dh59\" (UID: \"60fe9c3e-cc9c-4832-b1fb-be95737a6596\") " pod="openshift-monitoring/metrics-server-786f55c85c-6dh59"
Sep 09 02:31:52.297410 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:52.297306    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/60fe9c3e-cc9c-4832-b1fb-be95737a6596-secret-metrics-server-tls\") pod \"metrics-server-786f55c85c-6dh59\" (UID: \"60fe9c3e-cc9c-4832-b1fb-be95737a6596\") " pod="openshift-monitoring/metrics-server-786f55c85c-6dh59"
Sep 09 02:31:52.297410 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:52.297324    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/60fe9c3e-cc9c-4832-b1fb-be95737a6596-audit-log\") pod \"metrics-server-786f55c85c-6dh59\" (UID: \"60fe9c3e-cc9c-4832-b1fb-be95737a6596\") " pod="openshift-monitoring/metrics-server-786f55c85c-6dh59"
Sep 09 02:31:52.297410 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:52.297349    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4gvt\" (UniqueName: \"kubernetes.io/projected/60fe9c3e-cc9c-4832-b1fb-be95737a6596-kube-api-access-j4gvt\") pod \"metrics-server-786f55c85c-6dh59\" (UID: \"60fe9c3e-cc9c-4832-b1fb-be95737a6596\") " pod="openshift-monitoring/metrics-server-786f55c85c-6dh59"
Sep 09 02:31:52.297577 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:52.297440    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/60fe9c3e-cc9c-4832-b1fb-be95737a6596-secret-metrics-server-client-certs\") pod \"metrics-server-786f55c85c-6dh59\" (UID: \"60fe9c3e-cc9c-4832-b1fb-be95737a6596\") " pod="openshift-monitoring/metrics-server-786f55c85c-6dh59"
Sep 09 02:31:52.398726 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:52.398690    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/60fe9c3e-cc9c-4832-b1fb-be95737a6596-metrics-server-audit-profiles\") pod \"metrics-server-786f55c85c-6dh59\" (UID: \"60fe9c3e-cc9c-4832-b1fb-be95737a6596\") " pod="openshift-monitoring/metrics-server-786f55c85c-6dh59"
Sep 09 02:31:52.398726 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:52.398725    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/60fe9c3e-cc9c-4832-b1fb-be95737a6596-client-ca-bundle\") pod \"metrics-server-786f55c85c-6dh59\" (UID: \"60fe9c3e-cc9c-4832-b1fb-be95737a6596\") " pod="openshift-monitoring/metrics-server-786f55c85c-6dh59"
Sep 09 02:31:52.398968 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:52.398748    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/60fe9c3e-cc9c-4832-b1fb-be95737a6596-secret-metrics-server-tls\") pod \"metrics-server-786f55c85c-6dh59\" (UID: \"60fe9c3e-cc9c-4832-b1fb-be95737a6596\") " pod="openshift-monitoring/metrics-server-786f55c85c-6dh59"
Sep 09 02:31:52.398968 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:52.398773    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/60fe9c3e-cc9c-4832-b1fb-be95737a6596-audit-log\") pod \"metrics-server-786f55c85c-6dh59\" (UID: \"60fe9c3e-cc9c-4832-b1fb-be95737a6596\") " pod="openshift-monitoring/metrics-server-786f55c85c-6dh59"
Sep 09 02:31:52.398968 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:52.398810    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4gvt\" (UniqueName: \"kubernetes.io/projected/60fe9c3e-cc9c-4832-b1fb-be95737a6596-kube-api-access-j4gvt\") pod \"metrics-server-786f55c85c-6dh59\" (UID: \"60fe9c3e-cc9c-4832-b1fb-be95737a6596\") " pod="openshift-monitoring/metrics-server-786f55c85c-6dh59"
Sep 09 02:31:52.398968 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:52.398833    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/60fe9c3e-cc9c-4832-b1fb-be95737a6596-secret-metrics-server-client-certs\") pod \"metrics-server-786f55c85c-6dh59\" (UID: \"60fe9c3e-cc9c-4832-b1fb-be95737a6596\") " pod="openshift-monitoring/metrics-server-786f55c85c-6dh59"
Sep 09 02:31:52.398968 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:52.398861    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60fe9c3e-cc9c-4832-b1fb-be95737a6596-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-786f55c85c-6dh59\" (UID: \"60fe9c3e-cc9c-4832-b1fb-be95737a6596\") " pod="openshift-monitoring/metrics-server-786f55c85c-6dh59"
Sep 09 02:31:52.399423 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:52.399395    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/60fe9c3e-cc9c-4832-b1fb-be95737a6596-audit-log\") pod \"metrics-server-786f55c85c-6dh59\" (UID: \"60fe9c3e-cc9c-4832-b1fb-be95737a6596\") " pod="openshift-monitoring/metrics-server-786f55c85c-6dh59"
Sep 09 02:31:52.399679 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:52.399655    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60fe9c3e-cc9c-4832-b1fb-be95737a6596-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-786f55c85c-6dh59\" (UID: \"60fe9c3e-cc9c-4832-b1fb-be95737a6596\") " pod="openshift-monitoring/metrics-server-786f55c85c-6dh59"
Sep 09 02:31:52.399871 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:52.399852    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/60fe9c3e-cc9c-4832-b1fb-be95737a6596-metrics-server-audit-profiles\") pod \"metrics-server-786f55c85c-6dh59\" (UID: \"60fe9c3e-cc9c-4832-b1fb-be95737a6596\") " pod="openshift-monitoring/metrics-server-786f55c85c-6dh59"
Sep 09 02:31:52.401389 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:52.401361    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/60fe9c3e-cc9c-4832-b1fb-be95737a6596-client-ca-bundle\") pod \"metrics-server-786f55c85c-6dh59\" (UID: \"60fe9c3e-cc9c-4832-b1fb-be95737a6596\") " pod="openshift-monitoring/metrics-server-786f55c85c-6dh59"
Sep 09 02:31:52.401476 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:52.401428    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/60fe9c3e-cc9c-4832-b1fb-be95737a6596-secret-metrics-server-client-certs\") pod \"metrics-server-786f55c85c-6dh59\" (UID: \"60fe9c3e-cc9c-4832-b1fb-be95737a6596\") " pod="openshift-monitoring/metrics-server-786f55c85c-6dh59"
Sep 09 02:31:52.401587 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:52.401564    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/60fe9c3e-cc9c-4832-b1fb-be95737a6596-secret-metrics-server-tls\") pod \"metrics-server-786f55c85c-6dh59\" (UID: \"60fe9c3e-cc9c-4832-b1fb-be95737a6596\") " pod="openshift-monitoring/metrics-server-786f55c85c-6dh59"
Sep 09 02:31:52.410113 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:52.410088    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4gvt\" (UniqueName: \"kubernetes.io/projected/60fe9c3e-cc9c-4832-b1fb-be95737a6596-kube-api-access-j4gvt\") pod \"metrics-server-786f55c85c-6dh59\" (UID: \"60fe9c3e-cc9c-4832-b1fb-be95737a6596\") " pod="openshift-monitoring/metrics-server-786f55c85c-6dh59"
Sep 09 02:31:52.476297 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:52.476217    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-786f55c85c-6dh59"
Sep 09 02:31:52.757886 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:52.757862    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-786f55c85c-6dh59"]
Sep 09 02:31:52.760296 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:31:52.760271    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60fe9c3e_cc9c_4832_b1fb_be95737a6596.slice/crio-3f45ce9b8b001ea18498084fc5203791282ee5cc546c83af22e3b31c33325c7f WatchSource:0}: Error finding container 3f45ce9b8b001ea18498084fc5203791282ee5cc546c83af22e3b31c33325c7f: Status 404 returned error can't find the container with id 3f45ce9b8b001ea18498084fc5203791282ee5cc546c83af22e3b31c33325c7f
Sep 09 02:31:53.631135 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:53.631095    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-786f55c85c-6dh59" event={"ID":"60fe9c3e-cc9c-4832-b1fb-be95737a6596","Type":"ContainerStarted","Data":"3f45ce9b8b001ea18498084fc5203791282ee5cc546c83af22e3b31c33325c7f"}
Sep 09 02:31:54.637261 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:54.637221    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-786f55c85c-6dh59" event={"ID":"60fe9c3e-cc9c-4832-b1fb-be95737a6596","Type":"ContainerStarted","Data":"33ef49161f77744c2225ef57ea94ccbfa003cdf107b2e35a73c6c785691a0910"}
Sep 09 02:31:54.663286 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:54.663233    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-786f55c85c-6dh59" podStartSLOduration=1.14484961 podStartE2EDuration="2.66321806s" totalImagesPullingTime="1.51836845s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:31:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:31:54.661227182 +0000 UTC m=+96.915530819" watchObservedRunningTime="2026-09-09 02:31:54.66321806 +0000 UTC m=+96.917521697"
Sep 09 02:31:56.550485 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:31:56.550455    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-fb8975b97-h6747"
Sep 09 02:32:00.526144 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:32:00.526115    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-6gkgn"
Sep 09 02:32:03.174675 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:32:03.174637    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-85d98c998f-vslgh"]
Sep 09 02:32:03.208427 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:32:03.208392    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-vslgh"]
Sep 09 02:32:03.208601 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:32:03.208582    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-vslgh"
Sep 09 02:32:03.212109 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:32:03.212084    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 09 02:32:03.212248 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:32:03.212160    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-xpqzf\""
Sep 09 02:32:03.212248 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:32:03.212164    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 09 02:32:03.381277 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:32:03.381243    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5h4qj\" (UniqueName: \"kubernetes.io/projected/94b74aa9-c050-458e-af76-a2c72555c71b-kube-api-access-5h4qj\") pod \"downloads-85d98c998f-vslgh\" (UID: \"94b74aa9-c050-458e-af76-a2c72555c71b\") " pod="openshift-console/downloads-85d98c998f-vslgh"
Sep 09 02:32:03.482372 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:32:03.482279    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5h4qj\" (UniqueName: \"kubernetes.io/projected/94b74aa9-c050-458e-af76-a2c72555c71b-kube-api-access-5h4qj\") pod \"downloads-85d98c998f-vslgh\" (UID: \"94b74aa9-c050-458e-af76-a2c72555c71b\") " pod="openshift-console/downloads-85d98c998f-vslgh"
Sep 09 02:32:03.491985 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:32:03.491947    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5h4qj\" (UniqueName: \"kubernetes.io/projected/94b74aa9-c050-458e-af76-a2c72555c71b-kube-api-access-5h4qj\") pod \"downloads-85d98c998f-vslgh\" (UID: \"94b74aa9-c050-458e-af76-a2c72555c71b\") " pod="openshift-console/downloads-85d98c998f-vslgh"
Sep 09 02:32:03.519498 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:32:03.519472    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-vslgh"
Sep 09 02:32:03.834762 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:32:03.834731    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-vslgh"]
Sep 09 02:32:03.839656 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:32:03.839625    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94b74aa9_c050_458e_af76_a2c72555c71b.slice/crio-0a192d8739dd08243bcc97cb9da73a69c1b7792395fa0a52fca7d5c2c24e8a48 WatchSource:0}: Error finding container 0a192d8739dd08243bcc97cb9da73a69c1b7792395fa0a52fca7d5c2c24e8a48: Status 404 returned error can't find the container with id 0a192d8739dd08243bcc97cb9da73a69c1b7792395fa0a52fca7d5c2c24e8a48
Sep 09 02:32:04.674800 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:32:04.674765    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-vslgh" event={"ID":"94b74aa9-c050-458e-af76-a2c72555c71b","Type":"ContainerStarted","Data":"0a192d8739dd08243bcc97cb9da73a69c1b7792395fa0a52fca7d5c2c24e8a48"}
Sep 09 02:32:12.476328 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:32:12.476287    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-786f55c85c-6dh59"
Sep 09 02:32:12.476792 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:32:12.476365    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-786f55c85c-6dh59"
Sep 09 02:32:14.643432 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:32:14.643395    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-795f49d84b-p2245"]
Sep 09 02:32:14.646519 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:32:14.646496    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-795f49d84b-p2245"
Sep 09 02:32:14.649613 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:32:14.649591    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 09 02:32:14.649827 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:32:14.649812    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 09 02:32:14.650169 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:32:14.650148    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 09 02:32:14.651015 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:32:14.650994    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-vb7mz\""
Sep 09 02:32:14.651360 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:32:14.651342    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 09 02:32:14.651453 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:32:14.651342    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 09 02:32:14.661862 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:32:14.661842    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-795f49d84b-p2245"]
Sep 09 02:32:14.677686 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:32:14.677664    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5a37513f-bd71-4b5a-8a48-c102900df7e1-service-ca\") pod \"console-795f49d84b-p2245\" (UID: \"5a37513f-bd71-4b5a-8a48-c102900df7e1\") " pod="openshift-console/console-795f49d84b-p2245"
Sep 09 02:32:14.677810 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:32:14.677709    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5a37513f-bd71-4b5a-8a48-c102900df7e1-console-oauth-config\") pod \"console-795f49d84b-p2245\" (UID: \"5a37513f-bd71-4b5a-8a48-c102900df7e1\") " pod="openshift-console/console-795f49d84b-p2245"
Sep 09 02:32:14.677810 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:32:14.677730    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flw25\" (UniqueName: \"kubernetes.io/projected/5a37513f-bd71-4b5a-8a48-c102900df7e1-kube-api-access-flw25\") pod \"console-795f49d84b-p2245\" (UID: \"5a37513f-bd71-4b5a-8a48-c102900df7e1\") " pod="openshift-console/console-795f49d84b-p2245"
Sep 09 02:32:14.677927 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:32:14.677835    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5a37513f-bd71-4b5a-8a48-c102900df7e1-console-serving-cert\") pod \"console-795f49d84b-p2245\" (UID: \"5a37513f-bd71-4b5a-8a48-c102900df7e1\") " pod="openshift-console/console-795f49d84b-p2245"
Sep 09 02:32:14.677927 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:32:14.677874    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5a37513f-bd71-4b5a-8a48-c102900df7e1-oauth-serving-cert\") pod \"console-795f49d84b-p2245\" (UID: \"5a37513f-bd71-4b5a-8a48-c102900df7e1\") " pod="openshift-console/console-795f49d84b-p2245"
Sep 09 02:32:14.678059 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:32:14.677941    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5a37513f-bd71-4b5a-8a48-c102900df7e1-console-config\") pod \"console-795f49d84b-p2245\" (UID: \"5a37513f-bd71-4b5a-8a48-c102900df7e1\") " pod="openshift-console/console-795f49d84b-p2245"
Sep 09 02:32:14.778986 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:32:14.778945    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5a37513f-bd71-4b5a-8a48-c102900df7e1-service-ca\") pod \"console-795f49d84b-p2245\" (UID: \"5a37513f-bd71-4b5a-8a48-c102900df7e1\") " pod="openshift-console/console-795f49d84b-p2245"
Sep 09 02:32:14.779154 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:32:14.779043    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5a37513f-bd71-4b5a-8a48-c102900df7e1-console-oauth-config\") pod \"console-795f49d84b-p2245\" (UID: \"5a37513f-bd71-4b5a-8a48-c102900df7e1\") " pod="openshift-console/console-795f49d84b-p2245"
Sep 09 02:32:14.779154 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:32:14.779063    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flw25\" (UniqueName: \"kubernetes.io/projected/5a37513f-bd71-4b5a-8a48-c102900df7e1-kube-api-access-flw25\") pod \"console-795f49d84b-p2245\" (UID: \"5a37513f-bd71-4b5a-8a48-c102900df7e1\") " pod="openshift-console/console-795f49d84b-p2245"
Sep 09 02:32:14.779154 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:32:14.779109    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5a37513f-bd71-4b5a-8a48-c102900df7e1-console-serving-cert\") pod \"console-795f49d84b-p2245\" (UID: \"5a37513f-bd71-4b5a-8a48-c102900df7e1\") " pod="openshift-console/console-795f49d84b-p2245"
Sep 09 02:32:14.779154 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:32:14.779141    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5a37513f-bd71-4b5a-8a48-c102900df7e1-oauth-serving-cert\") pod \"console-795f49d84b-p2245\" (UID: \"5a37513f-bd71-4b5a-8a48-c102900df7e1\") " pod="openshift-console/console-795f49d84b-p2245"
Sep 09 02:32:14.779377 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:32:14.779190    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5a37513f-bd71-4b5a-8a48-c102900df7e1-console-config\") pod \"console-795f49d84b-p2245\" (UID: \"5a37513f-bd71-4b5a-8a48-c102900df7e1\") " pod="openshift-console/console-795f49d84b-p2245"
Sep 09 02:32:14.779803 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:32:14.779773    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5a37513f-bd71-4b5a-8a48-c102900df7e1-service-ca\") pod \"console-795f49d84b-p2245\" (UID: \"5a37513f-bd71-4b5a-8a48-c102900df7e1\") " pod="openshift-console/console-795f49d84b-p2245"
Sep 09 02:32:14.779934 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:32:14.779815    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5a37513f-bd71-4b5a-8a48-c102900df7e1-oauth-serving-cert\") pod \"console-795f49d84b-p2245\" (UID: \"5a37513f-bd71-4b5a-8a48-c102900df7e1\") " pod="openshift-console/console-795f49d84b-p2245"
Sep 09 02:32:14.780192 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:32:14.780159    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5a37513f-bd71-4b5a-8a48-c102900df7e1-console-config\") pod \"console-795f49d84b-p2245\" (UID: \"5a37513f-bd71-4b5a-8a48-c102900df7e1\") " pod="openshift-console/console-795f49d84b-p2245"
Sep 09 02:32:14.781732 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:32:14.781712    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5a37513f-bd71-4b5a-8a48-c102900df7e1-console-oauth-config\") pod \"console-795f49d84b-p2245\" (UID: \"5a37513f-bd71-4b5a-8a48-c102900df7e1\") " pod="openshift-console/console-795f49d84b-p2245"
Sep 09 02:32:14.781898 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:32:14.781872    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5a37513f-bd71-4b5a-8a48-c102900df7e1-console-serving-cert\") pod \"console-795f49d84b-p2245\" (UID: \"5a37513f-bd71-4b5a-8a48-c102900df7e1\") " pod="openshift-console/console-795f49d84b-p2245"
Sep 09 02:32:14.788376 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:32:14.788355    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-flw25\" (UniqueName: \"kubernetes.io/projected/5a37513f-bd71-4b5a-8a48-c102900df7e1-kube-api-access-flw25\") pod \"console-795f49d84b-p2245\" (UID: \"5a37513f-bd71-4b5a-8a48-c102900df7e1\") " pod="openshift-console/console-795f49d84b-p2245"
Sep 09 02:32:14.958471 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:32:14.958363    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-795f49d84b-p2245"
Sep 09 02:32:19.952126 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:32:19.952028    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-795f49d84b-p2245"]
Sep 09 02:32:19.955171 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:32:19.955143    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a37513f_bd71_4b5a_8a48_c102900df7e1.slice/crio-3595dfe58321703506214be92f23ed852340f0b5e0bd20633db8471ff9a20dd1 WatchSource:0}: Error finding container 3595dfe58321703506214be92f23ed852340f0b5e0bd20633db8471ff9a20dd1: Status 404 returned error can't find the container with id 3595dfe58321703506214be92f23ed852340f0b5e0bd20633db8471ff9a20dd1
Sep 09 02:32:20.744793 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:32:20.744747    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-vslgh" event={"ID":"94b74aa9-c050-458e-af76-a2c72555c71b","Type":"ContainerStarted","Data":"983791cf2edfff849b35e785ff88b6a9fbb3ea5d773b2bbac0411fa50d6a87a5"}
Sep 09 02:32:20.745295 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:32:20.745264    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-85d98c998f-vslgh"
Sep 09 02:32:20.747896 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:32:20.747863    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-795f49d84b-p2245" event={"ID":"5a37513f-bd71-4b5a-8a48-c102900df7e1","Type":"ContainerStarted","Data":"3595dfe58321703506214be92f23ed852340f0b5e0bd20633db8471ff9a20dd1"}
Sep 09 02:32:20.756610 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:32:20.756572    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-85d98c998f-vslgh"
Sep 09 02:32:20.770365 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:32:20.770315    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-85d98c998f-vslgh" podStartSLOduration=1.830323159 podStartE2EDuration="17.770295113s" totalImagesPullingTime="15.939971954s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:32:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:32:20.768065403 +0000 UTC m=+123.022369063" watchObservedRunningTime="2026-09-09 02:32:20.770295113 +0000 UTC m=+123.024598750"
Sep 09 02:32:23.766661 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:32:23.766584    2570 generic.go:356] "Generic (PLEG): container finished" podID="5a719fd4-b0a2-4fd8-9c0c-ac89d3f340dc" containerID="2e35c5f282af737f39d62b57ade0154823e79bd6f4bfc18c084f8214f8939407" exitCode=0
Sep 09 02:32:23.767150 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:32:23.766673    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-lp6p9" event={"ID":"5a719fd4-b0a2-4fd8-9c0c-ac89d3f340dc","Type":"ContainerDied","Data":"2e35c5f282af737f39d62b57ade0154823e79bd6f4bfc18c084f8214f8939407"}
Sep 09 02:32:23.767150 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:32:23.767059    2570 scope.go:117] "RemoveContainer" containerID="2e35c5f282af737f39d62b57ade0154823e79bd6f4bfc18c084f8214f8939407"
Sep 09 02:32:24.375089 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:32:24.375053    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-588b6849c-2vqlt"]
Sep 09 02:32:24.396108 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:32:24.396082    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-588b6849c-2vqlt"]
Sep 09 02:32:24.396271 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:32:24.396192    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-588b6849c-2vqlt"
Sep 09 02:32:24.406034 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:32:24.406010    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 09 02:32:24.469069 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:32:24.469039    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b2b65f59-2307-419b-ab4f-10a87a94e4e0-console-config\") pod \"console-588b6849c-2vqlt\" (UID: \"b2b65f59-2307-419b-ab4f-10a87a94e4e0\") " pod="openshift-console/console-588b6849c-2vqlt"
Sep 09 02:32:24.469222 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:32:24.469081    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b2b65f59-2307-419b-ab4f-10a87a94e4e0-console-oauth-config\") pod \"console-588b6849c-2vqlt\" (UID: \"b2b65f59-2307-419b-ab4f-10a87a94e4e0\") " pod="openshift-console/console-588b6849c-2vqlt"
Sep 09 02:32:24.469222 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:32:24.469151    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b2b65f59-2307-419b-ab4f-10a87a94e4e0-oauth-serving-cert\") pod \"console-588b6849c-2vqlt\" (UID: \"b2b65f59-2307-419b-ab4f-10a87a94e4e0\") " pod="openshift-console/console-588b6849c-2vqlt"
Sep 09 02:32:24.469222 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:32:24.469174    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b2b65f59-2307-419b-ab4f-10a87a94e4e0-service-ca\") pod \"console-588b6849c-2vqlt\" (UID: \"b2b65f59-2307-419b-ab4f-10a87a94e4e0\") " pod="openshift-console/console-588b6849c-2vqlt"
Sep 09 02:32:24.469350 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:32:24.469232    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b2b65f59-2307-419b-ab4f-10a87a94e4e0-console-serving-cert\") pod \"console-588b6849c-2vqlt\" (UID: \"b2b65f59-2307-419b-ab4f-10a87a94e4e0\") " pod="openshift-console/console-588b6849c-2vqlt"
Sep 09 02:32:24.469350 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:32:24.469257    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b2b65f59-2307-419b-ab4f-10a87a94e4e0-trusted-ca-bundle\") pod \"console-588b6849c-2vqlt\" (UID: \"b2b65f59-2307-419b-ab4f-10a87a94e4e0\") " pod="openshift-console/console-588b6849c-2vqlt"
Sep 09 02:32:24.469350 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:32:24.469282    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbhbg\" (UniqueName: \"kubernetes.io/projected/b2b65f59-2307-419b-ab4f-10a87a94e4e0-kube-api-access-zbhbg\") pod \"console-588b6849c-2vqlt\" (UID: \"b2b65f59-2307-419b-ab4f-10a87a94e4e0\") " pod="openshift-console/console-588b6849c-2vqlt"
Sep 09 02:32:24.570335 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:32:24.570301    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b2b65f59-2307-419b-ab4f-10a87a94e4e0-trusted-ca-bundle\") pod \"console-588b6849c-2vqlt\" (UID: \"b2b65f59-2307-419b-ab4f-10a87a94e4e0\") " pod="openshift-console/console-588b6849c-2vqlt"
Sep 09 02:32:24.570516 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:32:24.570340    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbhbg\" (UniqueName: \"kubernetes.io/projected/b2b65f59-2307-419b-ab4f-10a87a94e4e0-kube-api-access-zbhbg\") pod \"console-588b6849c-2vqlt\" (UID: \"b2b65f59-2307-419b-ab4f-10a87a94e4e0\") " pod="openshift-console/console-588b6849c-2vqlt"
Sep 09 02:32:24.570516 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:32:24.570381    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b2b65f59-2307-419b-ab4f-10a87a94e4e0-console-config\") pod \"console-588b6849c-2vqlt\" (UID: \"b2b65f59-2307-419b-ab4f-10a87a94e4e0\") " pod="openshift-console/console-588b6849c-2vqlt"
Sep 09 02:32:24.570516 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:32:24.570406    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b2b65f59-2307-419b-ab4f-10a87a94e4e0-console-oauth-config\") pod \"console-588b6849c-2vqlt\" (UID: \"b2b65f59-2307-419b-ab4f-10a87a94e4e0\") " pod="openshift-console/console-588b6849c-2vqlt"
Sep 09 02:32:24.570516 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:32:24.570452    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b2b65f59-2307-419b-ab4f-10a87a94e4e0-oauth-serving-cert\") pod \"console-588b6849c-2vqlt\" (UID: \"b2b65f59-2307-419b-ab4f-10a87a94e4e0\") " pod="openshift-console/console-588b6849c-2vqlt"
Sep 09 02:32:24.570516 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:32:24.570478    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b2b65f59-2307-419b-ab4f-10a87a94e4e0-service-ca\") pod \"console-588b6849c-2vqlt\" (UID: \"b2b65f59-2307-419b-ab4f-10a87a94e4e0\") " pod="openshift-console/console-588b6849c-2vqlt"
Sep 09 02:32:24.570779 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:32:24.570554    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b2b65f59-2307-419b-ab4f-10a87a94e4e0-console-serving-cert\") pod \"console-588b6849c-2vqlt\" (UID: \"b2b65f59-2307-419b-ab4f-10a87a94e4e0\") " pod="openshift-console/console-588b6849c-2vqlt"
Sep 09 02:32:24.571212 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:32:24.571184    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b2b65f59-2307-419b-ab4f-10a87a94e4e0-oauth-serving-cert\") pod \"console-588b6849c-2vqlt\" (UID: \"b2b65f59-2307-419b-ab4f-10a87a94e4e0\") " pod="openshift-console/console-588b6849c-2vqlt"
Sep 09 02:32:24.571427 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:32:24.571404    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b2b65f59-2307-419b-ab4f-10a87a94e4e0-trusted-ca-bundle\") pod \"console-588b6849c-2vqlt\" (UID: \"b2b65f59-2307-419b-ab4f-10a87a94e4e0\") " pod="openshift-console/console-588b6849c-2vqlt"
Sep 09 02:32:24.571807 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:32:24.571761    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b2b65f59-2307-419b-ab4f-10a87a94e4e0-console-config\") pod \"console-588b6849c-2vqlt\" (UID: \"b2b65f59-2307-419b-ab4f-10a87a94e4e0\") " pod="openshift-console/console-588b6849c-2vqlt"
Sep 09 02:32:24.571899 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:32:24.571843    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b2b65f59-2307-419b-ab4f-10a87a94e4e0-service-ca\") pod \"console-588b6849c-2vqlt\" (UID: \"b2b65f59-2307-419b-ab4f-10a87a94e4e0\") " pod="openshift-console/console-588b6849c-2vqlt"
Sep 09 02:32:24.573442 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:32:24.573416    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b2b65f59-2307-419b-ab4f-10a87a94e4e0-console-serving-cert\") pod \"console-588b6849c-2vqlt\" (UID: \"b2b65f59-2307-419b-ab4f-10a87a94e4e0\") " pod="openshift-console/console-588b6849c-2vqlt"
Sep 09 02:32:24.573542 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:32:24.573519    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b2b65f59-2307-419b-ab4f-10a87a94e4e0-console-oauth-config\") pod \"console-588b6849c-2vqlt\" (UID: \"b2b65f59-2307-419b-ab4f-10a87a94e4e0\") " pod="openshift-console/console-588b6849c-2vqlt"
Sep 09 02:32:24.585578 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:32:24.585555    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbhbg\" (UniqueName: \"kubernetes.io/projected/b2b65f59-2307-419b-ab4f-10a87a94e4e0-kube-api-access-zbhbg\") pod \"console-588b6849c-2vqlt\" (UID: \"b2b65f59-2307-419b-ab4f-10a87a94e4e0\") " pod="openshift-console/console-588b6849c-2vqlt"
Sep 09 02:32:24.709030 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:32:24.708905    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-588b6849c-2vqlt"
Sep 09 02:32:24.775261 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:32:24.774648    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-795f49d84b-p2245" event={"ID":"5a37513f-bd71-4b5a-8a48-c102900df7e1","Type":"ContainerStarted","Data":"a6c529ad9194f50a60f9d6adc2bd5c935ab9d07212ae67d4daebaeefc8ed73f7"}
Sep 09 02:32:24.779522 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:32:24.779371    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-lp6p9" event={"ID":"5a719fd4-b0a2-4fd8-9c0c-ac89d3f340dc","Type":"ContainerStarted","Data":"4020cf2686ada827fcf6c1d1cd1009bdeca216e296685f142fc8398b867f65b4"}
Sep 09 02:32:24.798692 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:32:24.798593    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-795f49d84b-p2245" podStartSLOduration=6.922819003 podStartE2EDuration="10.798573375s" totalImagesPullingTime="3.875754372s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:32:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:32:24.797226935 +0000 UTC m=+127.051530578" watchObservedRunningTime="2026-09-09 02:32:24.798573375 +0000 UTC m=+127.052877011"
Sep 09 02:32:24.959337 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:32:24.959259    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-795f49d84b-p2245"
Sep 09 02:32:24.959337 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:32:24.959329    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-795f49d84b-p2245"
Sep 09 02:32:24.964611 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:32:24.964588    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-795f49d84b-p2245"
Sep 09 02:32:25.020683 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:32:25.020652    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-588b6849c-2vqlt"]
Sep 09 02:32:25.023057 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:32:25.023028    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2b65f59_2307_419b_ab4f_10a87a94e4e0.slice/crio-5da4181e961094d70e0899f0b9f93c44da7eff5454c09539f3cad0cc949f4e17 WatchSource:0}: Error finding container 5da4181e961094d70e0899f0b9f93c44da7eff5454c09539f3cad0cc949f4e17: Status 404 returned error can't find the container with id 5da4181e961094d70e0899f0b9f93c44da7eff5454c09539f3cad0cc949f4e17
Sep 09 02:32:25.785309 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:32:25.785268    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-588b6849c-2vqlt" event={"ID":"b2b65f59-2307-419b-ab4f-10a87a94e4e0","Type":"ContainerStarted","Data":"e5a2929ad6458fe44a86e25954c94e3394639d042aa5c68afed4fe00b304ffd7"}
Sep 09 02:32:25.785867 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:32:25.785361    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-588b6849c-2vqlt" event={"ID":"b2b65f59-2307-419b-ab4f-10a87a94e4e0","Type":"ContainerStarted","Data":"5da4181e961094d70e0899f0b9f93c44da7eff5454c09539f3cad0cc949f4e17"}
Sep 09 02:32:25.790492 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:32:25.790465    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-795f49d84b-p2245"
Sep 09 02:32:25.814230 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:32:25.814173    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-588b6849c-2vqlt" podStartSLOduration=1.81415843 podStartE2EDuration="1.81415843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:32:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:32:25.812672333 +0000 UTC m=+128.066975971" watchObservedRunningTime="2026-09-09 02:32:25.81415843 +0000 UTC m=+128.068462067"
Sep 09 02:32:32.481899 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:32:32.481866    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-786f55c85c-6dh59"
Sep 09 02:32:32.485710 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:32:32.485689    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-786f55c85c-6dh59"
Sep 09 02:32:34.709750 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:32:34.709715    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-588b6849c-2vqlt"
Sep 09 02:32:34.710228 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:32:34.709779    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-588b6849c-2vqlt"
Sep 09 02:32:34.714474 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:32:34.714445    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-588b6849c-2vqlt"
Sep 09 02:32:34.826526 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:32:34.826499    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-588b6849c-2vqlt"
Sep 09 02:32:34.886909 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:32:34.886872    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-795f49d84b-p2245"]
Sep 09 02:32:38.837636 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:32:38.837603    2570 generic.go:356] "Generic (PLEG): container finished" podID="a00f5e8a-987c-4c31-a8a9-b6a9d713d064" containerID="0e145cb0d80b764b4ba8370225882eacd153809cfbf6e321ea59a732d2bc0402" exitCode=0
Sep 09 02:32:38.838008 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:32:38.837674    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-85dqd" event={"ID":"a00f5e8a-987c-4c31-a8a9-b6a9d713d064","Type":"ContainerDied","Data":"0e145cb0d80b764b4ba8370225882eacd153809cfbf6e321ea59a732d2bc0402"}
Sep 09 02:32:38.838063 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:32:38.838047    2570 scope.go:117] "RemoveContainer" containerID="0e145cb0d80b764b4ba8370225882eacd153809cfbf6e321ea59a732d2bc0402"
Sep 09 02:32:39.843412 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:32:39.843378    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-85dqd" event={"ID":"a00f5e8a-987c-4c31-a8a9-b6a9d713d064","Type":"ContainerStarted","Data":"5a7054f4ce16a8040c6a210a667614014fbfd885b74bf49fdcb6e4c9cdc81a4e"}
Sep 09 02:32:59.906656 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:32:59.906590    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-795f49d84b-p2245" podUID="5a37513f-bd71-4b5a-8a48-c102900df7e1" containerName="console" containerID="cri-o://a6c529ad9194f50a60f9d6adc2bd5c935ab9d07212ae67d4daebaeefc8ed73f7" gracePeriod=15
Sep 09 02:33:00.168468 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:33:00.168409    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-795f49d84b-p2245"
Sep 09 02:33:00.252545 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:33:00.252517    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5a37513f-bd71-4b5a-8a48-c102900df7e1-console-oauth-config\") pod \"5a37513f-bd71-4b5a-8a48-c102900df7e1\" (UID: \"5a37513f-bd71-4b5a-8a48-c102900df7e1\") "
Sep 09 02:33:00.252682 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:33:00.252586    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5a37513f-bd71-4b5a-8a48-c102900df7e1-oauth-serving-cert\") pod \"5a37513f-bd71-4b5a-8a48-c102900df7e1\" (UID: \"5a37513f-bd71-4b5a-8a48-c102900df7e1\") "
Sep 09 02:33:00.252682 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:33:00.252633    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-flw25\" (UniqueName: \"kubernetes.io/projected/5a37513f-bd71-4b5a-8a48-c102900df7e1-kube-api-access-flw25\") pod \"5a37513f-bd71-4b5a-8a48-c102900df7e1\" (UID: \"5a37513f-bd71-4b5a-8a48-c102900df7e1\") "
Sep 09 02:33:00.252682 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:33:00.252674    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5a37513f-bd71-4b5a-8a48-c102900df7e1-console-serving-cert\") pod \"5a37513f-bd71-4b5a-8a48-c102900df7e1\" (UID: \"5a37513f-bd71-4b5a-8a48-c102900df7e1\") "
Sep 09 02:33:00.252852 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:33:00.252707    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5a37513f-bd71-4b5a-8a48-c102900df7e1-console-config\") pod \"5a37513f-bd71-4b5a-8a48-c102900df7e1\" (UID: \"5a37513f-bd71-4b5a-8a48-c102900df7e1\") "
Sep 09 02:33:00.252852 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:33:00.252758    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5a37513f-bd71-4b5a-8a48-c102900df7e1-service-ca\") pod \"5a37513f-bd71-4b5a-8a48-c102900df7e1\" (UID: \"5a37513f-bd71-4b5a-8a48-c102900df7e1\") "
Sep 09 02:33:00.253100 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:33:00.253058    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a37513f-bd71-4b5a-8a48-c102900df7e1-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "5a37513f-bd71-4b5a-8a48-c102900df7e1" (UID: "5a37513f-bd71-4b5a-8a48-c102900df7e1"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 02:33:00.253229 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:33:00.253199    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a37513f-bd71-4b5a-8a48-c102900df7e1-console-config" (OuterVolumeSpecName: "console-config") pod "5a37513f-bd71-4b5a-8a48-c102900df7e1" (UID: "5a37513f-bd71-4b5a-8a48-c102900df7e1"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 02:33:00.253294 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:33:00.253267    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a37513f-bd71-4b5a-8a48-c102900df7e1-service-ca" (OuterVolumeSpecName: "service-ca") pod "5a37513f-bd71-4b5a-8a48-c102900df7e1" (UID: "5a37513f-bd71-4b5a-8a48-c102900df7e1"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 02:33:00.254876 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:33:00.254854    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a37513f-bd71-4b5a-8a48-c102900df7e1-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "5a37513f-bd71-4b5a-8a48-c102900df7e1" (UID: "5a37513f-bd71-4b5a-8a48-c102900df7e1"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 02:33:00.254953 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:33:00.254885    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a37513f-bd71-4b5a-8a48-c102900df7e1-kube-api-access-flw25" (OuterVolumeSpecName: "kube-api-access-flw25") pod "5a37513f-bd71-4b5a-8a48-c102900df7e1" (UID: "5a37513f-bd71-4b5a-8a48-c102900df7e1"). InnerVolumeSpecName "kube-api-access-flw25". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 02:33:00.255027 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:33:00.254958    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a37513f-bd71-4b5a-8a48-c102900df7e1-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "5a37513f-bd71-4b5a-8a48-c102900df7e1" (UID: "5a37513f-bd71-4b5a-8a48-c102900df7e1"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 02:33:00.353678 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:33:00.353642    2570 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5a37513f-bd71-4b5a-8a48-c102900df7e1-console-oauth-config\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:33:00.353678 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:33:00.353675    2570 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5a37513f-bd71-4b5a-8a48-c102900df7e1-oauth-serving-cert\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:33:00.353888 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:33:00.353688    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flw25\" (UniqueName: \"kubernetes.io/projected/5a37513f-bd71-4b5a-8a48-c102900df7e1-kube-api-access-flw25\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:33:00.353888 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:33:00.353702    2570 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5a37513f-bd71-4b5a-8a48-c102900df7e1-console-serving-cert\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:33:00.353888 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:33:00.353715    2570 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5a37513f-bd71-4b5a-8a48-c102900df7e1-console-config\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:33:00.353888 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:33:00.353725    2570 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5a37513f-bd71-4b5a-8a48-c102900df7e1-service-ca\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:33:00.920419 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:33:00.920385    2570 generic.go:356] "Generic (PLEG): container finished" podID="5a37513f-bd71-4b5a-8a48-c102900df7e1" containerID="a6c529ad9194f50a60f9d6adc2bd5c935ab9d07212ae67d4daebaeefc8ed73f7" exitCode=0
Sep 09 02:33:00.920822 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:33:00.920426    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-795f49d84b-p2245"
Sep 09 02:33:00.920822 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:33:00.920472    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-795f49d84b-p2245" event={"ID":"5a37513f-bd71-4b5a-8a48-c102900df7e1","Type":"ContainerDied","Data":"a6c529ad9194f50a60f9d6adc2bd5c935ab9d07212ae67d4daebaeefc8ed73f7"}
Sep 09 02:33:00.920822 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:33:00.920526    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-795f49d84b-p2245" event={"ID":"5a37513f-bd71-4b5a-8a48-c102900df7e1","Type":"ContainerDied","Data":"3595dfe58321703506214be92f23ed852340f0b5e0bd20633db8471ff9a20dd1"}
Sep 09 02:33:00.920822 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:33:00.920546    2570 scope.go:117] "RemoveContainer" containerID="a6c529ad9194f50a60f9d6adc2bd5c935ab9d07212ae67d4daebaeefc8ed73f7"
Sep 09 02:33:00.937900 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:33:00.937883    2570 scope.go:117] "RemoveContainer" containerID="a6c529ad9194f50a60f9d6adc2bd5c935ab9d07212ae67d4daebaeefc8ed73f7"
Sep 09 02:33:00.938230 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:33:00.938206    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6c529ad9194f50a60f9d6adc2bd5c935ab9d07212ae67d4daebaeefc8ed73f7\": container with ID starting with a6c529ad9194f50a60f9d6adc2bd5c935ab9d07212ae67d4daebaeefc8ed73f7 not found: ID does not exist" containerID="a6c529ad9194f50a60f9d6adc2bd5c935ab9d07212ae67d4daebaeefc8ed73f7"
Sep 09 02:33:00.938294 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:33:00.938238    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6c529ad9194f50a60f9d6adc2bd5c935ab9d07212ae67d4daebaeefc8ed73f7"} err="failed to get container status \"a6c529ad9194f50a60f9d6adc2bd5c935ab9d07212ae67d4daebaeefc8ed73f7\": rpc error: code = NotFound desc = could not find container \"a6c529ad9194f50a60f9d6adc2bd5c935ab9d07212ae67d4daebaeefc8ed73f7\": container with ID starting with a6c529ad9194f50a60f9d6adc2bd5c935ab9d07212ae67d4daebaeefc8ed73f7 not found: ID does not exist"
Sep 09 02:33:00.951038 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:33:00.951018    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-795f49d84b-p2245"]
Sep 09 02:33:00.956416 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:33:00.956395    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-795f49d84b-p2245"]
Sep 09 02:33:02.221415 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:33:02.221386    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a37513f-bd71-4b5a-8a48-c102900df7e1" path="/var/lib/kubelet/pods/5a37513f-bd71-4b5a-8a48-c102900df7e1/volumes"
Sep 09 02:33:15.597333 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:33:15.597251    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-55975b658b-kd4nx"]
Sep 09 02:33:15.597755 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:33:15.597520    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a37513f-bd71-4b5a-8a48-c102900df7e1" containerName="console"
Sep 09 02:33:15.597755 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:33:15.597531    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a37513f-bd71-4b5a-8a48-c102900df7e1" containerName="console"
Sep 09 02:33:15.597755 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:33:15.597582    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a37513f-bd71-4b5a-8a48-c102900df7e1" containerName="console"
Sep 09 02:33:15.599211 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:33:15.599195    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55975b658b-kd4nx"
Sep 09 02:33:15.615604 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:33:15.615572    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-55975b658b-kd4nx"]
Sep 09 02:33:15.675224 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:33:15.675196    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b23d83d0-d57e-4ab4-821e-6ce90d556121-console-config\") pod \"console-55975b658b-kd4nx\" (UID: \"b23d83d0-d57e-4ab4-821e-6ce90d556121\") " pod="openshift-console/console-55975b658b-kd4nx"
Sep 09 02:33:15.675356 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:33:15.675228    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b23d83d0-d57e-4ab4-821e-6ce90d556121-service-ca\") pod \"console-55975b658b-kd4nx\" (UID: \"b23d83d0-d57e-4ab4-821e-6ce90d556121\") " pod="openshift-console/console-55975b658b-kd4nx"
Sep 09 02:33:15.675356 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:33:15.675269    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b23d83d0-d57e-4ab4-821e-6ce90d556121-console-serving-cert\") pod \"console-55975b658b-kd4nx\" (UID: \"b23d83d0-d57e-4ab4-821e-6ce90d556121\") " pod="openshift-console/console-55975b658b-kd4nx"
Sep 09 02:33:15.675356 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:33:15.675292    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b23d83d0-d57e-4ab4-821e-6ce90d556121-trusted-ca-bundle\") pod \"console-55975b658b-kd4nx\" (UID: \"b23d83d0-d57e-4ab4-821e-6ce90d556121\") " pod="openshift-console/console-55975b658b-kd4nx"
Sep 09 02:33:15.675356 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:33:15.675309    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b23d83d0-d57e-4ab4-821e-6ce90d556121-console-oauth-config\") pod \"console-55975b658b-kd4nx\" (UID: \"b23d83d0-d57e-4ab4-821e-6ce90d556121\") " pod="openshift-console/console-55975b658b-kd4nx"
Sep 09 02:33:15.675356 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:33:15.675330    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b23d83d0-d57e-4ab4-821e-6ce90d556121-oauth-serving-cert\") pod \"console-55975b658b-kd4nx\" (UID: \"b23d83d0-d57e-4ab4-821e-6ce90d556121\") " pod="openshift-console/console-55975b658b-kd4nx"
Sep 09 02:33:15.675512 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:33:15.675405    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vb668\" (UniqueName: \"kubernetes.io/projected/b23d83d0-d57e-4ab4-821e-6ce90d556121-kube-api-access-vb668\") pod \"console-55975b658b-kd4nx\" (UID: \"b23d83d0-d57e-4ab4-821e-6ce90d556121\") " pod="openshift-console/console-55975b658b-kd4nx"
Sep 09 02:33:15.776359 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:33:15.776312    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb668\" (UniqueName: \"kubernetes.io/projected/b23d83d0-d57e-4ab4-821e-6ce90d556121-kube-api-access-vb668\") pod \"console-55975b658b-kd4nx\" (UID: \"b23d83d0-d57e-4ab4-821e-6ce90d556121\") " pod="openshift-console/console-55975b658b-kd4nx"
Sep 09 02:33:15.776496 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:33:15.776382    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b23d83d0-d57e-4ab4-821e-6ce90d556121-console-config\") pod \"console-55975b658b-kd4nx\" (UID: \"b23d83d0-d57e-4ab4-821e-6ce90d556121\") " pod="openshift-console/console-55975b658b-kd4nx"
Sep 09 02:33:15.776496 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:33:15.776398    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b23d83d0-d57e-4ab4-821e-6ce90d556121-service-ca\") pod \"console-55975b658b-kd4nx\" (UID: \"b23d83d0-d57e-4ab4-821e-6ce90d556121\") " pod="openshift-console/console-55975b658b-kd4nx"
Sep 09 02:33:15.776496 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:33:15.776427    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b23d83d0-d57e-4ab4-821e-6ce90d556121-console-serving-cert\") pod \"console-55975b658b-kd4nx\" (UID: \"b23d83d0-d57e-4ab4-821e-6ce90d556121\") " pod="openshift-console/console-55975b658b-kd4nx"
Sep 09 02:33:15.776496 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:33:15.776448    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b23d83d0-d57e-4ab4-821e-6ce90d556121-trusted-ca-bundle\") pod \"console-55975b658b-kd4nx\" (UID: \"b23d83d0-d57e-4ab4-821e-6ce90d556121\") " pod="openshift-console/console-55975b658b-kd4nx"
Sep 09 02:33:15.776496 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:33:15.776470    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b23d83d0-d57e-4ab4-821e-6ce90d556121-console-oauth-config\") pod \"console-55975b658b-kd4nx\" (UID: \"b23d83d0-d57e-4ab4-821e-6ce90d556121\") " pod="openshift-console/console-55975b658b-kd4nx"
Sep 09 02:33:15.776496 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:33:15.776493    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b23d83d0-d57e-4ab4-821e-6ce90d556121-oauth-serving-cert\") pod \"console-55975b658b-kd4nx\" (UID: \"b23d83d0-d57e-4ab4-821e-6ce90d556121\") " pod="openshift-console/console-55975b658b-kd4nx"
Sep 09 02:33:15.777238 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:33:15.777212    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b23d83d0-d57e-4ab4-821e-6ce90d556121-service-ca\") pod \"console-55975b658b-kd4nx\" (UID: \"b23d83d0-d57e-4ab4-821e-6ce90d556121\") " pod="openshift-console/console-55975b658b-kd4nx"
Sep 09 02:33:15.777238 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:33:15.777219    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b23d83d0-d57e-4ab4-821e-6ce90d556121-oauth-serving-cert\") pod \"console-55975b658b-kd4nx\" (UID: \"b23d83d0-d57e-4ab4-821e-6ce90d556121\") " pod="openshift-console/console-55975b658b-kd4nx"
Sep 09 02:33:15.777390 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:33:15.777302    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b23d83d0-d57e-4ab4-821e-6ce90d556121-console-config\") pod \"console-55975b658b-kd4nx\" (UID: \"b23d83d0-d57e-4ab4-821e-6ce90d556121\") " pod="openshift-console/console-55975b658b-kd4nx"
Sep 09 02:33:15.777427 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:33:15.777397    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b23d83d0-d57e-4ab4-821e-6ce90d556121-trusted-ca-bundle\") pod \"console-55975b658b-kd4nx\" (UID: \"b23d83d0-d57e-4ab4-821e-6ce90d556121\") " pod="openshift-console/console-55975b658b-kd4nx"
Sep 09 02:33:15.779027 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:33:15.779009    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b23d83d0-d57e-4ab4-821e-6ce90d556121-console-serving-cert\") pod \"console-55975b658b-kd4nx\" (UID: \"b23d83d0-d57e-4ab4-821e-6ce90d556121\") " pod="openshift-console/console-55975b658b-kd4nx"
Sep 09 02:33:15.779085 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:33:15.779046    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b23d83d0-d57e-4ab4-821e-6ce90d556121-console-oauth-config\") pod \"console-55975b658b-kd4nx\" (UID: \"b23d83d0-d57e-4ab4-821e-6ce90d556121\") " pod="openshift-console/console-55975b658b-kd4nx"
Sep 09 02:33:15.785227 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:33:15.785208    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vb668\" (UniqueName: \"kubernetes.io/projected/b23d83d0-d57e-4ab4-821e-6ce90d556121-kube-api-access-vb668\") pod \"console-55975b658b-kd4nx\" (UID: \"b23d83d0-d57e-4ab4-821e-6ce90d556121\") " pod="openshift-console/console-55975b658b-kd4nx"
Sep 09 02:33:15.908444 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:33:15.908384    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55975b658b-kd4nx"
Sep 09 02:33:16.182113 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:33:16.182085    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-55975b658b-kd4nx"]
Sep 09 02:33:16.184016 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:33:16.183967    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb23d83d0_d57e_4ab4_821e_6ce90d556121.slice/crio-2cd7df21616f474abc61d433adf4aef6586f6c34a8942c4df9f6f1b02caa6952 WatchSource:0}: Error finding container 2cd7df21616f474abc61d433adf4aef6586f6c34a8942c4df9f6f1b02caa6952: Status 404 returned error can't find the container with id 2cd7df21616f474abc61d433adf4aef6586f6c34a8942c4df9f6f1b02caa6952
Sep 09 02:33:16.978538 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:33:16.978502    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55975b658b-kd4nx" event={"ID":"b23d83d0-d57e-4ab4-821e-6ce90d556121","Type":"ContainerStarted","Data":"35e54519ae921ba75bd428b72e30435e1cb0064dc426ecc1a1e16fd10e787ca2"}
Sep 09 02:33:16.978928 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:33:16.978558    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55975b658b-kd4nx" event={"ID":"b23d83d0-d57e-4ab4-821e-6ce90d556121","Type":"ContainerStarted","Data":"2cd7df21616f474abc61d433adf4aef6586f6c34a8942c4df9f6f1b02caa6952"}
Sep 09 02:33:17.005242 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:33:17.005198    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-55975b658b-kd4nx" podStartSLOduration=2.005184452 podStartE2EDuration="2.005184452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:33:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:33:17.003328617 +0000 UTC m=+179.257632253" watchObservedRunningTime="2026-09-09 02:33:17.005184452 +0000 UTC m=+179.259488087"
Sep 09 02:33:25.909526 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:33:25.909493    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-55975b658b-kd4nx"
Sep 09 02:33:25.910013 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:33:25.909553    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-55975b658b-kd4nx"
Sep 09 02:33:25.914137 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:33:25.914114    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-55975b658b-kd4nx"
Sep 09 02:33:26.014742 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:33:26.014716    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-55975b658b-kd4nx"
Sep 09 02:33:26.078994 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:33:26.078949    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-588b6849c-2vqlt"]
Sep 09 02:33:51.099493 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:33:51.099442    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-588b6849c-2vqlt" podUID="b2b65f59-2307-419b-ab4f-10a87a94e4e0" containerName="console" containerID="cri-o://e5a2929ad6458fe44a86e25954c94e3394639d042aa5c68afed4fe00b304ffd7" gracePeriod=15
Sep 09 02:33:51.332378 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:33:51.332355    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-588b6849c-2vqlt"
Sep 09 02:33:51.429536 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:33:51.429463    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b2b65f59-2307-419b-ab4f-10a87a94e4e0-console-oauth-config\") pod \"b2b65f59-2307-419b-ab4f-10a87a94e4e0\" (UID: \"b2b65f59-2307-419b-ab4f-10a87a94e4e0\") "
Sep 09 02:33:51.429536 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:33:51.429527    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b2b65f59-2307-419b-ab4f-10a87a94e4e0-console-serving-cert\") pod \"b2b65f59-2307-419b-ab4f-10a87a94e4e0\" (UID: \"b2b65f59-2307-419b-ab4f-10a87a94e4e0\") "
Sep 09 02:33:51.429743 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:33:51.429552    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b2b65f59-2307-419b-ab4f-10a87a94e4e0-trusted-ca-bundle\") pod \"b2b65f59-2307-419b-ab4f-10a87a94e4e0\" (UID: \"b2b65f59-2307-419b-ab4f-10a87a94e4e0\") "
Sep 09 02:33:51.429743 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:33:51.429578    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b2b65f59-2307-419b-ab4f-10a87a94e4e0-console-config\") pod \"b2b65f59-2307-419b-ab4f-10a87a94e4e0\" (UID: \"b2b65f59-2307-419b-ab4f-10a87a94e4e0\") "
Sep 09 02:33:51.429743 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:33:51.429604    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b2b65f59-2307-419b-ab4f-10a87a94e4e0-service-ca\") pod \"b2b65f59-2307-419b-ab4f-10a87a94e4e0\" (UID: \"b2b65f59-2307-419b-ab4f-10a87a94e4e0\") "
Sep 09 02:33:51.429743 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:33:51.429652    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b2b65f59-2307-419b-ab4f-10a87a94e4e0-oauth-serving-cert\") pod \"b2b65f59-2307-419b-ab4f-10a87a94e4e0\" (UID: \"b2b65f59-2307-419b-ab4f-10a87a94e4e0\") "
Sep 09 02:33:51.429743 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:33:51.429685    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zbhbg\" (UniqueName: \"kubernetes.io/projected/b2b65f59-2307-419b-ab4f-10a87a94e4e0-kube-api-access-zbhbg\") pod \"b2b65f59-2307-419b-ab4f-10a87a94e4e0\" (UID: \"b2b65f59-2307-419b-ab4f-10a87a94e4e0\") "
Sep 09 02:33:51.430145 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:33:51.430100    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2b65f59-2307-419b-ab4f-10a87a94e4e0-console-config" (OuterVolumeSpecName: "console-config") pod "b2b65f59-2307-419b-ab4f-10a87a94e4e0" (UID: "b2b65f59-2307-419b-ab4f-10a87a94e4e0"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 02:33:51.430145 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:33:51.430113    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2b65f59-2307-419b-ab4f-10a87a94e4e0-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "b2b65f59-2307-419b-ab4f-10a87a94e4e0" (UID: "b2b65f59-2307-419b-ab4f-10a87a94e4e0"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 02:33:51.430318 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:33:51.430155    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2b65f59-2307-419b-ab4f-10a87a94e4e0-service-ca" (OuterVolumeSpecName: "service-ca") pod "b2b65f59-2307-419b-ab4f-10a87a94e4e0" (UID: "b2b65f59-2307-419b-ab4f-10a87a94e4e0"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 02:33:51.430318 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:33:51.430239    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2b65f59-2307-419b-ab4f-10a87a94e4e0-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b2b65f59-2307-419b-ab4f-10a87a94e4e0" (UID: "b2b65f59-2307-419b-ab4f-10a87a94e4e0"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 02:33:51.431755 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:33:51.431726    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2b65f59-2307-419b-ab4f-10a87a94e4e0-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b2b65f59-2307-419b-ab4f-10a87a94e4e0" (UID: "b2b65f59-2307-419b-ab4f-10a87a94e4e0"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 02:33:51.431755 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:33:51.431745    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2b65f59-2307-419b-ab4f-10a87a94e4e0-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b2b65f59-2307-419b-ab4f-10a87a94e4e0" (UID: "b2b65f59-2307-419b-ab4f-10a87a94e4e0"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 02:33:51.431891 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:33:51.431871    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2b65f59-2307-419b-ab4f-10a87a94e4e0-kube-api-access-zbhbg" (OuterVolumeSpecName: "kube-api-access-zbhbg") pod "b2b65f59-2307-419b-ab4f-10a87a94e4e0" (UID: "b2b65f59-2307-419b-ab4f-10a87a94e4e0"). InnerVolumeSpecName "kube-api-access-zbhbg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 02:33:51.530704 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:33:51.530681    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbhbg\" (UniqueName: \"kubernetes.io/projected/b2b65f59-2307-419b-ab4f-10a87a94e4e0-kube-api-access-zbhbg\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:33:51.530704 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:33:51.530703    2570 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b2b65f59-2307-419b-ab4f-10a87a94e4e0-console-oauth-config\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:33:51.530831 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:33:51.530712    2570 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b2b65f59-2307-419b-ab4f-10a87a94e4e0-console-serving-cert\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:33:51.530831 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:33:51.530722    2570 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b2b65f59-2307-419b-ab4f-10a87a94e4e0-trusted-ca-bundle\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:33:51.530831 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:33:51.530731    2570 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b2b65f59-2307-419b-ab4f-10a87a94e4e0-console-config\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:33:51.530831 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:33:51.530739    2570 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b2b65f59-2307-419b-ab4f-10a87a94e4e0-service-ca\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:33:51.530831 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:33:51.530748    2570 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b2b65f59-2307-419b-ab4f-10a87a94e4e0-oauth-serving-cert\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:33:52.104719 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:33:52.104684    2570 generic.go:356] "Generic (PLEG): container finished" podID="b2b65f59-2307-419b-ab4f-10a87a94e4e0" containerID="e5a2929ad6458fe44a86e25954c94e3394639d042aa5c68afed4fe00b304ffd7" exitCode=0
Sep 09 02:33:52.105132 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:33:52.104730    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-588b6849c-2vqlt"
Sep 09 02:33:52.105132 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:33:52.104760    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-588b6849c-2vqlt" event={"ID":"b2b65f59-2307-419b-ab4f-10a87a94e4e0","Type":"ContainerDied","Data":"e5a2929ad6458fe44a86e25954c94e3394639d042aa5c68afed4fe00b304ffd7"}
Sep 09 02:33:52.105132 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:33:52.104798    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-588b6849c-2vqlt" event={"ID":"b2b65f59-2307-419b-ab4f-10a87a94e4e0","Type":"ContainerDied","Data":"5da4181e961094d70e0899f0b9f93c44da7eff5454c09539f3cad0cc949f4e17"}
Sep 09 02:33:52.105132 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:33:52.104816    2570 scope.go:117] "RemoveContainer" containerID="e5a2929ad6458fe44a86e25954c94e3394639d042aa5c68afed4fe00b304ffd7"
Sep 09 02:33:52.113570 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:33:52.113553    2570 scope.go:117] "RemoveContainer" containerID="e5a2929ad6458fe44a86e25954c94e3394639d042aa5c68afed4fe00b304ffd7"
Sep 09 02:33:52.113818 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:33:52.113800    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5a2929ad6458fe44a86e25954c94e3394639d042aa5c68afed4fe00b304ffd7\": container with ID starting with e5a2929ad6458fe44a86e25954c94e3394639d042aa5c68afed4fe00b304ffd7 not found: ID does not exist" containerID="e5a2929ad6458fe44a86e25954c94e3394639d042aa5c68afed4fe00b304ffd7"
Sep 09 02:33:52.113886 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:33:52.113825    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5a2929ad6458fe44a86e25954c94e3394639d042aa5c68afed4fe00b304ffd7"} err="failed to get container status \"e5a2929ad6458fe44a86e25954c94e3394639d042aa5c68afed4fe00b304ffd7\": rpc error: code = NotFound desc = could not find container \"e5a2929ad6458fe44a86e25954c94e3394639d042aa5c68afed4fe00b304ffd7\": container with ID starting with e5a2929ad6458fe44a86e25954c94e3394639d042aa5c68afed4fe00b304ffd7 not found: ID does not exist"
Sep 09 02:33:52.126909 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:33:52.126890    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-588b6849c-2vqlt"]
Sep 09 02:33:52.131021 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:33:52.131002    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-588b6849c-2vqlt"]
Sep 09 02:33:52.222272 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:33:52.222245    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2b65f59-2307-419b-ab4f-10a87a94e4e0" path="/var/lib/kubelet/pods/b2b65f59-2307-419b-ab4f-10a87a94e4e0/volumes"
Sep 09 02:35:03.304562 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:35:03.304526    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-756c87c457-jhn45"]
Sep 09 02:35:03.305188 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:35:03.304782    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2b65f59-2307-419b-ab4f-10a87a94e4e0" containerName="console"
Sep 09 02:35:03.305188 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:35:03.304793    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2b65f59-2307-419b-ab4f-10a87a94e4e0" containerName="console"
Sep 09 02:35:03.305188 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:35:03.304847    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2b65f59-2307-419b-ab4f-10a87a94e4e0" containerName="console"
Sep 09 02:35:03.307631 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:35:03.307616    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-756c87c457-jhn45"
Sep 09 02:35:03.320480 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:35:03.320456    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-756c87c457-jhn45"]
Sep 09 02:35:03.331785 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:35:03.331761    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/87da7a65-1f6c-47ae-8325-aa9aa040fda3-oauth-serving-cert\") pod \"console-756c87c457-jhn45\" (UID: \"87da7a65-1f6c-47ae-8325-aa9aa040fda3\") " pod="openshift-console/console-756c87c457-jhn45"
Sep 09 02:35:03.331892 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:35:03.331790    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/87da7a65-1f6c-47ae-8325-aa9aa040fda3-console-config\") pod \"console-756c87c457-jhn45\" (UID: \"87da7a65-1f6c-47ae-8325-aa9aa040fda3\") " pod="openshift-console/console-756c87c457-jhn45"
Sep 09 02:35:03.331892 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:35:03.331816    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/87da7a65-1f6c-47ae-8325-aa9aa040fda3-console-serving-cert\") pod \"console-756c87c457-jhn45\" (UID: \"87da7a65-1f6c-47ae-8325-aa9aa040fda3\") " pod="openshift-console/console-756c87c457-jhn45"
Sep 09 02:35:03.331964 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:35:03.331892    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/87da7a65-1f6c-47ae-8325-aa9aa040fda3-trusted-ca-bundle\") pod \"console-756c87c457-jhn45\" (UID: \"87da7a65-1f6c-47ae-8325-aa9aa040fda3\") " pod="openshift-console/console-756c87c457-jhn45"
Sep 09 02:35:03.331964 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:35:03.331914    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/87da7a65-1f6c-47ae-8325-aa9aa040fda3-console-oauth-config\") pod \"console-756c87c457-jhn45\" (UID: \"87da7a65-1f6c-47ae-8325-aa9aa040fda3\") " pod="openshift-console/console-756c87c457-jhn45"
Sep 09 02:35:03.331964 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:35:03.331933    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/87da7a65-1f6c-47ae-8325-aa9aa040fda3-service-ca\") pod \"console-756c87c457-jhn45\" (UID: \"87da7a65-1f6c-47ae-8325-aa9aa040fda3\") " pod="openshift-console/console-756c87c457-jhn45"
Sep 09 02:35:03.332081 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:35:03.332010    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgbfj\" (UniqueName: \"kubernetes.io/projected/87da7a65-1f6c-47ae-8325-aa9aa040fda3-kube-api-access-pgbfj\") pod \"console-756c87c457-jhn45\" (UID: \"87da7a65-1f6c-47ae-8325-aa9aa040fda3\") " pod="openshift-console/console-756c87c457-jhn45"
Sep 09 02:35:03.432733 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:35:03.432695    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/87da7a65-1f6c-47ae-8325-aa9aa040fda3-oauth-serving-cert\") pod \"console-756c87c457-jhn45\" (UID: \"87da7a65-1f6c-47ae-8325-aa9aa040fda3\") " pod="openshift-console/console-756c87c457-jhn45"
Sep 09 02:35:03.432733 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:35:03.432732    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/87da7a65-1f6c-47ae-8325-aa9aa040fda3-console-config\") pod \"console-756c87c457-jhn45\" (UID: \"87da7a65-1f6c-47ae-8325-aa9aa040fda3\") " pod="openshift-console/console-756c87c457-jhn45"
Sep 09 02:35:03.433002 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:35:03.432755    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/87da7a65-1f6c-47ae-8325-aa9aa040fda3-console-serving-cert\") pod \"console-756c87c457-jhn45\" (UID: \"87da7a65-1f6c-47ae-8325-aa9aa040fda3\") " pod="openshift-console/console-756c87c457-jhn45"
Sep 09 02:35:03.433002 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:35:03.432776    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/87da7a65-1f6c-47ae-8325-aa9aa040fda3-trusted-ca-bundle\") pod \"console-756c87c457-jhn45\" (UID: \"87da7a65-1f6c-47ae-8325-aa9aa040fda3\") " pod="openshift-console/console-756c87c457-jhn45"
Sep 09 02:35:03.433002 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:35:03.432795    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/87da7a65-1f6c-47ae-8325-aa9aa040fda3-console-oauth-config\") pod \"console-756c87c457-jhn45\" (UID: \"87da7a65-1f6c-47ae-8325-aa9aa040fda3\") " pod="openshift-console/console-756c87c457-jhn45"
Sep 09 02:35:03.433002 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:35:03.432809    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/87da7a65-1f6c-47ae-8325-aa9aa040fda3-service-ca\") pod \"console-756c87c457-jhn45\" (UID: \"87da7a65-1f6c-47ae-8325-aa9aa040fda3\") " pod="openshift-console/console-756c87c457-jhn45"
Sep 09 02:35:03.433204 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:35:03.433017    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgbfj\" (UniqueName: \"kubernetes.io/projected/87da7a65-1f6c-47ae-8325-aa9aa040fda3-kube-api-access-pgbfj\") pod \"console-756c87c457-jhn45\" (UID: \"87da7a65-1f6c-47ae-8325-aa9aa040fda3\") " pod="openshift-console/console-756c87c457-jhn45"
Sep 09 02:35:03.433604 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:35:03.433579    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/87da7a65-1f6c-47ae-8325-aa9aa040fda3-oauth-serving-cert\") pod \"console-756c87c457-jhn45\" (UID: \"87da7a65-1f6c-47ae-8325-aa9aa040fda3\") " pod="openshift-console/console-756c87c457-jhn45"
Sep 09 02:35:03.433722 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:35:03.433688    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/87da7a65-1f6c-47ae-8325-aa9aa040fda3-service-ca\") pod \"console-756c87c457-jhn45\" (UID: \"87da7a65-1f6c-47ae-8325-aa9aa040fda3\") " pod="openshift-console/console-756c87c457-jhn45"
Sep 09 02:35:03.433722 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:35:03.433688    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/87da7a65-1f6c-47ae-8325-aa9aa040fda3-console-config\") pod \"console-756c87c457-jhn45\" (UID: \"87da7a65-1f6c-47ae-8325-aa9aa040fda3\") " pod="openshift-console/console-756c87c457-jhn45"
Sep 09 02:35:03.433842 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:35:03.433708    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/87da7a65-1f6c-47ae-8325-aa9aa040fda3-trusted-ca-bundle\") pod \"console-756c87c457-jhn45\" (UID: \"87da7a65-1f6c-47ae-8325-aa9aa040fda3\") " pod="openshift-console/console-756c87c457-jhn45"
Sep 09 02:35:03.435275 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:35:03.435250    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/87da7a65-1f6c-47ae-8325-aa9aa040fda3-console-oauth-config\") pod \"console-756c87c457-jhn45\" (UID: \"87da7a65-1f6c-47ae-8325-aa9aa040fda3\") " pod="openshift-console/console-756c87c457-jhn45"
Sep 09 02:35:03.435439 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:35:03.435419    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/87da7a65-1f6c-47ae-8325-aa9aa040fda3-console-serving-cert\") pod \"console-756c87c457-jhn45\" (UID: \"87da7a65-1f6c-47ae-8325-aa9aa040fda3\") " pod="openshift-console/console-756c87c457-jhn45"
Sep 09 02:35:03.444033 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:35:03.443948    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgbfj\" (UniqueName: \"kubernetes.io/projected/87da7a65-1f6c-47ae-8325-aa9aa040fda3-kube-api-access-pgbfj\") pod \"console-756c87c457-jhn45\" (UID: \"87da7a65-1f6c-47ae-8325-aa9aa040fda3\") " pod="openshift-console/console-756c87c457-jhn45"
Sep 09 02:35:03.615841 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:35:03.615814    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-756c87c457-jhn45"
Sep 09 02:35:03.893577 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:35:03.893500    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-756c87c457-jhn45"]
Sep 09 02:35:03.898402 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:35:03.898376    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87da7a65_1f6c_47ae_8325_aa9aa040fda3.slice/crio-0828511a29591fda242d33583bb9618656cc36d1d2b27c3e1ebb6046bb34ddbd WatchSource:0}: Error finding container 0828511a29591fda242d33583bb9618656cc36d1d2b27c3e1ebb6046bb34ddbd: Status 404 returned error can't find the container with id 0828511a29591fda242d33583bb9618656cc36d1d2b27c3e1ebb6046bb34ddbd
Sep 09 02:35:04.355594 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:35:04.355559    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-756c87c457-jhn45" event={"ID":"87da7a65-1f6c-47ae-8325-aa9aa040fda3","Type":"ContainerStarted","Data":"93fd96709994e9447baf0660f5e000f3daaea90f9d9fbe3b8411ea13d730fb52"}
Sep 09 02:35:04.356007 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:35:04.355613    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-756c87c457-jhn45" event={"ID":"87da7a65-1f6c-47ae-8325-aa9aa040fda3","Type":"ContainerStarted","Data":"0828511a29591fda242d33583bb9618656cc36d1d2b27c3e1ebb6046bb34ddbd"}
Sep 09 02:35:04.378033 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:35:04.377962    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-756c87c457-jhn45" podStartSLOduration=1.377945157 podStartE2EDuration="1.377945157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:35:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:35:04.376803817 +0000 UTC m=+286.631107445" watchObservedRunningTime="2026-09-09 02:35:04.377945157 +0000 UTC m=+286.632248793"
Sep 09 02:35:13.616640 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:35:13.616591    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-756c87c457-jhn45"
Sep 09 02:35:13.617112 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:35:13.616660    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-756c87c457-jhn45"
Sep 09 02:35:13.621596 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:35:13.621574    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-756c87c457-jhn45"
Sep 09 02:35:14.394002 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:35:14.393958    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-756c87c457-jhn45"
Sep 09 02:35:14.453392 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:35:14.453359    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-55975b658b-kd4nx"]
Sep 09 02:35:18.175886 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:35:18.175859    2570 kubelet.go:1648] "Image garbage collection succeeded"
Sep 09 02:35:39.476804 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:35:39.476758    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-55975b658b-kd4nx" podUID="b23d83d0-d57e-4ab4-821e-6ce90d556121" containerName="console" containerID="cri-o://35e54519ae921ba75bd428b72e30435e1cb0064dc426ecc1a1e16fd10e787ca2" gracePeriod=15
Sep 09 02:35:39.711377 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:35:39.711353    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55975b658b-kd4nx"
Sep 09 02:35:39.803924 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:35:39.803835    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b23d83d0-d57e-4ab4-821e-6ce90d556121-trusted-ca-bundle\") pod \"b23d83d0-d57e-4ab4-821e-6ce90d556121\" (UID: \"b23d83d0-d57e-4ab4-821e-6ce90d556121\") "
Sep 09 02:35:39.803924 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:35:39.803879    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b23d83d0-d57e-4ab4-821e-6ce90d556121-console-oauth-config\") pod \"b23d83d0-d57e-4ab4-821e-6ce90d556121\" (UID: \"b23d83d0-d57e-4ab4-821e-6ce90d556121\") "
Sep 09 02:35:39.803924 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:35:39.803914    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b23d83d0-d57e-4ab4-821e-6ce90d556121-service-ca\") pod \"b23d83d0-d57e-4ab4-821e-6ce90d556121\" (UID: \"b23d83d0-d57e-4ab4-821e-6ce90d556121\") "
Sep 09 02:35:39.804201 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:35:39.804035    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vb668\" (UniqueName: \"kubernetes.io/projected/b23d83d0-d57e-4ab4-821e-6ce90d556121-kube-api-access-vb668\") pod \"b23d83d0-d57e-4ab4-821e-6ce90d556121\" (UID: \"b23d83d0-d57e-4ab4-821e-6ce90d556121\") "
Sep 09 02:35:39.804201 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:35:39.804069    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b23d83d0-d57e-4ab4-821e-6ce90d556121-oauth-serving-cert\") pod \"b23d83d0-d57e-4ab4-821e-6ce90d556121\" (UID: \"b23d83d0-d57e-4ab4-821e-6ce90d556121\") "
Sep 09 02:35:39.804201 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:35:39.804106    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b23d83d0-d57e-4ab4-821e-6ce90d556121-console-config\") pod \"b23d83d0-d57e-4ab4-821e-6ce90d556121\" (UID: \"b23d83d0-d57e-4ab4-821e-6ce90d556121\") "
Sep 09 02:35:39.804201 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:35:39.804143    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b23d83d0-d57e-4ab4-821e-6ce90d556121-console-serving-cert\") pod \"b23d83d0-d57e-4ab4-821e-6ce90d556121\" (UID: \"b23d83d0-d57e-4ab4-821e-6ce90d556121\") "
Sep 09 02:35:39.804399 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:35:39.804252    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b23d83d0-d57e-4ab4-821e-6ce90d556121-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "b23d83d0-d57e-4ab4-821e-6ce90d556121" (UID: "b23d83d0-d57e-4ab4-821e-6ce90d556121"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 02:35:39.804399 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:35:39.804297    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b23d83d0-d57e-4ab4-821e-6ce90d556121-service-ca" (OuterVolumeSpecName: "service-ca") pod "b23d83d0-d57e-4ab4-821e-6ce90d556121" (UID: "b23d83d0-d57e-4ab4-821e-6ce90d556121"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 02:35:39.804516 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:35:39.804497    2570 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b23d83d0-d57e-4ab4-821e-6ce90d556121-trusted-ca-bundle\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:35:39.804568 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:35:39.804507    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b23d83d0-d57e-4ab4-821e-6ce90d556121-console-config" (OuterVolumeSpecName: "console-config") pod "b23d83d0-d57e-4ab4-821e-6ce90d556121" (UID: "b23d83d0-d57e-4ab4-821e-6ce90d556121"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 02:35:39.804568 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:35:39.804522    2570 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b23d83d0-d57e-4ab4-821e-6ce90d556121-service-ca\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:35:39.804568 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:35:39.804512    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b23d83d0-d57e-4ab4-821e-6ce90d556121-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b23d83d0-d57e-4ab4-821e-6ce90d556121" (UID: "b23d83d0-d57e-4ab4-821e-6ce90d556121"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 02:35:39.806122 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:35:39.806098    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b23d83d0-d57e-4ab4-821e-6ce90d556121-kube-api-access-vb668" (OuterVolumeSpecName: "kube-api-access-vb668") pod "b23d83d0-d57e-4ab4-821e-6ce90d556121" (UID: "b23d83d0-d57e-4ab4-821e-6ce90d556121"). InnerVolumeSpecName "kube-api-access-vb668". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 02:35:39.806516 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:35:39.806491    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b23d83d0-d57e-4ab4-821e-6ce90d556121-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b23d83d0-d57e-4ab4-821e-6ce90d556121" (UID: "b23d83d0-d57e-4ab4-821e-6ce90d556121"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 02:35:39.806590 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:35:39.806506    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b23d83d0-d57e-4ab4-821e-6ce90d556121-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b23d83d0-d57e-4ab4-821e-6ce90d556121" (UID: "b23d83d0-d57e-4ab4-821e-6ce90d556121"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 02:35:39.905800 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:35:39.905770    2570 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b23d83d0-d57e-4ab4-821e-6ce90d556121-console-oauth-config\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:35:39.905800 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:35:39.905794    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vb668\" (UniqueName: \"kubernetes.io/projected/b23d83d0-d57e-4ab4-821e-6ce90d556121-kube-api-access-vb668\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:35:39.905959 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:35:39.905807    2570 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b23d83d0-d57e-4ab4-821e-6ce90d556121-oauth-serving-cert\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:35:39.905959 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:35:39.905819    2570 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b23d83d0-d57e-4ab4-821e-6ce90d556121-console-config\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:35:39.905959 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:35:39.905833    2570 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b23d83d0-d57e-4ab4-821e-6ce90d556121-console-serving-cert\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:35:40.485919 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:35:40.485825    2570 generic.go:356] "Generic (PLEG): container finished" podID="b23d83d0-d57e-4ab4-821e-6ce90d556121" containerID="35e54519ae921ba75bd428b72e30435e1cb0064dc426ecc1a1e16fd10e787ca2" exitCode=0
Sep 09 02:35:40.485919 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:35:40.485884    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55975b658b-kd4nx"
Sep 09 02:35:40.485919 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:35:40.485904    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55975b658b-kd4nx" event={"ID":"b23d83d0-d57e-4ab4-821e-6ce90d556121","Type":"ContainerDied","Data":"35e54519ae921ba75bd428b72e30435e1cb0064dc426ecc1a1e16fd10e787ca2"}
Sep 09 02:35:40.486481 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:35:40.486004    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55975b658b-kd4nx" event={"ID":"b23d83d0-d57e-4ab4-821e-6ce90d556121","Type":"ContainerDied","Data":"2cd7df21616f474abc61d433adf4aef6586f6c34a8942c4df9f6f1b02caa6952"}
Sep 09 02:35:40.486481 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:35:40.486038    2570 scope.go:117] "RemoveContainer" containerID="35e54519ae921ba75bd428b72e30435e1cb0064dc426ecc1a1e16fd10e787ca2"
Sep 09 02:35:40.493914 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:35:40.493891    2570 scope.go:117] "RemoveContainer" containerID="35e54519ae921ba75bd428b72e30435e1cb0064dc426ecc1a1e16fd10e787ca2"
Sep 09 02:35:40.494205 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:35:40.494185    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35e54519ae921ba75bd428b72e30435e1cb0064dc426ecc1a1e16fd10e787ca2\": container with ID starting with 35e54519ae921ba75bd428b72e30435e1cb0064dc426ecc1a1e16fd10e787ca2 not found: ID does not exist" containerID="35e54519ae921ba75bd428b72e30435e1cb0064dc426ecc1a1e16fd10e787ca2"
Sep 09 02:35:40.494289 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:35:40.494217    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35e54519ae921ba75bd428b72e30435e1cb0064dc426ecc1a1e16fd10e787ca2"} err="failed to get container status \"35e54519ae921ba75bd428b72e30435e1cb0064dc426ecc1a1e16fd10e787ca2\": rpc error: code = NotFound desc = could not find container \"35e54519ae921ba75bd428b72e30435e1cb0064dc426ecc1a1e16fd10e787ca2\": container with ID starting with 35e54519ae921ba75bd428b72e30435e1cb0064dc426ecc1a1e16fd10e787ca2 not found: ID does not exist"
Sep 09 02:35:40.505267 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:35:40.505243    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-55975b658b-kd4nx"]
Sep 09 02:35:40.507124 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:35:40.507106    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-55975b658b-kd4nx"]
Sep 09 02:35:42.221313 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:35:42.221284    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b23d83d0-d57e-4ab4-821e-6ce90d556121" path="/var/lib/kubelet/pods/b23d83d0-d57e-4ab4-821e-6ce90d556121/volumes"
Sep 09 02:36:32.353363 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:36:32.353332    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-7b84x"]
Sep 09 02:36:32.353774 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:36:32.353570    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b23d83d0-d57e-4ab4-821e-6ce90d556121" containerName="console"
Sep 09 02:36:32.353774 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:36:32.353581    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="b23d83d0-d57e-4ab4-821e-6ce90d556121" containerName="console"
Sep 09 02:36:32.353774 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:36:32.353637    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="b23d83d0-d57e-4ab4-821e-6ce90d556121" containerName="console"
Sep 09 02:36:32.356770 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:36:32.356754    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-7b84x"
Sep 09 02:36:32.360909 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:36:32.360887    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-vvl6b\""
Sep 09 02:36:32.362053 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:36:32.362028    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 09 02:36:32.362159 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:36:32.362090    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 09 02:36:32.372166 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:36:32.372144    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-7b84x"]
Sep 09 02:36:32.389050 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:36:32.389025    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a3e3c25e-a676-448a-8bb8-dd2b6571b3a4-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-7b84x\" (UID: \"a3e3c25e-a676-448a-8bb8-dd2b6571b3a4\") " pod="cert-manager/cert-manager-webhook-6486ffd997-7b84x"
Sep 09 02:36:32.389136 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:36:32.389078    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wf7vm\" (UniqueName: \"kubernetes.io/projected/a3e3c25e-a676-448a-8bb8-dd2b6571b3a4-kube-api-access-wf7vm\") pod \"cert-manager-webhook-6486ffd997-7b84x\" (UID: \"a3e3c25e-a676-448a-8bb8-dd2b6571b3a4\") " pod="cert-manager/cert-manager-webhook-6486ffd997-7b84x"
Sep 09 02:36:32.490184 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:36:32.490163    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a3e3c25e-a676-448a-8bb8-dd2b6571b3a4-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-7b84x\" (UID: \"a3e3c25e-a676-448a-8bb8-dd2b6571b3a4\") " pod="cert-manager/cert-manager-webhook-6486ffd997-7b84x"
Sep 09 02:36:32.490280 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:36:32.490196    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wf7vm\" (UniqueName: \"kubernetes.io/projected/a3e3c25e-a676-448a-8bb8-dd2b6571b3a4-kube-api-access-wf7vm\") pod \"cert-manager-webhook-6486ffd997-7b84x\" (UID: \"a3e3c25e-a676-448a-8bb8-dd2b6571b3a4\") " pod="cert-manager/cert-manager-webhook-6486ffd997-7b84x"
Sep 09 02:36:32.499408 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:36:32.499387    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a3e3c25e-a676-448a-8bb8-dd2b6571b3a4-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-7b84x\" (UID: \"a3e3c25e-a676-448a-8bb8-dd2b6571b3a4\") " pod="cert-manager/cert-manager-webhook-6486ffd997-7b84x"
Sep 09 02:36:32.499516 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:36:32.499470    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wf7vm\" (UniqueName: \"kubernetes.io/projected/a3e3c25e-a676-448a-8bb8-dd2b6571b3a4-kube-api-access-wf7vm\") pod \"cert-manager-webhook-6486ffd997-7b84x\" (UID: \"a3e3c25e-a676-448a-8bb8-dd2b6571b3a4\") " pod="cert-manager/cert-manager-webhook-6486ffd997-7b84x"
Sep 09 02:36:32.682180 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:36:32.682107    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-7b84x"
Sep 09 02:36:32.961366 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:36:32.961283    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-7b84x"]
Sep 09 02:36:32.964737 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:36:32.964711    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda3e3c25e_a676_448a_8bb8_dd2b6571b3a4.slice/crio-b990cc83b87d0cac50fd0dfcc8a5dd3536bf46915a1b35981c1983aa11c20407 WatchSource:0}: Error finding container b990cc83b87d0cac50fd0dfcc8a5dd3536bf46915a1b35981c1983aa11c20407: Status 404 returned error can't find the container with id b990cc83b87d0cac50fd0dfcc8a5dd3536bf46915a1b35981c1983aa11c20407
Sep 09 02:36:32.966716 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:36:32.966698    2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 02:36:33.676222 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:36:33.676183    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-7b84x" event={"ID":"a3e3c25e-a676-448a-8bb8-dd2b6571b3a4","Type":"ContainerStarted","Data":"b990cc83b87d0cac50fd0dfcc8a5dd3536bf46915a1b35981c1983aa11c20407"}
Sep 09 02:36:36.689872 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:36:36.689836    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-7b84x" event={"ID":"a3e3c25e-a676-448a-8bb8-dd2b6571b3a4","Type":"ContainerStarted","Data":"ed777ffed5ac4242815661685cc251aa37184b21edc0cafdfaa0d82d1f7cdd57"}
Sep 09 02:36:36.690333 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:36:36.689885    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-7b84x"
Sep 09 02:36:36.709657 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:36:36.709576    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-7b84x" podStartSLOduration=1.228937894 podStartE2EDuration="4.709560588s" totalImagesPullingTime="3.480622694s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:36:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:36:36.707727645 +0000 UTC m=+378.962031280" watchObservedRunningTime="2026-09-09 02:36:36.709560588 +0000 UTC m=+378.963864224"
Sep 09 02:36:42.696439 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:36:42.696402    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-7b84x"
Sep 09 02:37:09.319828 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:09.319794    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6c4bb7846b-xfr4t"]
Sep 09 02:37:09.323126 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:09.323111    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6c4bb7846b-xfr4t"
Sep 09 02:37:09.326202 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:09.326182    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 09 02:37:09.327694 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:09.327673    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 09 02:37:09.327814 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:09.327701    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 09 02:37:09.327814 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:09.327675    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 09 02:37:09.327925 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:09.327868    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-vxnzz\""
Sep 09 02:37:09.327995 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:09.327952    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 09 02:37:09.336496 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:09.336475    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6c4bb7846b-xfr4t"]
Sep 09 02:37:09.482046 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:09.481997    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/32d09487-e544-4730-89eb-815e2beda070-metrics-cert\") pod \"lws-controller-manager-6c4bb7846b-xfr4t\" (UID: \"32d09487-e544-4730-89eb-815e2beda070\") " pod="openshift-lws-operator/lws-controller-manager-6c4bb7846b-xfr4t"
Sep 09 02:37:09.482194 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:09.482070    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvf5r\" (UniqueName: \"kubernetes.io/projected/32d09487-e544-4730-89eb-815e2beda070-kube-api-access-kvf5r\") pod \"lws-controller-manager-6c4bb7846b-xfr4t\" (UID: \"32d09487-e544-4730-89eb-815e2beda070\") " pod="openshift-lws-operator/lws-controller-manager-6c4bb7846b-xfr4t"
Sep 09 02:37:09.482194 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:09.482092    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/32d09487-e544-4730-89eb-815e2beda070-manager-config\") pod \"lws-controller-manager-6c4bb7846b-xfr4t\" (UID: \"32d09487-e544-4730-89eb-815e2beda070\") " pod="openshift-lws-operator/lws-controller-manager-6c4bb7846b-xfr4t"
Sep 09 02:37:09.482194 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:09.482115    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/32d09487-e544-4730-89eb-815e2beda070-cert\") pod \"lws-controller-manager-6c4bb7846b-xfr4t\" (UID: \"32d09487-e544-4730-89eb-815e2beda070\") " pod="openshift-lws-operator/lws-controller-manager-6c4bb7846b-xfr4t"
Sep 09 02:37:09.583095 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:09.583026    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/32d09487-e544-4730-89eb-815e2beda070-metrics-cert\") pod \"lws-controller-manager-6c4bb7846b-xfr4t\" (UID: \"32d09487-e544-4730-89eb-815e2beda070\") " pod="openshift-lws-operator/lws-controller-manager-6c4bb7846b-xfr4t"
Sep 09 02:37:09.583095 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:09.583070    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvf5r\" (UniqueName: \"kubernetes.io/projected/32d09487-e544-4730-89eb-815e2beda070-kube-api-access-kvf5r\") pod \"lws-controller-manager-6c4bb7846b-xfr4t\" (UID: \"32d09487-e544-4730-89eb-815e2beda070\") " pod="openshift-lws-operator/lws-controller-manager-6c4bb7846b-xfr4t"
Sep 09 02:37:09.583095 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:09.583091    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/32d09487-e544-4730-89eb-815e2beda070-manager-config\") pod \"lws-controller-manager-6c4bb7846b-xfr4t\" (UID: \"32d09487-e544-4730-89eb-815e2beda070\") " pod="openshift-lws-operator/lws-controller-manager-6c4bb7846b-xfr4t"
Sep 09 02:37:09.583300 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:09.583114    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/32d09487-e544-4730-89eb-815e2beda070-cert\") pod \"lws-controller-manager-6c4bb7846b-xfr4t\" (UID: \"32d09487-e544-4730-89eb-815e2beda070\") " pod="openshift-lws-operator/lws-controller-manager-6c4bb7846b-xfr4t"
Sep 09 02:37:09.584022 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:09.583999    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/32d09487-e544-4730-89eb-815e2beda070-manager-config\") pod \"lws-controller-manager-6c4bb7846b-xfr4t\" (UID: \"32d09487-e544-4730-89eb-815e2beda070\") " pod="openshift-lws-operator/lws-controller-manager-6c4bb7846b-xfr4t"
Sep 09 02:37:09.585635 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:09.585613    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/32d09487-e544-4730-89eb-815e2beda070-metrics-cert\") pod \"lws-controller-manager-6c4bb7846b-xfr4t\" (UID: \"32d09487-e544-4730-89eb-815e2beda070\") " pod="openshift-lws-operator/lws-controller-manager-6c4bb7846b-xfr4t"
Sep 09 02:37:09.585635 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:09.585622    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/32d09487-e544-4730-89eb-815e2beda070-cert\") pod \"lws-controller-manager-6c4bb7846b-xfr4t\" (UID: \"32d09487-e544-4730-89eb-815e2beda070\") " pod="openshift-lws-operator/lws-controller-manager-6c4bb7846b-xfr4t"
Sep 09 02:37:09.595864 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:09.595842    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvf5r\" (UniqueName: \"kubernetes.io/projected/32d09487-e544-4730-89eb-815e2beda070-kube-api-access-kvf5r\") pod \"lws-controller-manager-6c4bb7846b-xfr4t\" (UID: \"32d09487-e544-4730-89eb-815e2beda070\") " pod="openshift-lws-operator/lws-controller-manager-6c4bb7846b-xfr4t"
Sep 09 02:37:09.632191 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:09.632167    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6c4bb7846b-xfr4t"
Sep 09 02:37:09.931096 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:09.931066    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6c4bb7846b-xfr4t"]
Sep 09 02:37:09.933902 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:37:09.933876    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32d09487_e544_4730_89eb_815e2beda070.slice/crio-57ceaa423307ea2fcf482e3fe923eb608cdb220c7a0a9f9116cae7475661af48 WatchSource:0}: Error finding container 57ceaa423307ea2fcf482e3fe923eb608cdb220c7a0a9f9116cae7475661af48: Status 404 returned error can't find the container with id 57ceaa423307ea2fcf482e3fe923eb608cdb220c7a0a9f9116cae7475661af48
Sep 09 02:37:10.813321 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:10.813271    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6c4bb7846b-xfr4t" event={"ID":"32d09487-e544-4730-89eb-815e2beda070","Type":"ContainerStarted","Data":"57ceaa423307ea2fcf482e3fe923eb608cdb220c7a0a9f9116cae7475661af48"}
Sep 09 02:37:10.983727 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:10.983695    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6dfc788bdd-djpwc"]
Sep 09 02:37:10.986103 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:10.986074    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6dfc788bdd-djpwc"
Sep 09 02:37:10.992242 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:10.992216    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 09 02:37:10.992386 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:10.992313    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 09 02:37:10.992455 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:10.992402    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-b6csz\""
Sep 09 02:37:10.992639 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:10.992621    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 09 02:37:10.992913 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:10.992884    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 09 02:37:11.008292 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:11.008264    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6dfc788bdd-djpwc"]
Sep 09 02:37:11.096152 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:11.096115    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e24b4e11-a947-453d-9760-d7e27016e656-webhook-cert\") pod \"opendatahub-operator-controller-manager-6dfc788bdd-djpwc\" (UID: \"e24b4e11-a947-453d-9760-d7e27016e656\") " pod="opendatahub/opendatahub-operator-controller-manager-6dfc788bdd-djpwc"
Sep 09 02:37:11.096334 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:11.096204    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkxmj\" (UniqueName: \"kubernetes.io/projected/e24b4e11-a947-453d-9760-d7e27016e656-kube-api-access-lkxmj\") pod \"opendatahub-operator-controller-manager-6dfc788bdd-djpwc\" (UID: \"e24b4e11-a947-453d-9760-d7e27016e656\") " pod="opendatahub/opendatahub-operator-controller-manager-6dfc788bdd-djpwc"
Sep 09 02:37:11.096334 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:11.096320    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e24b4e11-a947-453d-9760-d7e27016e656-tmp\") pod \"opendatahub-operator-controller-manager-6dfc788bdd-djpwc\" (UID: \"e24b4e11-a947-453d-9760-d7e27016e656\") " pod="opendatahub/opendatahub-operator-controller-manager-6dfc788bdd-djpwc"
Sep 09 02:37:11.096480 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:11.096359    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e24b4e11-a947-453d-9760-d7e27016e656-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6dfc788bdd-djpwc\" (UID: \"e24b4e11-a947-453d-9760-d7e27016e656\") " pod="opendatahub/opendatahub-operator-controller-manager-6dfc788bdd-djpwc"
Sep 09 02:37:11.096480 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:11.096399    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e24b4e11-a947-453d-9760-d7e27016e656-manifests\") pod \"opendatahub-operator-controller-manager-6dfc788bdd-djpwc\" (UID: \"e24b4e11-a947-453d-9760-d7e27016e656\") " pod="opendatahub/opendatahub-operator-controller-manager-6dfc788bdd-djpwc"
Sep 09 02:37:11.197062 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:11.197023    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkxmj\" (UniqueName: \"kubernetes.io/projected/e24b4e11-a947-453d-9760-d7e27016e656-kube-api-access-lkxmj\") pod \"opendatahub-operator-controller-manager-6dfc788bdd-djpwc\" (UID: \"e24b4e11-a947-453d-9760-d7e27016e656\") " pod="opendatahub/opendatahub-operator-controller-manager-6dfc788bdd-djpwc"
Sep 09 02:37:11.197254 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:11.197083    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e24b4e11-a947-453d-9760-d7e27016e656-tmp\") pod \"opendatahub-operator-controller-manager-6dfc788bdd-djpwc\" (UID: \"e24b4e11-a947-453d-9760-d7e27016e656\") " pod="opendatahub/opendatahub-operator-controller-manager-6dfc788bdd-djpwc"
Sep 09 02:37:11.197254 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:11.197109    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e24b4e11-a947-453d-9760-d7e27016e656-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6dfc788bdd-djpwc\" (UID: \"e24b4e11-a947-453d-9760-d7e27016e656\") " pod="opendatahub/opendatahub-operator-controller-manager-6dfc788bdd-djpwc"
Sep 09 02:37:11.197254 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:11.197135    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e24b4e11-a947-453d-9760-d7e27016e656-manifests\") pod \"opendatahub-operator-controller-manager-6dfc788bdd-djpwc\" (UID: \"e24b4e11-a947-453d-9760-d7e27016e656\") " pod="opendatahub/opendatahub-operator-controller-manager-6dfc788bdd-djpwc"
Sep 09 02:37:11.197254 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:11.197163    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e24b4e11-a947-453d-9760-d7e27016e656-webhook-cert\") pod \"opendatahub-operator-controller-manager-6dfc788bdd-djpwc\" (UID: \"e24b4e11-a947-453d-9760-d7e27016e656\") " pod="opendatahub/opendatahub-operator-controller-manager-6dfc788bdd-djpwc"
Sep 09 02:37:11.197616 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:11.197589    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e24b4e11-a947-453d-9760-d7e27016e656-manifests\") pod \"opendatahub-operator-controller-manager-6dfc788bdd-djpwc\" (UID: \"e24b4e11-a947-453d-9760-d7e27016e656\") " pod="opendatahub/opendatahub-operator-controller-manager-6dfc788bdd-djpwc"
Sep 09 02:37:11.200003 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:11.199962    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e24b4e11-a947-453d-9760-d7e27016e656-tmp\") pod \"opendatahub-operator-controller-manager-6dfc788bdd-djpwc\" (UID: \"e24b4e11-a947-453d-9760-d7e27016e656\") " pod="opendatahub/opendatahub-operator-controller-manager-6dfc788bdd-djpwc"
Sep 09 02:37:11.200003 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:11.199993    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e24b4e11-a947-453d-9760-d7e27016e656-webhook-cert\") pod \"opendatahub-operator-controller-manager-6dfc788bdd-djpwc\" (UID: \"e24b4e11-a947-453d-9760-d7e27016e656\") " pod="opendatahub/opendatahub-operator-controller-manager-6dfc788bdd-djpwc"
Sep 09 02:37:11.200163 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:11.200077    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e24b4e11-a947-453d-9760-d7e27016e656-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6dfc788bdd-djpwc\" (UID: \"e24b4e11-a947-453d-9760-d7e27016e656\") " pod="opendatahub/opendatahub-operator-controller-manager-6dfc788bdd-djpwc"
Sep 09 02:37:11.222604 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:11.222577    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkxmj\" (UniqueName: \"kubernetes.io/projected/e24b4e11-a947-453d-9760-d7e27016e656-kube-api-access-lkxmj\") pod \"opendatahub-operator-controller-manager-6dfc788bdd-djpwc\" (UID: \"e24b4e11-a947-453d-9760-d7e27016e656\") " pod="opendatahub/opendatahub-operator-controller-manager-6dfc788bdd-djpwc"
Sep 09 02:37:11.299445 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:11.299414    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6dfc788bdd-djpwc"
Sep 09 02:37:11.596151 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:11.596094    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6dfc788bdd-djpwc"]
Sep 09 02:37:11.599256 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:37:11.599224    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode24b4e11_a947_453d_9760_d7e27016e656.slice/crio-06d819fb32245d4bb99742cf219e3f1ca1e2f3c7ad8d36ce0502a976ccec357d WatchSource:0}: Error finding container 06d819fb32245d4bb99742cf219e3f1ca1e2f3c7ad8d36ce0502a976ccec357d: Status 404 returned error can't find the container with id 06d819fb32245d4bb99742cf219e3f1ca1e2f3c7ad8d36ce0502a976ccec357d
Sep 09 02:37:11.818845 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:11.818760    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6dfc788bdd-djpwc" event={"ID":"e24b4e11-a947-453d-9760-d7e27016e656","Type":"ContainerStarted","Data":"06d819fb32245d4bb99742cf219e3f1ca1e2f3c7ad8d36ce0502a976ccec357d"}
Sep 09 02:37:15.818422 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:15.818388    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-54fd64c946-w5jf9"]
Sep 09 02:37:15.824938 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:15.824915    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54fd64c946-w5jf9"
Sep 09 02:37:15.838472 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:15.838449    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-54fd64c946-w5jf9"]
Sep 09 02:37:15.839608 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:15.839583    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6c4bb7846b-xfr4t" event={"ID":"32d09487-e544-4730-89eb-815e2beda070","Type":"ContainerStarted","Data":"bf0fad99d4b7b69d70bbb5604a3aa431368dd2a3ece579349f4d047ed01f6cd4"}
Sep 09 02:37:15.839734 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:15.839636    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6c4bb7846b-xfr4t"
Sep 09 02:37:15.841450 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:15.841427    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6dfc788bdd-djpwc" event={"ID":"e24b4e11-a947-453d-9760-d7e27016e656","Type":"ContainerStarted","Data":"6274a6afcae0685505dd0760747aa8ab530d3975fc2f9240b6b31157042be34e"}
Sep 09 02:37:15.913538 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:15.913491    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6c4bb7846b-xfr4t" podStartSLOduration=1.7786885350000001 podStartE2EDuration="6.91347576s" totalImagesPullingTime="5.134787225s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:37:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:37:15.911721818 +0000 UTC m=+418.166025455" watchObservedRunningTime="2026-09-09 02:37:15.91347576 +0000 UTC m=+418.167779395"
Sep 09 02:37:15.943062 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:15.943037    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bc4a6464-355e-4063-9cd8-359fe8db6cfd-console-oauth-config\") pod \"console-54fd64c946-w5jf9\" (UID: \"bc4a6464-355e-4063-9cd8-359fe8db6cfd\") " pod="openshift-console/console-54fd64c946-w5jf9"
Sep 09 02:37:15.943189 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:15.943139    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krl4v\" (UniqueName: \"kubernetes.io/projected/bc4a6464-355e-4063-9cd8-359fe8db6cfd-kube-api-access-krl4v\") pod \"console-54fd64c946-w5jf9\" (UID: \"bc4a6464-355e-4063-9cd8-359fe8db6cfd\") " pod="openshift-console/console-54fd64c946-w5jf9"
Sep 09 02:37:15.943254 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:15.943223    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bc4a6464-355e-4063-9cd8-359fe8db6cfd-oauth-serving-cert\") pod \"console-54fd64c946-w5jf9\" (UID: \"bc4a6464-355e-4063-9cd8-359fe8db6cfd\") " pod="openshift-console/console-54fd64c946-w5jf9"
Sep 09 02:37:15.943313 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:15.943272    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bc4a6464-355e-4063-9cd8-359fe8db6cfd-console-config\") pod \"console-54fd64c946-w5jf9\" (UID: \"bc4a6464-355e-4063-9cd8-359fe8db6cfd\") " pod="openshift-console/console-54fd64c946-w5jf9"
Sep 09 02:37:15.943358 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:15.943309    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc4a6464-355e-4063-9cd8-359fe8db6cfd-trusted-ca-bundle\") pod \"console-54fd64c946-w5jf9\" (UID: \"bc4a6464-355e-4063-9cd8-359fe8db6cfd\") " pod="openshift-console/console-54fd64c946-w5jf9"
Sep 09 02:37:15.943397 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:15.943379    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bc4a6464-355e-4063-9cd8-359fe8db6cfd-console-serving-cert\") pod \"console-54fd64c946-w5jf9\" (UID: \"bc4a6464-355e-4063-9cd8-359fe8db6cfd\") " pod="openshift-console/console-54fd64c946-w5jf9"
Sep 09 02:37:15.943440 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:15.943406    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bc4a6464-355e-4063-9cd8-359fe8db6cfd-service-ca\") pod \"console-54fd64c946-w5jf9\" (UID: \"bc4a6464-355e-4063-9cd8-359fe8db6cfd\") " pod="openshift-console/console-54fd64c946-w5jf9"
Sep 09 02:37:16.044614 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:16.044582    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bc4a6464-355e-4063-9cd8-359fe8db6cfd-console-config\") pod \"console-54fd64c946-w5jf9\" (UID: \"bc4a6464-355e-4063-9cd8-359fe8db6cfd\") " pod="openshift-console/console-54fd64c946-w5jf9"
Sep 09 02:37:16.044749 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:16.044624    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc4a6464-355e-4063-9cd8-359fe8db6cfd-trusted-ca-bundle\") pod \"console-54fd64c946-w5jf9\" (UID: \"bc4a6464-355e-4063-9cd8-359fe8db6cfd\") " pod="openshift-console/console-54fd64c946-w5jf9"
Sep 09 02:37:16.044749 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:16.044649    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bc4a6464-355e-4063-9cd8-359fe8db6cfd-console-serving-cert\") pod \"console-54fd64c946-w5jf9\" (UID: \"bc4a6464-355e-4063-9cd8-359fe8db6cfd\") " pod="openshift-console/console-54fd64c946-w5jf9"
Sep 09 02:37:16.044749 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:16.044662    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bc4a6464-355e-4063-9cd8-359fe8db6cfd-service-ca\") pod \"console-54fd64c946-w5jf9\" (UID: \"bc4a6464-355e-4063-9cd8-359fe8db6cfd\") " pod="openshift-console/console-54fd64c946-w5jf9"
Sep 09 02:37:16.044749 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:16.044687    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bc4a6464-355e-4063-9cd8-359fe8db6cfd-console-oauth-config\") pod \"console-54fd64c946-w5jf9\" (UID: \"bc4a6464-355e-4063-9cd8-359fe8db6cfd\") " pod="openshift-console/console-54fd64c946-w5jf9"
Sep 09 02:37:16.044749 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:16.044712    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krl4v\" (UniqueName: \"kubernetes.io/projected/bc4a6464-355e-4063-9cd8-359fe8db6cfd-kube-api-access-krl4v\") pod \"console-54fd64c946-w5jf9\" (UID: \"bc4a6464-355e-4063-9cd8-359fe8db6cfd\") " pod="openshift-console/console-54fd64c946-w5jf9"
Sep 09 02:37:16.045039 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:16.044768    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bc4a6464-355e-4063-9cd8-359fe8db6cfd-oauth-serving-cert\") pod \"console-54fd64c946-w5jf9\" (UID: \"bc4a6464-355e-4063-9cd8-359fe8db6cfd\") " pod="openshift-console/console-54fd64c946-w5jf9"
Sep 09 02:37:16.045475 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:16.045443    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bc4a6464-355e-4063-9cd8-359fe8db6cfd-service-ca\") pod \"console-54fd64c946-w5jf9\" (UID: \"bc4a6464-355e-4063-9cd8-359fe8db6cfd\") " pod="openshift-console/console-54fd64c946-w5jf9"
Sep 09 02:37:16.045584 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:16.045449    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bc4a6464-355e-4063-9cd8-359fe8db6cfd-oauth-serving-cert\") pod \"console-54fd64c946-w5jf9\" (UID: \"bc4a6464-355e-4063-9cd8-359fe8db6cfd\") " pod="openshift-console/console-54fd64c946-w5jf9"
Sep 09 02:37:16.045584 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:16.045447    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bc4a6464-355e-4063-9cd8-359fe8db6cfd-console-config\") pod \"console-54fd64c946-w5jf9\" (UID: \"bc4a6464-355e-4063-9cd8-359fe8db6cfd\") " pod="openshift-console/console-54fd64c946-w5jf9"
Sep 09 02:37:16.045727 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:16.045701    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc4a6464-355e-4063-9cd8-359fe8db6cfd-trusted-ca-bundle\") pod \"console-54fd64c946-w5jf9\" (UID: \"bc4a6464-355e-4063-9cd8-359fe8db6cfd\") " pod="openshift-console/console-54fd64c946-w5jf9"
Sep 09 02:37:16.047302 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:16.047278    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bc4a6464-355e-4063-9cd8-359fe8db6cfd-console-oauth-config\") pod \"console-54fd64c946-w5jf9\" (UID: \"bc4a6464-355e-4063-9cd8-359fe8db6cfd\") " pod="openshift-console/console-54fd64c946-w5jf9"
Sep 09 02:37:16.047556 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:16.047531    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bc4a6464-355e-4063-9cd8-359fe8db6cfd-console-serving-cert\") pod \"console-54fd64c946-w5jf9\" (UID: \"bc4a6464-355e-4063-9cd8-359fe8db6cfd\") " pod="openshift-console/console-54fd64c946-w5jf9"
Sep 09 02:37:16.059719 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:16.059698    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-krl4v\" (UniqueName: \"kubernetes.io/projected/bc4a6464-355e-4063-9cd8-359fe8db6cfd-kube-api-access-krl4v\") pod \"console-54fd64c946-w5jf9\" (UID: \"bc4a6464-355e-4063-9cd8-359fe8db6cfd\") " pod="openshift-console/console-54fd64c946-w5jf9"
Sep 09 02:37:16.133874 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:16.133841    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54fd64c946-w5jf9"
Sep 09 02:37:16.411307 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:16.411254    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-54fd64c946-w5jf9"]
Sep 09 02:37:16.413825 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:37:16.413789    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc4a6464_355e_4063_9cd8_359fe8db6cfd.slice/crio-bfe7d8a2dd038e5bf346bc9b723f076566ed3a6a72b51ba6113240c59f2aa7fd WatchSource:0}: Error finding container bfe7d8a2dd038e5bf346bc9b723f076566ed3a6a72b51ba6113240c59f2aa7fd: Status 404 returned error can't find the container with id bfe7d8a2dd038e5bf346bc9b723f076566ed3a6a72b51ba6113240c59f2aa7fd
Sep 09 02:37:16.847784 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:16.847754    2570 generic.go:356] "Generic (PLEG): container finished" podID="e24b4e11-a947-453d-9760-d7e27016e656" containerID="6274a6afcae0685505dd0760747aa8ab530d3975fc2f9240b6b31157042be34e" exitCode=0
Sep 09 02:37:16.848189 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:16.847839    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6dfc788bdd-djpwc" event={"ID":"e24b4e11-a947-453d-9760-d7e27016e656","Type":"ContainerDied","Data":"6274a6afcae0685505dd0760747aa8ab530d3975fc2f9240b6b31157042be34e"}
Sep 09 02:37:16.849612 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:16.849592    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54fd64c946-w5jf9" event={"ID":"bc4a6464-355e-4063-9cd8-359fe8db6cfd","Type":"ContainerStarted","Data":"45c72d733bb8b2b13496a9006de6212930a83e4ff4c9a60c64109c3a619b8f7d"}
Sep 09 02:37:16.849696 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:16.849629    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54fd64c946-w5jf9" event={"ID":"bc4a6464-355e-4063-9cd8-359fe8db6cfd","Type":"ContainerStarted","Data":"bfe7d8a2dd038e5bf346bc9b723f076566ed3a6a72b51ba6113240c59f2aa7fd"}
Sep 09 02:37:16.893965 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:16.893924    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-54fd64c946-w5jf9" podStartSLOduration=1.8939121810000001 podStartE2EDuration="1.893912181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:37:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:37:16.89340953 +0000 UTC m=+419.147713167" watchObservedRunningTime="2026-09-09 02:37:16.893912181 +0000 UTC m=+419.148215817"
Sep 09 02:37:17.856322 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:17.856291    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6dfc788bdd-djpwc" event={"ID":"e24b4e11-a947-453d-9760-d7e27016e656","Type":"ContainerStarted","Data":"3436ea729953323ebeeb37826008d0784ef8059f3ae8b78ffa9ba07337fa7e56"}
Sep 09 02:37:17.856748 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:17.856391    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6dfc788bdd-djpwc"
Sep 09 02:37:26.134833 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:26.134794    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-54fd64c946-w5jf9"
Sep 09 02:37:26.135452 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:26.134893    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-54fd64c946-w5jf9"
Sep 09 02:37:26.139388 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:26.139365    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-54fd64c946-w5jf9"
Sep 09 02:37:26.207817 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:26.207761    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6dfc788bdd-djpwc" podStartSLOduration=11.211691137 podStartE2EDuration="16.207746032s" totalImagesPullingTime="3.825238918s" totalInitContainerRuntime="1.170815977s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:37:10 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:37:17.885883624 +0000 UTC m=+420.140187259" watchObservedRunningTime="2026-09-09 02:37:26.207746032 +0000 UTC m=+428.462049668"
Sep 09 02:37:26.851988 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:26.851954    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6c4bb7846b-xfr4t"
Sep 09 02:37:26.904713 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:26.904683    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-54fd64c946-w5jf9"
Sep 09 02:37:26.967246 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:26.967210    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-756c87c457-jhn45"]
Sep 09 02:37:28.862685 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:28.862654    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6dfc788bdd-djpwc"
Sep 09 02:37:48.908114 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:48.908031    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-gxnf5"]
Sep 09 02:37:48.992243 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:48.992210    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-gxnf5"]
Sep 09 02:37:48.992416 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:48.992334    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-gxnf5"
Sep 09 02:37:48.996316 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:48.996294    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 09 02:37:48.996440 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:48.996396    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-8s9rv\""
Sep 09 02:37:49.103207 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:49.103170    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9x44\" (UniqueName: \"kubernetes.io/projected/af6f2234-9c64-45a6-9129-c708d33ad253-kube-api-access-l9x44\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-gxnf5\" (UID: \"af6f2234-9c64-45a6-9129-c708d33ad253\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-gxnf5"
Sep 09 02:37:49.103207 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:49.103208    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af6f2234-9c64-45a6-9129-c708d33ad253-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-gxnf5\" (UID: \"af6f2234-9c64-45a6-9129-c708d33ad253\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-gxnf5"
Sep 09 02:37:49.103373 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:49.103237    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af6f2234-9c64-45a6-9129-c708d33ad253-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-gxnf5\" (UID: \"af6f2234-9c64-45a6-9129-c708d33ad253\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-gxnf5"
Sep 09 02:37:49.103373 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:49.103254    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af6f2234-9c64-45a6-9129-c708d33ad253-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-gxnf5\" (UID: \"af6f2234-9c64-45a6-9129-c708d33ad253\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-gxnf5"
Sep 09 02:37:49.103373 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:49.103294    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af6f2234-9c64-45a6-9129-c708d33ad253-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-gxnf5\" (UID: \"af6f2234-9c64-45a6-9129-c708d33ad253\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-gxnf5"
Sep 09 02:37:49.103373 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:49.103311    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af6f2234-9c64-45a6-9129-c708d33ad253-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-gxnf5\" (UID: \"af6f2234-9c64-45a6-9129-c708d33ad253\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-gxnf5"
Sep 09 02:37:49.103373 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:49.103358    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af6f2234-9c64-45a6-9129-c708d33ad253-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-gxnf5\" (UID: \"af6f2234-9c64-45a6-9129-c708d33ad253\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-gxnf5"
Sep 09 02:37:49.103522 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:49.103390    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af6f2234-9c64-45a6-9129-c708d33ad253-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-gxnf5\" (UID: \"af6f2234-9c64-45a6-9129-c708d33ad253\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-gxnf5"
Sep 09 02:37:49.103522 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:49.103411    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af6f2234-9c64-45a6-9129-c708d33ad253-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-gxnf5\" (UID: \"af6f2234-9c64-45a6-9129-c708d33ad253\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-gxnf5"
Sep 09 02:37:49.204157 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:49.204086    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af6f2234-9c64-45a6-9129-c708d33ad253-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-gxnf5\" (UID: \"af6f2234-9c64-45a6-9129-c708d33ad253\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-gxnf5"
Sep 09 02:37:49.204157 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:49.204117    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af6f2234-9c64-45a6-9129-c708d33ad253-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-gxnf5\" (UID: \"af6f2234-9c64-45a6-9129-c708d33ad253\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-gxnf5"
Sep 09 02:37:49.204157 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:49.204137    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af6f2234-9c64-45a6-9129-c708d33ad253-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-gxnf5\" (UID: \"af6f2234-9c64-45a6-9129-c708d33ad253\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-gxnf5"
Sep 09 02:37:49.204157 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:49.204157    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af6f2234-9c64-45a6-9129-c708d33ad253-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-gxnf5\" (UID: \"af6f2234-9c64-45a6-9129-c708d33ad253\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-gxnf5"
Sep 09 02:37:49.204453 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:49.204190    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af6f2234-9c64-45a6-9129-c708d33ad253-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-gxnf5\" (UID: \"af6f2234-9c64-45a6-9129-c708d33ad253\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-gxnf5"
Sep 09 02:37:49.204453 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:49.204216    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af6f2234-9c64-45a6-9129-c708d33ad253-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-gxnf5\" (UID: \"af6f2234-9c64-45a6-9129-c708d33ad253\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-gxnf5"
Sep 09 02:37:49.204453 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:49.204240    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af6f2234-9c64-45a6-9129-c708d33ad253-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-gxnf5\" (UID: \"af6f2234-9c64-45a6-9129-c708d33ad253\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-gxnf5"
Sep 09 02:37:49.204453 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:49.204297    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9x44\" (UniqueName: \"kubernetes.io/projected/af6f2234-9c64-45a6-9129-c708d33ad253-kube-api-access-l9x44\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-gxnf5\" (UID: \"af6f2234-9c64-45a6-9129-c708d33ad253\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-gxnf5"
Sep 09 02:37:49.204453 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:49.204329    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af6f2234-9c64-45a6-9129-c708d33ad253-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-gxnf5\" (UID: \"af6f2234-9c64-45a6-9129-c708d33ad253\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-gxnf5"
Sep 09 02:37:49.204690 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:49.204565    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af6f2234-9c64-45a6-9129-c708d33ad253-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-gxnf5\" (UID: \"af6f2234-9c64-45a6-9129-c708d33ad253\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-gxnf5"
Sep 09 02:37:49.204690 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:49.204646    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af6f2234-9c64-45a6-9129-c708d33ad253-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-gxnf5\" (UID: \"af6f2234-9c64-45a6-9129-c708d33ad253\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-gxnf5"
Sep 09 02:37:49.204843 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:49.204815    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af6f2234-9c64-45a6-9129-c708d33ad253-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-gxnf5\" (UID: \"af6f2234-9c64-45a6-9129-c708d33ad253\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-gxnf5"
Sep 09 02:37:49.204966 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:49.204839    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af6f2234-9c64-45a6-9129-c708d33ad253-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-gxnf5\" (UID: \"af6f2234-9c64-45a6-9129-c708d33ad253\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-gxnf5"
Sep 09 02:37:49.204966 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:49.204913    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af6f2234-9c64-45a6-9129-c708d33ad253-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-gxnf5\" (UID: \"af6f2234-9c64-45a6-9129-c708d33ad253\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-gxnf5"
Sep 09 02:37:49.206568 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:49.206539    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af6f2234-9c64-45a6-9129-c708d33ad253-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-gxnf5\" (UID: \"af6f2234-9c64-45a6-9129-c708d33ad253\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-gxnf5"
Sep 09 02:37:49.206644 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:49.206614    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af6f2234-9c64-45a6-9129-c708d33ad253-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-gxnf5\" (UID: \"af6f2234-9c64-45a6-9129-c708d33ad253\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-gxnf5"
Sep 09 02:37:49.214296 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:49.214277    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af6f2234-9c64-45a6-9129-c708d33ad253-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-gxnf5\" (UID: \"af6f2234-9c64-45a6-9129-c708d33ad253\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-gxnf5"
Sep 09 02:37:49.215182 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:49.215165    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9x44\" (UniqueName: \"kubernetes.io/projected/af6f2234-9c64-45a6-9129-c708d33ad253-kube-api-access-l9x44\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-gxnf5\" (UID: \"af6f2234-9c64-45a6-9129-c708d33ad253\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-gxnf5"
Sep 09 02:37:49.302847 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:49.302818    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-gxnf5"
Sep 09 02:37:49.607105 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:49.607063    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-gxnf5"]
Sep 09 02:37:49.609647 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:37:49.609620    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf6f2234_9c64_45a6_9129_c708d33ad253.slice/crio-beaddb2ce36293a70f87ff963c5c3259c0a82157ca0767512868f4a32212765e WatchSource:0}: Error finding container beaddb2ce36293a70f87ff963c5c3259c0a82157ca0767512868f4a32212765e: Status 404 returned error can't find the container with id beaddb2ce36293a70f87ff963c5c3259c0a82157ca0767512868f4a32212765e
Sep 09 02:37:49.997140 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:49.997059    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-gxnf5" event={"ID":"af6f2234-9c64-45a6-9129-c708d33ad253","Type":"ContainerStarted","Data":"beaddb2ce36293a70f87ff963c5c3259c0a82157ca0767512868f4a32212765e"}
Sep 09 02:37:51.882266 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:51.882231    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-mx5jk"]
Sep 09 02:37:51.888769 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:51.888738    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-mx5jk"
Sep 09 02:37:51.891758 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:51.891735    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Sep 09 02:37:51.891905 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:51.891860    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-kllg8\""
Sep 09 02:37:51.901461 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:51.901441    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-mx5jk"]
Sep 09 02:37:51.926388 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:51.926356    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/256e03f1-1f0b-491a-8381-1280c89caa9f-cert\") pod \"odh-model-controller-7998c67dc4-mx5jk\" (UID: \"256e03f1-1f0b-491a-8381-1280c89caa9f\") " pod="opendatahub/odh-model-controller-7998c67dc4-mx5jk"
Sep 09 02:37:51.926502 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:51.926409    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgtb5\" (UniqueName: \"kubernetes.io/projected/256e03f1-1f0b-491a-8381-1280c89caa9f-kube-api-access-tgtb5\") pod \"odh-model-controller-7998c67dc4-mx5jk\" (UID: \"256e03f1-1f0b-491a-8381-1280c89caa9f\") " pod="opendatahub/odh-model-controller-7998c67dc4-mx5jk"
Sep 09 02:37:51.987355 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:51.987320    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-756c87c457-jhn45" podUID="87da7a65-1f6c-47ae-8325-aa9aa040fda3" containerName="console" containerID="cri-o://93fd96709994e9447baf0660f5e000f3daaea90f9d9fbe3b8411ea13d730fb52" gracePeriod=15
Sep 09 02:37:52.026948 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:52.026923    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/256e03f1-1f0b-491a-8381-1280c89caa9f-cert\") pod \"odh-model-controller-7998c67dc4-mx5jk\" (UID: \"256e03f1-1f0b-491a-8381-1280c89caa9f\") " pod="opendatahub/odh-model-controller-7998c67dc4-mx5jk"
Sep 09 02:37:52.027073 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:52.026958    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgtb5\" (UniqueName: \"kubernetes.io/projected/256e03f1-1f0b-491a-8381-1280c89caa9f-kube-api-access-tgtb5\") pod \"odh-model-controller-7998c67dc4-mx5jk\" (UID: \"256e03f1-1f0b-491a-8381-1280c89caa9f\") " pod="opendatahub/odh-model-controller-7998c67dc4-mx5jk"
Sep 09 02:37:52.027117 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:37:52.027096    2570 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Sep 09 02:37:52.027173 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:37:52.027160    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/256e03f1-1f0b-491a-8381-1280c89caa9f-cert podName:256e03f1-1f0b-491a-8381-1280c89caa9f nodeName:}" failed. No retries permitted until 2026-09-09 02:37:52.52713894 +0000 UTC m=+454.781442555 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/256e03f1-1f0b-491a-8381-1280c89caa9f-cert") pod "odh-model-controller-7998c67dc4-mx5jk" (UID: "256e03f1-1f0b-491a-8381-1280c89caa9f") : secret "odh-model-controller-webhook-cert" not found
Sep 09 02:37:52.036959 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:52.036925    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgtb5\" (UniqueName: \"kubernetes.io/projected/256e03f1-1f0b-491a-8381-1280c89caa9f-kube-api-access-tgtb5\") pod \"odh-model-controller-7998c67dc4-mx5jk\" (UID: \"256e03f1-1f0b-491a-8381-1280c89caa9f\") " pod="opendatahub/odh-model-controller-7998c67dc4-mx5jk"
Sep 09 02:37:52.530057 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:52.530018    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-7z594"]
Sep 09 02:37:52.531435 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:52.531329    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/256e03f1-1f0b-491a-8381-1280c89caa9f-cert\") pod \"odh-model-controller-7998c67dc4-mx5jk\" (UID: \"256e03f1-1f0b-491a-8381-1280c89caa9f\") " pod="opendatahub/odh-model-controller-7998c67dc4-mx5jk"
Sep 09 02:37:52.534076 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:52.534050    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/256e03f1-1f0b-491a-8381-1280c89caa9f-cert\") pod \"odh-model-controller-7998c67dc4-mx5jk\" (UID: \"256e03f1-1f0b-491a-8381-1280c89caa9f\") " pod="opendatahub/odh-model-controller-7998c67dc4-mx5jk"
Sep 09 02:37:52.538511 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:52.538489    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-7z594"
Sep 09 02:37:52.545123 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:52.545097    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Sep 09 02:37:52.547049 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:52.546991    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-7z594"]
Sep 09 02:37:52.631873 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:52.631842    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ab4a0e6-a9fd-40a4-b403-964a44380c43-tls-certs\") pod \"model-serving-api-54d89bd5fc-7z594\" (UID: \"5ab4a0e6-a9fd-40a4-b403-964a44380c43\") " pod="opendatahub/model-serving-api-54d89bd5fc-7z594"
Sep 09 02:37:52.632016 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:52.631909    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwdf5\" (UniqueName: \"kubernetes.io/projected/5ab4a0e6-a9fd-40a4-b403-964a44380c43-kube-api-access-xwdf5\") pod \"model-serving-api-54d89bd5fc-7z594\" (UID: \"5ab4a0e6-a9fd-40a4-b403-964a44380c43\") " pod="opendatahub/model-serving-api-54d89bd5fc-7z594"
Sep 09 02:37:52.642516 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:52.642481    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 02:37:52.642624 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:52.642571    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 02:37:52.642624 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:52.642616    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 02:37:52.718591 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:52.718565    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-756c87c457-jhn45"
Sep 09 02:37:52.732613 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:52.732584    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ab4a0e6-a9fd-40a4-b403-964a44380c43-tls-certs\") pod \"model-serving-api-54d89bd5fc-7z594\" (UID: \"5ab4a0e6-a9fd-40a4-b403-964a44380c43\") " pod="opendatahub/model-serving-api-54d89bd5fc-7z594"
Sep 09 02:37:52.732744 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:52.732658    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwdf5\" (UniqueName: \"kubernetes.io/projected/5ab4a0e6-a9fd-40a4-b403-964a44380c43-kube-api-access-xwdf5\") pod \"model-serving-api-54d89bd5fc-7z594\" (UID: \"5ab4a0e6-a9fd-40a4-b403-964a44380c43\") " pod="opendatahub/model-serving-api-54d89bd5fc-7z594"
Sep 09 02:37:52.732744 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:37:52.732724    2570 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found
Sep 09 02:37:52.732836 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:37:52.732771    2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ab4a0e6-a9fd-40a4-b403-964a44380c43-tls-certs podName:5ab4a0e6-a9fd-40a4-b403-964a44380c43 nodeName:}" failed. No retries permitted until 2026-09-09 02:37:53.232758416 +0000 UTC m=+455.487062030 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/5ab4a0e6-a9fd-40a4-b403-964a44380c43-tls-certs") pod "model-serving-api-54d89bd5fc-7z594" (UID: "5ab4a0e6-a9fd-40a4-b403-964a44380c43") : secret "model-serving-api-tls" not found
Sep 09 02:37:52.744394 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:52.744369    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwdf5\" (UniqueName: \"kubernetes.io/projected/5ab4a0e6-a9fd-40a4-b403-964a44380c43-kube-api-access-xwdf5\") pod \"model-serving-api-54d89bd5fc-7z594\" (UID: \"5ab4a0e6-a9fd-40a4-b403-964a44380c43\") " pod="opendatahub/model-serving-api-54d89bd5fc-7z594"
Sep 09 02:37:52.800903 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:52.800826    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-mx5jk"
Sep 09 02:37:52.833209 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:52.833184    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/87da7a65-1f6c-47ae-8325-aa9aa040fda3-oauth-serving-cert\") pod \"87da7a65-1f6c-47ae-8325-aa9aa040fda3\" (UID: \"87da7a65-1f6c-47ae-8325-aa9aa040fda3\") "
Sep 09 02:37:52.833355 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:52.833239    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/87da7a65-1f6c-47ae-8325-aa9aa040fda3-console-serving-cert\") pod \"87da7a65-1f6c-47ae-8325-aa9aa040fda3\" (UID: \"87da7a65-1f6c-47ae-8325-aa9aa040fda3\") "
Sep 09 02:37:52.833355 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:52.833280    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/87da7a65-1f6c-47ae-8325-aa9aa040fda3-console-config\") pod \"87da7a65-1f6c-47ae-8325-aa9aa040fda3\" (UID: \"87da7a65-1f6c-47ae-8325-aa9aa040fda3\") "
Sep 09 02:37:52.833355 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:52.833322    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pgbfj\" (UniqueName: \"kubernetes.io/projected/87da7a65-1f6c-47ae-8325-aa9aa040fda3-kube-api-access-pgbfj\") pod \"87da7a65-1f6c-47ae-8325-aa9aa040fda3\" (UID: \"87da7a65-1f6c-47ae-8325-aa9aa040fda3\") "
Sep 09 02:37:52.833503 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:52.833357    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/87da7a65-1f6c-47ae-8325-aa9aa040fda3-console-oauth-config\") pod \"87da7a65-1f6c-47ae-8325-aa9aa040fda3\" (UID: \"87da7a65-1f6c-47ae-8325-aa9aa040fda3\") "
Sep 09 02:37:52.833503 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:52.833413    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/87da7a65-1f6c-47ae-8325-aa9aa040fda3-service-ca\") pod \"87da7a65-1f6c-47ae-8325-aa9aa040fda3\" (UID: \"87da7a65-1f6c-47ae-8325-aa9aa040fda3\") "
Sep 09 02:37:52.833503 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:52.833459    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/87da7a65-1f6c-47ae-8325-aa9aa040fda3-trusted-ca-bundle\") pod \"87da7a65-1f6c-47ae-8325-aa9aa040fda3\" (UID: \"87da7a65-1f6c-47ae-8325-aa9aa040fda3\") "
Sep 09 02:37:52.833644 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:52.833561    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87da7a65-1f6c-47ae-8325-aa9aa040fda3-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "87da7a65-1f6c-47ae-8325-aa9aa040fda3" (UID: "87da7a65-1f6c-47ae-8325-aa9aa040fda3"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 02:37:52.833870 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:52.833846    2570 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/87da7a65-1f6c-47ae-8325-aa9aa040fda3-oauth-serving-cert\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:37:52.833946 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:52.833880    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87da7a65-1f6c-47ae-8325-aa9aa040fda3-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "87da7a65-1f6c-47ae-8325-aa9aa040fda3" (UID: "87da7a65-1f6c-47ae-8325-aa9aa040fda3"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 02:37:52.834242 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:52.834192    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87da7a65-1f6c-47ae-8325-aa9aa040fda3-console-config" (OuterVolumeSpecName: "console-config") pod "87da7a65-1f6c-47ae-8325-aa9aa040fda3" (UID: "87da7a65-1f6c-47ae-8325-aa9aa040fda3"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 02:37:52.834242 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:52.834209    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87da7a65-1f6c-47ae-8325-aa9aa040fda3-service-ca" (OuterVolumeSpecName: "service-ca") pod "87da7a65-1f6c-47ae-8325-aa9aa040fda3" (UID: "87da7a65-1f6c-47ae-8325-aa9aa040fda3"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 02:37:52.835700 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:52.835676    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87da7a65-1f6c-47ae-8325-aa9aa040fda3-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "87da7a65-1f6c-47ae-8325-aa9aa040fda3" (UID: "87da7a65-1f6c-47ae-8325-aa9aa040fda3"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 02:37:52.835819 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:52.835795    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87da7a65-1f6c-47ae-8325-aa9aa040fda3-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "87da7a65-1f6c-47ae-8325-aa9aa040fda3" (UID: "87da7a65-1f6c-47ae-8325-aa9aa040fda3"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 02:37:52.836201 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:52.836180    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87da7a65-1f6c-47ae-8325-aa9aa040fda3-kube-api-access-pgbfj" (OuterVolumeSpecName: "kube-api-access-pgbfj") pod "87da7a65-1f6c-47ae-8325-aa9aa040fda3" (UID: "87da7a65-1f6c-47ae-8325-aa9aa040fda3"). InnerVolumeSpecName "kube-api-access-pgbfj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 02:37:52.934613 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:52.934575    2570 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/87da7a65-1f6c-47ae-8325-aa9aa040fda3-trusted-ca-bundle\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:37:52.934613 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:52.934611    2570 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/87da7a65-1f6c-47ae-8325-aa9aa040fda3-console-serving-cert\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:37:52.934613 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:52.934621    2570 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/87da7a65-1f6c-47ae-8325-aa9aa040fda3-console-config\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:37:52.935007 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:52.934630    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgbfj\" (UniqueName: \"kubernetes.io/projected/87da7a65-1f6c-47ae-8325-aa9aa040fda3-kube-api-access-pgbfj\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:37:52.935007 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:52.934638    2570 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/87da7a65-1f6c-47ae-8325-aa9aa040fda3-console-oauth-config\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:37:52.935007 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:52.934647    2570 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/87da7a65-1f6c-47ae-8325-aa9aa040fda3-service-ca\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:37:53.012533 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:53.012499    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-gxnf5" event={"ID":"af6f2234-9c64-45a6-9129-c708d33ad253","Type":"ContainerStarted","Data":"1269a6552d80374a4df02ee9c4602367dece6adc51e72b98dea3d59bf76005ea"}
Sep 09 02:37:53.013849 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:53.013827    2570 generic.go:356] "Generic (PLEG): container finished" podID="87da7a65-1f6c-47ae-8325-aa9aa040fda3" containerID="93fd96709994e9447baf0660f5e000f3daaea90f9d9fbe3b8411ea13d730fb52" exitCode=0
Sep 09 02:37:53.013958 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:53.013874    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-756c87c457-jhn45"
Sep 09 02:37:53.013958 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:53.013888    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-756c87c457-jhn45" event={"ID":"87da7a65-1f6c-47ae-8325-aa9aa040fda3","Type":"ContainerDied","Data":"93fd96709994e9447baf0660f5e000f3daaea90f9d9fbe3b8411ea13d730fb52"}
Sep 09 02:37:53.013958 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:53.013916    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-756c87c457-jhn45" event={"ID":"87da7a65-1f6c-47ae-8325-aa9aa040fda3","Type":"ContainerDied","Data":"0828511a29591fda242d33583bb9618656cc36d1d2b27c3e1ebb6046bb34ddbd"}
Sep 09 02:37:53.013958 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:53.013936    2570 scope.go:117] "RemoveContainer" containerID="93fd96709994e9447baf0660f5e000f3daaea90f9d9fbe3b8411ea13d730fb52"
Sep 09 02:37:53.023120 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:53.023104    2570 scope.go:117] "RemoveContainer" containerID="93fd96709994e9447baf0660f5e000f3daaea90f9d9fbe3b8411ea13d730fb52"
Sep 09 02:37:53.023356 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:37:53.023337    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93fd96709994e9447baf0660f5e000f3daaea90f9d9fbe3b8411ea13d730fb52\": container with ID starting with 93fd96709994e9447baf0660f5e000f3daaea90f9d9fbe3b8411ea13d730fb52 not found: ID does not exist" containerID="93fd96709994e9447baf0660f5e000f3daaea90f9d9fbe3b8411ea13d730fb52"
Sep 09 02:37:53.023408 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:53.023365    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93fd96709994e9447baf0660f5e000f3daaea90f9d9fbe3b8411ea13d730fb52"} err="failed to get container status \"93fd96709994e9447baf0660f5e000f3daaea90f9d9fbe3b8411ea13d730fb52\": rpc error: code = NotFound desc = could not find container \"93fd96709994e9447baf0660f5e000f3daaea90f9d9fbe3b8411ea13d730fb52\": container with ID starting with 93fd96709994e9447baf0660f5e000f3daaea90f9d9fbe3b8411ea13d730fb52 not found: ID does not exist"
Sep 09 02:37:53.040584 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:53.040545    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-gxnf5" podStartSLOduration=2.010177359 podStartE2EDuration="5.040532449s" totalImagesPullingTime="3.03035509s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:37:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:37:53.038701391 +0000 UTC m=+455.293005027" watchObservedRunningTime="2026-09-09 02:37:53.040532449 +0000 UTC m=+455.294836084"
Sep 09 02:37:53.058105 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:53.058052    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-756c87c457-jhn45"]
Sep 09 02:37:53.067906 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:53.067887    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-756c87c457-jhn45"]
Sep 09 02:37:53.085122 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:53.085100    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-mx5jk"]
Sep 09 02:37:53.088282 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:37:53.088261    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod256e03f1_1f0b_491a_8381_1280c89caa9f.slice/crio-403b42e3cf2ae4270d1d517033a8a310a93d2c3234ab04afd108fa14f87276b4 WatchSource:0}: Error finding container 403b42e3cf2ae4270d1d517033a8a310a93d2c3234ab04afd108fa14f87276b4: Status 404 returned error can't find the container with id 403b42e3cf2ae4270d1d517033a8a310a93d2c3234ab04afd108fa14f87276b4
Sep 09 02:37:53.237126 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:53.237094    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ab4a0e6-a9fd-40a4-b403-964a44380c43-tls-certs\") pod \"model-serving-api-54d89bd5fc-7z594\" (UID: \"5ab4a0e6-a9fd-40a4-b403-964a44380c43\") " pod="opendatahub/model-serving-api-54d89bd5fc-7z594"
Sep 09 02:37:53.239527 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:53.239500    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ab4a0e6-a9fd-40a4-b403-964a44380c43-tls-certs\") pod \"model-serving-api-54d89bd5fc-7z594\" (UID: \"5ab4a0e6-a9fd-40a4-b403-964a44380c43\") " pod="opendatahub/model-serving-api-54d89bd5fc-7z594"
Sep 09 02:37:53.303808 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:53.303779    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-gxnf5"
Sep 09 02:37:53.308010 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:53.307970    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-gxnf5"
Sep 09 02:37:53.453469 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:53.453427    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-7z594"
Sep 09 02:37:53.729236 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:53.729076    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-7z594"]
Sep 09 02:37:53.731645 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:37:53.731576    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ab4a0e6_a9fd_40a4_b403_964a44380c43.slice/crio-23fae546b2401864d19d5dfc66b4b0a83c9e0df9a7a8c4f23f0ad11972568fdd WatchSource:0}: Error finding container 23fae546b2401864d19d5dfc66b4b0a83c9e0df9a7a8c4f23f0ad11972568fdd: Status 404 returned error can't find the container with id 23fae546b2401864d19d5dfc66b4b0a83c9e0df9a7a8c4f23f0ad11972568fdd
Sep 09 02:37:54.022786 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:54.022706    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-7z594" event={"ID":"5ab4a0e6-a9fd-40a4-b403-964a44380c43","Type":"ContainerStarted","Data":"23fae546b2401864d19d5dfc66b4b0a83c9e0df9a7a8c4f23f0ad11972568fdd"}
Sep 09 02:37:54.025125 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:54.025094    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-mx5jk" event={"ID":"256e03f1-1f0b-491a-8381-1280c89caa9f","Type":"ContainerStarted","Data":"403b42e3cf2ae4270d1d517033a8a310a93d2c3234ab04afd108fa14f87276b4"}
Sep 09 02:37:54.025450 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:54.025364    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-gxnf5"
Sep 09 02:37:54.026628 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:54.026579    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-gxnf5"
Sep 09 02:37:54.225935 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:54.225894    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87da7a65-1f6c-47ae-8325-aa9aa040fda3" path="/var/lib/kubelet/pods/87da7a65-1f6c-47ae-8325-aa9aa040fda3/volumes"
Sep 09 02:37:58.047030 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:58.046970    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-mx5jk" event={"ID":"256e03f1-1f0b-491a-8381-1280c89caa9f","Type":"ContainerStarted","Data":"6e4a788a7c8da5866fd43fe77c5396805003b92769e0013e4adc1bbd905f54b3"}
Sep 09 02:37:58.047470 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:58.047090    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-mx5jk"
Sep 09 02:37:58.048520 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:58.048492    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-7z594" event={"ID":"5ab4a0e6-a9fd-40a4-b403-964a44380c43","Type":"ContainerStarted","Data":"dbfe08eeeaa3aaccde0bdd7624d6c759a100a4e2d9b177fe0660367c7f35d011"}
Sep 09 02:37:58.048652 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:58.048633    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-7z594"
Sep 09 02:37:58.068247 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:58.068206    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-mx5jk" podStartSLOduration=3.187325507 podStartE2EDuration="7.068193485s" totalImagesPullingTime="3.880867978s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:37:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:37:58.067818303 +0000 UTC m=+460.322121940" watchObservedRunningTime="2026-09-09 02:37:58.068193485 +0000 UTC m=+460.322497121"
Sep 09 02:37:58.095103 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:37:58.095065    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-7z594" podStartSLOduration=2.817838192 podStartE2EDuration="6.09505125s" totalImagesPullingTime="3.277213058s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:37:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:37:58.094162337 +0000 UTC m=+460.348465973" watchObservedRunningTime="2026-09-09 02:37:58.09505125 +0000 UTC m=+460.349354886"
Sep 09 02:38:09.054120 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:38:09.054087    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-mx5jk"
Sep 09 02:38:09.056212 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:38:09.056191    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-7z594"
Sep 09 02:39:11.682071 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:39:11.681983    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-j8zkb"]
Sep 09 02:39:11.682574 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:39:11.682468    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87da7a65-1f6c-47ae-8325-aa9aa040fda3" containerName="console"
Sep 09 02:39:11.682574 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:39:11.682488    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="87da7a65-1f6c-47ae-8325-aa9aa040fda3" containerName="console"
Sep 09 02:39:11.682693 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:39:11.682577    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="87da7a65-1f6c-47ae-8325-aa9aa040fda3" containerName="console"
Sep 09 02:39:11.686493 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:39:11.686472    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-j8zkb"
Sep 09 02:39:11.690077 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:39:11.690057    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 09 02:39:11.690192 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:39:11.690065    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 09 02:39:11.691424 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:39:11.691408    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-w2w2b\""
Sep 09 02:39:11.697348 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:39:11.697328    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-j8zkb"]
Sep 09 02:39:11.703034 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:39:11.703005    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hr88r\" (UniqueName: \"kubernetes.io/projected/a46e6a0e-0beb-4dc2-be59-d82b625dbed6-kube-api-access-hr88r\") pod \"authorino-operator-76b478bd78-j8zkb\" (UID: \"a46e6a0e-0beb-4dc2-be59-d82b625dbed6\") " pod="kuadrant-system/authorino-operator-76b478bd78-j8zkb"
Sep 09 02:39:11.804383 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:39:11.804343    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hr88r\" (UniqueName: \"kubernetes.io/projected/a46e6a0e-0beb-4dc2-be59-d82b625dbed6-kube-api-access-hr88r\") pod \"authorino-operator-76b478bd78-j8zkb\" (UID: \"a46e6a0e-0beb-4dc2-be59-d82b625dbed6\") " pod="kuadrant-system/authorino-operator-76b478bd78-j8zkb"
Sep 09 02:39:11.824449 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:39:11.824421    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hr88r\" (UniqueName: \"kubernetes.io/projected/a46e6a0e-0beb-4dc2-be59-d82b625dbed6-kube-api-access-hr88r\") pod \"authorino-operator-76b478bd78-j8zkb\" (UID: \"a46e6a0e-0beb-4dc2-be59-d82b625dbed6\") " pod="kuadrant-system/authorino-operator-76b478bd78-j8zkb"
Sep 09 02:39:11.997147 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:39:11.997041    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-j8zkb"
Sep 09 02:39:12.276400 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:39:12.276375    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-j8zkb"]
Sep 09 02:39:12.278999 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:39:12.278955    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda46e6a0e_0beb_4dc2_be59_d82b625dbed6.slice/crio-06d12371794610e557f8968dcd999ea705a155d0429e6376918e0d162eb5990e WatchSource:0}: Error finding container 06d12371794610e557f8968dcd999ea705a155d0429e6376918e0d162eb5990e: Status 404 returned error can't find the container with id 06d12371794610e557f8968dcd999ea705a155d0429e6376918e0d162eb5990e
Sep 09 02:39:12.364741 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:39:12.364708    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-j8zkb" event={"ID":"a46e6a0e-0beb-4dc2-be59-d82b625dbed6","Type":"ContainerStarted","Data":"06d12371794610e557f8968dcd999ea705a155d0429e6376918e0d162eb5990e"}
Sep 09 02:39:15.385426 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:39:15.385388    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-j8zkb" event={"ID":"a46e6a0e-0beb-4dc2-be59-d82b625dbed6","Type":"ContainerStarted","Data":"732a77d48e6c68444a94bb7f9aec9df33a7b592fd75706976a7535f514c0813c"}
Sep 09 02:39:15.385893 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:39:15.385501    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-j8zkb"
Sep 09 02:39:15.408843 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:39:15.408772    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-76b478bd78-j8zkb" podStartSLOduration=1.8830884270000001 podStartE2EDuration="4.408756846s" totalImagesPullingTime="2.525668419s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:39:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:39:15.407836826 +0000 UTC m=+537.662140463" watchObservedRunningTime="2026-09-09 02:39:15.408756846 +0000 UTC m=+537.663060479"
Sep 09 02:39:26.393006 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:39:26.392957    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-j8zkb"
Sep 09 02:39:27.942782 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:39:27.942744    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-r2j5r"]
Sep 09 02:39:27.945933 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:39:27.945915    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-r2j5r"
Sep 09 02:39:27.951651 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:39:27.951627    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 09 02:39:27.951782 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:39:27.951627    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 09 02:39:27.951782 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:39:27.951668    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-xm4tk\""
Sep 09 02:39:27.960814 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:39:27.960791    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-r2j5r"]
Sep 09 02:39:28.038807 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:39:28.038773    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c90ca26e-f6b3-4e41-9458-5a64bb1db802-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-r2j5r\" (UID: \"c90ca26e-f6b3-4e41-9458-5a64bb1db802\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-r2j5r"
Sep 09 02:39:28.038968 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:39:28.038814    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/c90ca26e-f6b3-4e41-9458-5a64bb1db802-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-r2j5r\" (UID: \"c90ca26e-f6b3-4e41-9458-5a64bb1db802\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-r2j5r"
Sep 09 02:39:28.038968 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:39:28.038833    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtbtl\" (UniqueName: \"kubernetes.io/projected/c90ca26e-f6b3-4e41-9458-5a64bb1db802-kube-api-access-rtbtl\") pod \"kuadrant-console-plugin-fbdd4f445-r2j5r\" (UID: \"c90ca26e-f6b3-4e41-9458-5a64bb1db802\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-r2j5r"
Sep 09 02:39:28.140296 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:39:28.140248    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c90ca26e-f6b3-4e41-9458-5a64bb1db802-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-r2j5r\" (UID: \"c90ca26e-f6b3-4e41-9458-5a64bb1db802\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-r2j5r"
Sep 09 02:39:28.140464 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:39:28.140306    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/c90ca26e-f6b3-4e41-9458-5a64bb1db802-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-r2j5r\" (UID: \"c90ca26e-f6b3-4e41-9458-5a64bb1db802\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-r2j5r"
Sep 09 02:39:28.140464 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:39:28.140334    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtbtl\" (UniqueName: \"kubernetes.io/projected/c90ca26e-f6b3-4e41-9458-5a64bb1db802-kube-api-access-rtbtl\") pod \"kuadrant-console-plugin-fbdd4f445-r2j5r\" (UID: \"c90ca26e-f6b3-4e41-9458-5a64bb1db802\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-r2j5r"
Sep 09 02:39:28.140904 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:39:28.140883    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c90ca26e-f6b3-4e41-9458-5a64bb1db802-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-r2j5r\" (UID: \"c90ca26e-f6b3-4e41-9458-5a64bb1db802\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-r2j5r"
Sep 09 02:39:28.142753 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:39:28.142725    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/c90ca26e-f6b3-4e41-9458-5a64bb1db802-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-r2j5r\" (UID: \"c90ca26e-f6b3-4e41-9458-5a64bb1db802\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-r2j5r"
Sep 09 02:39:28.150791 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:39:28.150769    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtbtl\" (UniqueName: \"kubernetes.io/projected/c90ca26e-f6b3-4e41-9458-5a64bb1db802-kube-api-access-rtbtl\") pod \"kuadrant-console-plugin-fbdd4f445-r2j5r\" (UID: \"c90ca26e-f6b3-4e41-9458-5a64bb1db802\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-r2j5r"
Sep 09 02:39:28.254902 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:39:28.254817    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-r2j5r"
Sep 09 02:39:28.531028 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:39:28.531003    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-r2j5r"]
Sep 09 02:39:28.533500 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:39:28.533476    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc90ca26e_f6b3_4e41_9458_5a64bb1db802.slice/crio-a73318a1339c53130a9666076dbb6af044925003d424bc1d7df6a2280ec0434e WatchSource:0}: Error finding container a73318a1339c53130a9666076dbb6af044925003d424bc1d7df6a2280ec0434e: Status 404 returned error can't find the container with id a73318a1339c53130a9666076dbb6af044925003d424bc1d7df6a2280ec0434e
Sep 09 02:39:29.446518 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:39:29.446481    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-r2j5r" event={"ID":"c90ca26e-f6b3-4e41-9458-5a64bb1db802","Type":"ContainerStarted","Data":"a73318a1339c53130a9666076dbb6af044925003d424bc1d7df6a2280ec0434e"}
Sep 09 02:39:33.470083 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:39:33.470035    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-r2j5r" event={"ID":"c90ca26e-f6b3-4e41-9458-5a64bb1db802","Type":"ContainerStarted","Data":"f11ccb3815608c66a5036e909d6b0bd54ff96fa31d41f35764771d0305132ebf"}
Sep 09 02:39:33.488781 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:39:33.488735    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-r2j5r" podStartSLOduration=1.668338646 podStartE2EDuration="6.488716328s" totalImagesPullingTime="4.820377682s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:39:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:39:33.488387196 +0000 UTC m=+555.742690829" watchObservedRunningTime="2026-09-09 02:39:33.488716328 +0000 UTC m=+555.743019961"
Sep 09 02:40:08.262947 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:40:08.262910    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lfhw7"]
Sep 09 02:40:08.285921 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:40:08.285889    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lfhw7"]
Sep 09 02:40:08.286109 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:40:08.286047    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lfhw7"
Sep 09 02:40:08.289375 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:40:08.289354    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-gbhlx\""
Sep 09 02:40:08.294713 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:40:08.294692    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/797e5120-d8b6-4405-8beb-6d48f3391f65-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lfhw7\" (UID: \"797e5120-d8b6-4405-8beb-6d48f3391f65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lfhw7"
Sep 09 02:40:08.294788 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:40:08.294723    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/797e5120-d8b6-4405-8beb-6d48f3391f65-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lfhw7\" (UID: \"797e5120-d8b6-4405-8beb-6d48f3391f65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lfhw7"
Sep 09 02:40:08.294788 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:40:08.294742    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/797e5120-d8b6-4405-8beb-6d48f3391f65-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lfhw7\" (UID: \"797e5120-d8b6-4405-8beb-6d48f3391f65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lfhw7"
Sep 09 02:40:08.294918 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:40:08.294805    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/797e5120-d8b6-4405-8beb-6d48f3391f65-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lfhw7\" (UID: \"797e5120-d8b6-4405-8beb-6d48f3391f65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lfhw7"
Sep 09 02:40:08.294918 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:40:08.294850    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/797e5120-d8b6-4405-8beb-6d48f3391f65-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lfhw7\" (UID: \"797e5120-d8b6-4405-8beb-6d48f3391f65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lfhw7"
Sep 09 02:40:08.294918 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:40:08.294892    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/797e5120-d8b6-4405-8beb-6d48f3391f65-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lfhw7\" (UID: \"797e5120-d8b6-4405-8beb-6d48f3391f65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lfhw7"
Sep 09 02:40:08.295041 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:40:08.294943    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/797e5120-d8b6-4405-8beb-6d48f3391f65-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lfhw7\" (UID: \"797e5120-d8b6-4405-8beb-6d48f3391f65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lfhw7"
Sep 09 02:40:08.295083 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:40:08.295060    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lvwn\" (UniqueName: \"kubernetes.io/projected/797e5120-d8b6-4405-8beb-6d48f3391f65-kube-api-access-4lvwn\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lfhw7\" (UID: \"797e5120-d8b6-4405-8beb-6d48f3391f65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lfhw7"
Sep 09 02:40:08.295119 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:40:08.295088    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/797e5120-d8b6-4405-8beb-6d48f3391f65-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lfhw7\" (UID: \"797e5120-d8b6-4405-8beb-6d48f3391f65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lfhw7"
Sep 09 02:40:08.396236 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:40:08.396198    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/797e5120-d8b6-4405-8beb-6d48f3391f65-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lfhw7\" (UID: \"797e5120-d8b6-4405-8beb-6d48f3391f65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lfhw7"
Sep 09 02:40:08.396236 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:40:08.396236    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/797e5120-d8b6-4405-8beb-6d48f3391f65-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lfhw7\" (UID: \"797e5120-d8b6-4405-8beb-6d48f3391f65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lfhw7"
Sep 09 02:40:08.396444 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:40:08.396255    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/797e5120-d8b6-4405-8beb-6d48f3391f65-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lfhw7\" (UID: \"797e5120-d8b6-4405-8beb-6d48f3391f65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lfhw7"
Sep 09 02:40:08.396444 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:40:08.396290    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/797e5120-d8b6-4405-8beb-6d48f3391f65-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lfhw7\" (UID: \"797e5120-d8b6-4405-8beb-6d48f3391f65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lfhw7"
Sep 09 02:40:08.396444 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:40:08.396420    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/797e5120-d8b6-4405-8beb-6d48f3391f65-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lfhw7\" (UID: \"797e5120-d8b6-4405-8beb-6d48f3391f65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lfhw7"
Sep 09 02:40:08.396562 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:40:08.396477    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/797e5120-d8b6-4405-8beb-6d48f3391f65-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lfhw7\" (UID: \"797e5120-d8b6-4405-8beb-6d48f3391f65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lfhw7"
Sep 09 02:40:08.396562 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:40:08.396534    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/797e5120-d8b6-4405-8beb-6d48f3391f65-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lfhw7\" (UID: \"797e5120-d8b6-4405-8beb-6d48f3391f65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lfhw7"
Sep 09 02:40:08.396664 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:40:08.396597    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lvwn\" (UniqueName: \"kubernetes.io/projected/797e5120-d8b6-4405-8beb-6d48f3391f65-kube-api-access-4lvwn\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lfhw7\" (UID: \"797e5120-d8b6-4405-8beb-6d48f3391f65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lfhw7"
Sep 09 02:40:08.396664 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:40:08.396639    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/797e5120-d8b6-4405-8beb-6d48f3391f65-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lfhw7\" (UID: \"797e5120-d8b6-4405-8beb-6d48f3391f65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lfhw7"
Sep 09 02:40:08.396664 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:40:08.396645    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/797e5120-d8b6-4405-8beb-6d48f3391f65-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lfhw7\" (UID: \"797e5120-d8b6-4405-8beb-6d48f3391f65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lfhw7"
Sep 09 02:40:08.396815 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:40:08.396714    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/797e5120-d8b6-4405-8beb-6d48f3391f65-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lfhw7\" (UID: \"797e5120-d8b6-4405-8beb-6d48f3391f65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lfhw7"
Sep 09 02:40:08.396815 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:40:08.396732    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/797e5120-d8b6-4405-8beb-6d48f3391f65-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lfhw7\" (UID: \"797e5120-d8b6-4405-8beb-6d48f3391f65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lfhw7"
Sep 09 02:40:08.396967 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:40:08.396935    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/797e5120-d8b6-4405-8beb-6d48f3391f65-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lfhw7\" (UID: \"797e5120-d8b6-4405-8beb-6d48f3391f65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lfhw7"
Sep 09 02:40:08.397089 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:40:08.396998    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/797e5120-d8b6-4405-8beb-6d48f3391f65-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lfhw7\" (UID: \"797e5120-d8b6-4405-8beb-6d48f3391f65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lfhw7"
Sep 09 02:40:08.398842 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:40:08.398821    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/797e5120-d8b6-4405-8beb-6d48f3391f65-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lfhw7\" (UID: \"797e5120-d8b6-4405-8beb-6d48f3391f65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lfhw7"
Sep 09 02:40:08.399100 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:40:08.399081    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/797e5120-d8b6-4405-8beb-6d48f3391f65-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lfhw7\" (UID: \"797e5120-d8b6-4405-8beb-6d48f3391f65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lfhw7"
Sep 09 02:40:08.409125 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:40:08.409096    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/797e5120-d8b6-4405-8beb-6d48f3391f65-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lfhw7\" (UID: \"797e5120-d8b6-4405-8beb-6d48f3391f65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lfhw7"
Sep 09 02:40:08.409844 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:40:08.409825    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lvwn\" (UniqueName: \"kubernetes.io/projected/797e5120-d8b6-4405-8beb-6d48f3391f65-kube-api-access-4lvwn\") pod \"maas-default-gateway-openshift-default-78bb78ff78-lfhw7\" (UID: \"797e5120-d8b6-4405-8beb-6d48f3391f65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lfhw7"
Sep 09 02:40:08.596509 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:40:08.596475    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lfhw7"
Sep 09 02:40:08.877439 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:40:08.877412    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lfhw7"]
Sep 09 02:40:08.879744 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:40:08.879719    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod797e5120_d8b6_4405_8beb_6d48f3391f65.slice/crio-4377e58592eb70229221a0973566e593f12821038c1b1151077de3b8d0282e00 WatchSource:0}: Error finding container 4377e58592eb70229221a0973566e593f12821038c1b1151077de3b8d0282e00: Status 404 returned error can't find the container with id 4377e58592eb70229221a0973566e593f12821038c1b1151077de3b8d0282e00
Sep 09 02:40:08.881701 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:40:08.881671    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 02:40:08.881772 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:40:08.881734    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 02:40:08.881772 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:40:08.881763    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 02:40:09.622897 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:40:09.622862    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lfhw7" event={"ID":"797e5120-d8b6-4405-8beb-6d48f3391f65","Type":"ContainerStarted","Data":"3a54cd9ec522997df6caf98eed8b3df410e72632dd3540eee81e96ba40926eef"}
Sep 09 02:40:09.623286 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:40:09.622915    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lfhw7" event={"ID":"797e5120-d8b6-4405-8beb-6d48f3391f65","Type":"ContainerStarted","Data":"4377e58592eb70229221a0973566e593f12821038c1b1151077de3b8d0282e00"}
Sep 09 02:40:09.647035 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:40:09.646983    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lfhw7" podStartSLOduration=1.646956192 podStartE2EDuration="1.646956192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:40:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:40:09.645288367 +0000 UTC m=+591.899592015" watchObservedRunningTime="2026-09-09 02:40:09.646956192 +0000 UTC m=+591.901259828"
Sep 09 02:40:10.596918 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:40:10.596878    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lfhw7"
Sep 09 02:40:10.601993 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:40:10.601940    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lfhw7"
Sep 09 02:40:10.627410 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:40:10.627376    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lfhw7"
Sep 09 02:40:10.628353 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:40:10.628329    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lfhw7"
Sep 09 02:40:23.446796 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:40:23.446761    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-99qjf"]
Sep 09 02:40:23.450154 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:40:23.450136    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-99qjf"
Sep 09 02:40:23.453169 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:40:23.453153    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 09 02:40:23.460681 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:40:23.460660    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-99qjf"]
Sep 09 02:40:23.498132 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:40:23.498098    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-99qjf"]
Sep 09 02:40:23.529508 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:40:23.529465    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e4119393-5203-4016-b923-7950e6111467-config-file\") pod \"limitador-limitador-84994f7ffc-99qjf\" (UID: \"e4119393-5203-4016-b923-7950e6111467\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-99qjf"
Sep 09 02:40:23.529508 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:40:23.529511    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntmz2\" (UniqueName: \"kubernetes.io/projected/e4119393-5203-4016-b923-7950e6111467-kube-api-access-ntmz2\") pod \"limitador-limitador-84994f7ffc-99qjf\" (UID: \"e4119393-5203-4016-b923-7950e6111467\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-99qjf"
Sep 09 02:40:23.630443 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:40:23.630392    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e4119393-5203-4016-b923-7950e6111467-config-file\") pod \"limitador-limitador-84994f7ffc-99qjf\" (UID: \"e4119393-5203-4016-b923-7950e6111467\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-99qjf"
Sep 09 02:40:23.630622 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:40:23.630468    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntmz2\" (UniqueName: \"kubernetes.io/projected/e4119393-5203-4016-b923-7950e6111467-kube-api-access-ntmz2\") pod \"limitador-limitador-84994f7ffc-99qjf\" (UID: \"e4119393-5203-4016-b923-7950e6111467\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-99qjf"
Sep 09 02:40:23.631089 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:40:23.631066    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e4119393-5203-4016-b923-7950e6111467-config-file\") pod \"limitador-limitador-84994f7ffc-99qjf\" (UID: \"e4119393-5203-4016-b923-7950e6111467\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-99qjf"
Sep 09 02:40:23.642681 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:40:23.642653    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntmz2\" (UniqueName: \"kubernetes.io/projected/e4119393-5203-4016-b923-7950e6111467-kube-api-access-ntmz2\") pod \"limitador-limitador-84994f7ffc-99qjf\" (UID: \"e4119393-5203-4016-b923-7950e6111467\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-99qjf"
Sep 09 02:40:23.761238 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:40:23.761155    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-99qjf"
Sep 09 02:40:24.041049 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:40:24.041026    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-99qjf"]
Sep 09 02:40:24.043613 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:40:24.043570    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4119393_5203_4016_b923_7950e6111467.slice/crio-b3b3ff71da7e3b3952771164f97e2eb0c5fe485b9e60194cd86a373ee82879d4 WatchSource:0}: Error finding container b3b3ff71da7e3b3952771164f97e2eb0c5fe485b9e60194cd86a373ee82879d4: Status 404 returned error can't find the container with id b3b3ff71da7e3b3952771164f97e2eb0c5fe485b9e60194cd86a373ee82879d4
Sep 09 02:40:24.054558 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:40:24.054535    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-skh6k"]
Sep 09 02:40:24.058780 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:40:24.058761    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-skh6k"
Sep 09 02:40:24.062187 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:40:24.062168    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-nlct4\""
Sep 09 02:40:24.067334 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:40:24.067313    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-skh6k"]
Sep 09 02:40:24.135378 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:40:24.135342    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8d85l\" (UniqueName: \"kubernetes.io/projected/15de4f04-1fc4-4d94-ac81-b61465c9ec20-kube-api-access-8d85l\") pod \"authorino-59c974bd48-skh6k\" (UID: \"15de4f04-1fc4-4d94-ac81-b61465c9ec20\") " pod="kuadrant-system/authorino-59c974bd48-skh6k"
Sep 09 02:40:24.236479 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:40:24.236427    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8d85l\" (UniqueName: \"kubernetes.io/projected/15de4f04-1fc4-4d94-ac81-b61465c9ec20-kube-api-access-8d85l\") pod \"authorino-59c974bd48-skh6k\" (UID: \"15de4f04-1fc4-4d94-ac81-b61465c9ec20\") " pod="kuadrant-system/authorino-59c974bd48-skh6k"
Sep 09 02:40:24.245419 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:40:24.245387    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8d85l\" (UniqueName: \"kubernetes.io/projected/15de4f04-1fc4-4d94-ac81-b61465c9ec20-kube-api-access-8d85l\") pod \"authorino-59c974bd48-skh6k\" (UID: \"15de4f04-1fc4-4d94-ac81-b61465c9ec20\") " pod="kuadrant-system/authorino-59c974bd48-skh6k"
Sep 09 02:40:24.368675 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:40:24.368639    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-skh6k"
Sep 09 02:40:24.638392 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:40:24.638308    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-skh6k"]
Sep 09 02:40:24.642065 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:40:24.642033    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod15de4f04_1fc4_4d94_ac81_b61465c9ec20.slice/crio-27e7eb6fa421e0bbf792ab7a3045ad017f225f0bf4b2062c201270373efa77cb WatchSource:0}: Error finding container 27e7eb6fa421e0bbf792ab7a3045ad017f225f0bf4b2062c201270373efa77cb: Status 404 returned error can't find the container with id 27e7eb6fa421e0bbf792ab7a3045ad017f225f0bf4b2062c201270373efa77cb
Sep 09 02:40:24.688027 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:40:24.687992    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-skh6k" event={"ID":"15de4f04-1fc4-4d94-ac81-b61465c9ec20","Type":"ContainerStarted","Data":"27e7eb6fa421e0bbf792ab7a3045ad017f225f0bf4b2062c201270373efa77cb"}
Sep 09 02:40:24.689087 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:40:24.689064    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-99qjf" event={"ID":"e4119393-5203-4016-b923-7950e6111467","Type":"ContainerStarted","Data":"b3b3ff71da7e3b3952771164f97e2eb0c5fe485b9e60194cd86a373ee82879d4"}
Sep 09 02:40:25.697223 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:40:25.697184    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-99qjf" event={"ID":"e4119393-5203-4016-b923-7950e6111467","Type":"ContainerStarted","Data":"0508f7db4b6e45dd47a29796c33be547ad02c5c4527111a2d069f783c5f06a81"}
Sep 09 02:40:25.697696 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:40:25.697312    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-99qjf"
Sep 09 02:40:25.715328 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:40:25.715280    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-99qjf" podStartSLOduration=1.45984768 podStartE2EDuration="2.715262737s" totalImagesPullingTime="1.255415057s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:40:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:40:25.714618143 +0000 UTC m=+607.968921780" watchObservedRunningTime="2026-09-09 02:40:25.715262737 +0000 UTC m=+607.969566374"
Sep 09 02:40:26.705105 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:40:26.705073    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-skh6k" event={"ID":"15de4f04-1fc4-4d94-ac81-b61465c9ec20","Type":"ContainerStarted","Data":"83c084dd1f4a9bb43f739657a789c5ffad6f0b6dc08123350a913898e74a45ec"}
Sep 09 02:40:26.724943 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:40:26.724890    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-skh6k" podStartSLOduration=0.816493741 podStartE2EDuration="2.724873982s" totalImagesPullingTime="1.908380241s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:40:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:40:26.724041383 +0000 UTC m=+608.978345020" watchObservedRunningTime="2026-09-09 02:40:26.724873982 +0000 UTC m=+608.979177619"
Sep 09 02:40:36.706526 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:40:36.706491    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-99qjf"
Sep 09 02:40:58.512506 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:40:58.512474    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-skh6k"]
Sep 09 02:40:58.512965 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:40:58.512654    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-skh6k" podUID="15de4f04-1fc4-4d94-ac81-b61465c9ec20" containerName="authorino" containerID="cri-o://83c084dd1f4a9bb43f739657a789c5ffad6f0b6dc08123350a913898e74a45ec" gracePeriod=30
Sep 09 02:40:58.762774 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:40:58.762722    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-skh6k"
Sep 09 02:40:58.836200 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:40:58.836171    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8d85l\" (UniqueName: \"kubernetes.io/projected/15de4f04-1fc4-4d94-ac81-b61465c9ec20-kube-api-access-8d85l\") pod \"15de4f04-1fc4-4d94-ac81-b61465c9ec20\" (UID: \"15de4f04-1fc4-4d94-ac81-b61465c9ec20\") "
Sep 09 02:40:58.838258 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:40:58.838236    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15de4f04-1fc4-4d94-ac81-b61465c9ec20-kube-api-access-8d85l" (OuterVolumeSpecName: "kube-api-access-8d85l") pod "15de4f04-1fc4-4d94-ac81-b61465c9ec20" (UID: "15de4f04-1fc4-4d94-ac81-b61465c9ec20"). InnerVolumeSpecName "kube-api-access-8d85l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 02:40:58.849206 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:40:58.849171    2570 generic.go:356] "Generic (PLEG): container finished" podID="15de4f04-1fc4-4d94-ac81-b61465c9ec20" containerID="83c084dd1f4a9bb43f739657a789c5ffad6f0b6dc08123350a913898e74a45ec" exitCode=0
Sep 09 02:40:58.849284 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:40:58.849212    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-skh6k"
Sep 09 02:40:58.849284 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:40:58.849220    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-skh6k" event={"ID":"15de4f04-1fc4-4d94-ac81-b61465c9ec20","Type":"ContainerDied","Data":"83c084dd1f4a9bb43f739657a789c5ffad6f0b6dc08123350a913898e74a45ec"}
Sep 09 02:40:58.849284 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:40:58.849258    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-skh6k" event={"ID":"15de4f04-1fc4-4d94-ac81-b61465c9ec20","Type":"ContainerDied","Data":"27e7eb6fa421e0bbf792ab7a3045ad017f225f0bf4b2062c201270373efa77cb"}
Sep 09 02:40:58.849284 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:40:58.849279    2570 scope.go:117] "RemoveContainer" containerID="83c084dd1f4a9bb43f739657a789c5ffad6f0b6dc08123350a913898e74a45ec"
Sep 09 02:40:58.857543 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:40:58.857524    2570 scope.go:117] "RemoveContainer" containerID="83c084dd1f4a9bb43f739657a789c5ffad6f0b6dc08123350a913898e74a45ec"
Sep 09 02:40:58.857815 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:40:58.857796    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83c084dd1f4a9bb43f739657a789c5ffad6f0b6dc08123350a913898e74a45ec\": container with ID starting with 83c084dd1f4a9bb43f739657a789c5ffad6f0b6dc08123350a913898e74a45ec not found: ID does not exist" containerID="83c084dd1f4a9bb43f739657a789c5ffad6f0b6dc08123350a913898e74a45ec"
Sep 09 02:40:58.857874 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:40:58.857823    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83c084dd1f4a9bb43f739657a789c5ffad6f0b6dc08123350a913898e74a45ec"} err="failed to get container status \"83c084dd1f4a9bb43f739657a789c5ffad6f0b6dc08123350a913898e74a45ec\": rpc error: code = NotFound desc = could not find container \"83c084dd1f4a9bb43f739657a789c5ffad6f0b6dc08123350a913898e74a45ec\": container with ID starting with 83c084dd1f4a9bb43f739657a789c5ffad6f0b6dc08123350a913898e74a45ec not found: ID does not exist"
Sep 09 02:40:58.871647 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:40:58.871627    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-skh6k"]
Sep 09 02:40:58.876143 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:40:58.876123    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-skh6k"]
Sep 09 02:40:58.937687 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:40:58.937659    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8d85l\" (UniqueName: \"kubernetes.io/projected/15de4f04-1fc4-4d94-ac81-b61465c9ec20-kube-api-access-8d85l\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:41:00.227500 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:41:00.227466    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="15de4f04-1fc4-4d94-ac81-b61465c9ec20" path="/var/lib/kubelet/pods/15de4f04-1fc4-4d94-ac81-b61465c9ec20/volumes"
Sep 09 02:41:41.010710 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:41:41.010673    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-99qjf"]
Sep 09 02:41:50.089847 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:41:50.089818    2570 generic.go:356] "Generic (PLEG): container finished" podID="797e5120-d8b6-4405-8beb-6d48f3391f65" containerID="3a54cd9ec522997df6caf98eed8b3df410e72632dd3540eee81e96ba40926eef" exitCode=0
Sep 09 02:41:50.090258 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:41:50.089903    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lfhw7" event={"ID":"797e5120-d8b6-4405-8beb-6d48f3391f65","Type":"ContainerDied","Data":"3a54cd9ec522997df6caf98eed8b3df410e72632dd3540eee81e96ba40926eef"}
Sep 09 02:41:50.090328 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:41:50.090314    2570 scope.go:117] "RemoveContainer" containerID="3a54cd9ec522997df6caf98eed8b3df410e72632dd3540eee81e96ba40926eef"
Sep 09 02:41:50.091287 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:41:50.091261    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 02:41:50.091336 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:41:50.091324    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 02:41:50.091370 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:41:50.091356    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 02:41:51.095563 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:41:51.095532    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lfhw7" event={"ID":"797e5120-d8b6-4405-8beb-6d48f3391f65","Type":"ContainerStarted","Data":"d81c737366bfbe3860d7efd6cf484186b8d13d9d49dff963a8f7e3574802b96d"}
Sep 09 02:41:51.596757 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:41:51.596725    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lfhw7"
Sep 09 02:41:51.597674 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:41:51.597639    2570 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-lfhw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body=
Sep 09 02:41:51.597791 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:41:51.597696    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lfhw7" podUID="797e5120-d8b6-4405-8beb-6d48f3391f65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused"
Sep 09 02:41:52.597831 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:41:52.597795    2570 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-lfhw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body=
Sep 09 02:41:52.598230 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:41:52.597860    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lfhw7" podUID="797e5120-d8b6-4405-8beb-6d48f3391f65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused"
Sep 09 02:41:53.596864 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:41:53.596829    2570 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-lfhw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body=
Sep 09 02:41:53.597060 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:41:53.596884    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lfhw7" podUID="797e5120-d8b6-4405-8beb-6d48f3391f65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused"
Sep 09 02:41:54.597164 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:41:54.597127    2570 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-lfhw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body=
Sep 09 02:41:54.597539 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:41:54.597178    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lfhw7" podUID="797e5120-d8b6-4405-8beb-6d48f3391f65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused"
Sep 09 02:41:55.597141 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:41:55.597108    2570 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-lfhw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body=
Sep 09 02:41:55.597275 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:41:55.597169    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lfhw7" podUID="797e5120-d8b6-4405-8beb-6d48f3391f65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused"
Sep 09 02:41:55.627298 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:41:55.627275    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lfhw7"
Sep 09 02:41:56.601685 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:41:56.601654    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lfhw7"
Sep 09 02:41:56.602668 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:41:56.602648    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-lfhw7"
Sep 09 02:42:02.963833 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:42:02.963795    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-99qjf"]
Sep 09 02:42:13.248212 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:42:13.248133    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-99qjf"]
Sep 09 02:42:27.942828 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:42:27.942797    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-snsjr"]
Sep 09 02:42:27.943398 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:42:27.943145    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15de4f04-1fc4-4d94-ac81-b61465c9ec20" containerName="authorino"
Sep 09 02:42:27.943398 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:42:27.943157    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="15de4f04-1fc4-4d94-ac81-b61465c9ec20" containerName="authorino"
Sep 09 02:42:27.943398 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:42:27.943212    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="15de4f04-1fc4-4d94-ac81-b61465c9ec20" containerName="authorino"
Sep 09 02:42:27.946247 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:42:27.946231    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-snsjr"
Sep 09 02:42:27.949193 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:42:27.949175    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 09 02:42:27.950394 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:42:27.950378    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 09 02:42:27.950502 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:42:27.950482    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-t6s5c\""
Sep 09 02:42:27.950559 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:42:27.950519    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Sep 09 02:42:27.956349 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:42:27.956327    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-snsjr"]
Sep 09 02:42:28.018786 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:42:28.018753    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c287727b-944e-4401-8697-896eee8a9e71-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-snsjr\" (UID: \"c287727b-944e-4401-8697-896eee8a9e71\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-snsjr"
Sep 09 02:42:28.019186 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:42:28.018816    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c287727b-944e-4401-8697-896eee8a9e71-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-snsjr\" (UID: \"c287727b-944e-4401-8697-896eee8a9e71\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-snsjr"
Sep 09 02:42:28.019186 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:42:28.018839    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c287727b-944e-4401-8697-896eee8a9e71-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-snsjr\" (UID: \"c287727b-944e-4401-8697-896eee8a9e71\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-snsjr"
Sep 09 02:42:28.019186 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:42:28.018886    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c287727b-944e-4401-8697-896eee8a9e71-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-snsjr\" (UID: \"c287727b-944e-4401-8697-896eee8a9e71\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-snsjr"
Sep 09 02:42:28.019186 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:42:28.019016    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c287727b-944e-4401-8697-896eee8a9e71-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-snsjr\" (UID: \"c287727b-944e-4401-8697-896eee8a9e71\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-snsjr"
Sep 09 02:42:28.019186 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:42:28.019057    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7q5mv\" (UniqueName: \"kubernetes.io/projected/c287727b-944e-4401-8697-896eee8a9e71-kube-api-access-7q5mv\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-snsjr\" (UID: \"c287727b-944e-4401-8697-896eee8a9e71\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-snsjr"
Sep 09 02:42:28.119924 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:42:28.119884    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c287727b-944e-4401-8697-896eee8a9e71-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-snsjr\" (UID: \"c287727b-944e-4401-8697-896eee8a9e71\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-snsjr"
Sep 09 02:42:28.120085 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:42:28.119925    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7q5mv\" (UniqueName: \"kubernetes.io/projected/c287727b-944e-4401-8697-896eee8a9e71-kube-api-access-7q5mv\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-snsjr\" (UID: \"c287727b-944e-4401-8697-896eee8a9e71\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-snsjr"
Sep 09 02:42:28.120085 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:42:28.119961    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c287727b-944e-4401-8697-896eee8a9e71-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-snsjr\" (UID: \"c287727b-944e-4401-8697-896eee8a9e71\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-snsjr"
Sep 09 02:42:28.120085 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:42:28.120016    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c287727b-944e-4401-8697-896eee8a9e71-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-snsjr\" (UID: \"c287727b-944e-4401-8697-896eee8a9e71\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-snsjr"
Sep 09 02:42:28.120085 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:42:28.120031    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c287727b-944e-4401-8697-896eee8a9e71-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-snsjr\" (UID: \"c287727b-944e-4401-8697-896eee8a9e71\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-snsjr"
Sep 09 02:42:28.120085 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:42:28.120078    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c287727b-944e-4401-8697-896eee8a9e71-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-snsjr\" (UID: \"c287727b-944e-4401-8697-896eee8a9e71\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-snsjr"
Sep 09 02:42:28.120373 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:42:28.120347    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c287727b-944e-4401-8697-896eee8a9e71-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-snsjr\" (UID: \"c287727b-944e-4401-8697-896eee8a9e71\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-snsjr"
Sep 09 02:42:28.120450 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:42:28.120379    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c287727b-944e-4401-8697-896eee8a9e71-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-snsjr\" (UID: \"c287727b-944e-4401-8697-896eee8a9e71\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-snsjr"
Sep 09 02:42:28.120450 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:42:28.120439    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c287727b-944e-4401-8697-896eee8a9e71-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-snsjr\" (UID: \"c287727b-944e-4401-8697-896eee8a9e71\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-snsjr"
Sep 09 02:42:28.122227 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:42:28.122209    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c287727b-944e-4401-8697-896eee8a9e71-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-snsjr\" (UID: \"c287727b-944e-4401-8697-896eee8a9e71\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-snsjr"
Sep 09 02:42:28.122419 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:42:28.122392    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c287727b-944e-4401-8697-896eee8a9e71-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-snsjr\" (UID: \"c287727b-944e-4401-8697-896eee8a9e71\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-snsjr"
Sep 09 02:42:28.127779 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:42:28.127758    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7q5mv\" (UniqueName: \"kubernetes.io/projected/c287727b-944e-4401-8697-896eee8a9e71-kube-api-access-7q5mv\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-snsjr\" (UID: \"c287727b-944e-4401-8697-896eee8a9e71\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-snsjr"
Sep 09 02:42:28.257045 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:42:28.256953    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-snsjr"
Sep 09 02:42:28.530700 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:42:28.530678    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-snsjr"]
Sep 09 02:42:28.532650 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:42:28.532627    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc287727b_944e_4401_8697_896eee8a9e71.slice/crio-6f508b6ffba74d6e126206ac4de7459abee5d48a311d7b90ed9f5f1b61fb828b WatchSource:0}: Error finding container 6f508b6ffba74d6e126206ac4de7459abee5d48a311d7b90ed9f5f1b61fb828b: Status 404 returned error can't find the container with id 6f508b6ffba74d6e126206ac4de7459abee5d48a311d7b90ed9f5f1b61fb828b
Sep 09 02:42:28.533923 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:42:28.533907    2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 02:42:28.948841 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:42:28.948799    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-99qjf"]
Sep 09 02:42:29.268079 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:42:29.267990    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-snsjr" event={"ID":"c287727b-944e-4401-8697-896eee8a9e71","Type":"ContainerStarted","Data":"6f508b6ffba74d6e126206ac4de7459abee5d48a311d7b90ed9f5f1b61fb828b"}
Sep 09 02:42:30.273801 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:42:30.273766    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-snsjr" event={"ID":"c287727b-944e-4401-8697-896eee8a9e71","Type":"ContainerStarted","Data":"f265d6ae3902af77e66b183fd2cc6e8cc7905248d389021b311dc49f613566a7"}
Sep 09 02:42:30.292838 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:42:30.292781    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-snsjr" podStartSLOduration=2.334787038 podStartE2EDuration="3.292767588s" totalImagesPullingTime="957.98055ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:42:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:42:30.291587701 +0000 UTC m=+732.545891350" watchObservedRunningTime="2026-09-09 02:42:30.292767588 +0000 UTC m=+732.547071225"
Sep 09 02:42:37.751900 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:42:37.751861    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-99qjf"]
Sep 09 02:42:38.257192 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:42:38.257156    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-snsjr"
Sep 09 02:42:38.257364 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:42:38.257233    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-snsjr"
Sep 09 02:42:38.269933 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:42:38.269904    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-snsjr"
Sep 09 02:42:38.320312 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:42:38.320283    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-snsjr"
Sep 09 02:42:46.749754 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:42:46.749722    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhsd6k"]
Sep 09 02:42:46.753120 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:42:46.753101    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhsd6k"
Sep 09 02:42:46.756254 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:42:46.756234    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Sep 09 02:42:46.766416 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:42:46.766385    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhsd6k"]
Sep 09 02:42:46.878481 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:42:46.878444    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d58ee39c-1a09-4a8b-b02c-6d38a7b84255-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhsd6k\" (UID: \"d58ee39c-1a09-4a8b-b02c-6d38a7b84255\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhsd6k"
Sep 09 02:42:46.878481 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:42:46.878486    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d58ee39c-1a09-4a8b-b02c-6d38a7b84255-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhsd6k\" (UID: \"d58ee39c-1a09-4a8b-b02c-6d38a7b84255\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhsd6k"
Sep 09 02:42:46.878677 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:42:46.878505    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9b2v\" (UniqueName: \"kubernetes.io/projected/d58ee39c-1a09-4a8b-b02c-6d38a7b84255-kube-api-access-m9b2v\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhsd6k\" (UID: \"d58ee39c-1a09-4a8b-b02c-6d38a7b84255\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhsd6k"
Sep 09 02:42:46.878677 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:42:46.878580    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d58ee39c-1a09-4a8b-b02c-6d38a7b84255-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhsd6k\" (UID: \"d58ee39c-1a09-4a8b-b02c-6d38a7b84255\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhsd6k"
Sep 09 02:42:46.878677 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:42:46.878668    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d58ee39c-1a09-4a8b-b02c-6d38a7b84255-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhsd6k\" (UID: \"d58ee39c-1a09-4a8b-b02c-6d38a7b84255\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhsd6k"
Sep 09 02:42:46.878772 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:42:46.878698    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d58ee39c-1a09-4a8b-b02c-6d38a7b84255-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhsd6k\" (UID: \"d58ee39c-1a09-4a8b-b02c-6d38a7b84255\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhsd6k"
Sep 09 02:42:46.979618 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:42:46.979578    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d58ee39c-1a09-4a8b-b02c-6d38a7b84255-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhsd6k\" (UID: \"d58ee39c-1a09-4a8b-b02c-6d38a7b84255\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhsd6k"
Sep 09 02:42:46.979618 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:42:46.979624    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d58ee39c-1a09-4a8b-b02c-6d38a7b84255-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhsd6k\" (UID: \"d58ee39c-1a09-4a8b-b02c-6d38a7b84255\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhsd6k"
Sep 09 02:42:46.979857 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:42:46.979642    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9b2v\" (UniqueName: \"kubernetes.io/projected/d58ee39c-1a09-4a8b-b02c-6d38a7b84255-kube-api-access-m9b2v\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhsd6k\" (UID: \"d58ee39c-1a09-4a8b-b02c-6d38a7b84255\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhsd6k"
Sep 09 02:42:46.979857 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:42:46.979668    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d58ee39c-1a09-4a8b-b02c-6d38a7b84255-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhsd6k\" (UID: \"d58ee39c-1a09-4a8b-b02c-6d38a7b84255\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhsd6k"
Sep 09 02:42:46.979971 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:42:46.979863    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d58ee39c-1a09-4a8b-b02c-6d38a7b84255-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhsd6k\" (UID: \"d58ee39c-1a09-4a8b-b02c-6d38a7b84255\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhsd6k"
Sep 09 02:42:46.979971 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:42:46.979942    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d58ee39c-1a09-4a8b-b02c-6d38a7b84255-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhsd6k\" (UID: \"d58ee39c-1a09-4a8b-b02c-6d38a7b84255\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhsd6k"
Sep 09 02:42:46.980126 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:42:46.980099    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d58ee39c-1a09-4a8b-b02c-6d38a7b84255-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhsd6k\" (UID: \"d58ee39c-1a09-4a8b-b02c-6d38a7b84255\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhsd6k"
Sep 09 02:42:46.980165 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:42:46.980131    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d58ee39c-1a09-4a8b-b02c-6d38a7b84255-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhsd6k\" (UID: \"d58ee39c-1a09-4a8b-b02c-6d38a7b84255\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhsd6k"
Sep 09 02:42:46.980246 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:42:46.980227    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d58ee39c-1a09-4a8b-b02c-6d38a7b84255-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhsd6k\" (UID: \"d58ee39c-1a09-4a8b-b02c-6d38a7b84255\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhsd6k"
Sep 09 02:42:46.982093 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:42:46.982068    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d58ee39c-1a09-4a8b-b02c-6d38a7b84255-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhsd6k\" (UID: \"d58ee39c-1a09-4a8b-b02c-6d38a7b84255\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhsd6k"
Sep 09 02:42:46.982346 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:42:46.982328    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d58ee39c-1a09-4a8b-b02c-6d38a7b84255-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhsd6k\" (UID: \"d58ee39c-1a09-4a8b-b02c-6d38a7b84255\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhsd6k"
Sep 09 02:42:46.989037 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:42:46.989013    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9b2v\" (UniqueName: \"kubernetes.io/projected/d58ee39c-1a09-4a8b-b02c-6d38a7b84255-kube-api-access-m9b2v\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhsd6k\" (UID: \"d58ee39c-1a09-4a8b-b02c-6d38a7b84255\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhsd6k"
Sep 09 02:42:47.062825 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:42:47.062748    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhsd6k"
Sep 09 02:42:47.347379 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:42:47.347348    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhsd6k"]
Sep 09 02:42:47.349610 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:42:47.349587    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd58ee39c_1a09_4a8b_b02c_6d38a7b84255.slice/crio-2c6996cb164a4f75d4ba690e019c7ca551f29edb9043b9f9c662afcdac438722 WatchSource:0}: Error finding container 2c6996cb164a4f75d4ba690e019c7ca551f29edb9043b9f9c662afcdac438722: Status 404 returned error can't find the container with id 2c6996cb164a4f75d4ba690e019c7ca551f29edb9043b9f9c662afcdac438722
Sep 09 02:42:48.354684 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:42:48.354651    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-99qjf"]
Sep 09 02:42:48.355763 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:42:48.355737    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhsd6k" event={"ID":"d58ee39c-1a09-4a8b-b02c-6d38a7b84255","Type":"ContainerStarted","Data":"dfecde38642a67feb7a1829bc7f01a375f607fe215a504d15347b259687d4d94"}
Sep 09 02:42:48.355863 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:42:48.355789    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhsd6k" event={"ID":"d58ee39c-1a09-4a8b-b02c-6d38a7b84255","Type":"ContainerStarted","Data":"2c6996cb164a4f75d4ba690e019c7ca551f29edb9043b9f9c662afcdac438722"}
Sep 09 02:42:48.383324 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:42:48.383277    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhsd6k" podStartSLOduration=2.216857992 podStartE2EDuration="2.383255935s" totalImagesPullingTime="166.397943ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:42:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:42:48.381250732 +0000 UTC m=+750.635554365" watchObservedRunningTime="2026-09-09 02:42:48.383255935 +0000 UTC m=+750.637559564"
Sep 09 02:42:56.249328 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:42:56.249293    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-99qjf"]
Sep 09 02:42:57.063222 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:42:57.063173    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhsd6k"
Sep 09 02:42:57.063431 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:42:57.063252    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhsd6k"
Sep 09 02:42:57.082916 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:42:57.082886    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhsd6k"
Sep 09 02:42:57.408696 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:42:57.408659    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhsd6k"
Sep 09 02:43:16.251797 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:43:16.251758    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-9bxfs"]
Sep 09 02:43:16.255621 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:43:16.255597    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9bxfs"
Sep 09 02:43:16.258692 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:43:16.258670    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 09 02:43:16.266189 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:43:16.266167    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-9bxfs"]
Sep 09 02:43:16.343812 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:43:16.343772    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6aab1e4-77bc-415b-bb29-d2885f6edb0d-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-9bxfs\" (UID: \"c6aab1e4-77bc-415b-bb29-d2885f6edb0d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9bxfs"
Sep 09 02:43:16.344000 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:43:16.343819    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6aab1e4-77bc-415b-bb29-d2885f6edb0d-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-9bxfs\" (UID: \"c6aab1e4-77bc-415b-bb29-d2885f6edb0d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9bxfs"
Sep 09 02:43:16.344000 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:43:16.343841    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6aab1e4-77bc-415b-bb29-d2885f6edb0d-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-9bxfs\" (UID: \"c6aab1e4-77bc-415b-bb29-d2885f6edb0d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9bxfs"
Sep 09 02:43:16.344000 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:43:16.343901    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6aab1e4-77bc-415b-bb29-d2885f6edb0d-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-9bxfs\" (UID: \"c6aab1e4-77bc-415b-bb29-d2885f6edb0d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9bxfs"
Sep 09 02:43:16.344000 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:43:16.343935    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6aab1e4-77bc-415b-bb29-d2885f6edb0d-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-9bxfs\" (UID: \"c6aab1e4-77bc-415b-bb29-d2885f6edb0d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9bxfs"
Sep 09 02:43:16.344000 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:43:16.343956    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjhgb\" (UniqueName: \"kubernetes.io/projected/c6aab1e4-77bc-415b-bb29-d2885f6edb0d-kube-api-access-tjhgb\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-9bxfs\" (UID: \"c6aab1e4-77bc-415b-bb29-d2885f6edb0d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9bxfs"
Sep 09 02:43:16.444826 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:43:16.444788    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6aab1e4-77bc-415b-bb29-d2885f6edb0d-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-9bxfs\" (UID: \"c6aab1e4-77bc-415b-bb29-d2885f6edb0d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9bxfs"
Sep 09 02:43:16.444826 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:43:16.444832    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6aab1e4-77bc-415b-bb29-d2885f6edb0d-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-9bxfs\" (UID: \"c6aab1e4-77bc-415b-bb29-d2885f6edb0d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9bxfs"
Sep 09 02:43:16.445087 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:43:16.444856    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6aab1e4-77bc-415b-bb29-d2885f6edb0d-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-9bxfs\" (UID: \"c6aab1e4-77bc-415b-bb29-d2885f6edb0d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9bxfs"
Sep 09 02:43:16.445087 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:43:16.444890    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6aab1e4-77bc-415b-bb29-d2885f6edb0d-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-9bxfs\" (UID: \"c6aab1e4-77bc-415b-bb29-d2885f6edb0d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9bxfs"
Sep 09 02:43:16.445087 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:43:16.444920    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6aab1e4-77bc-415b-bb29-d2885f6edb0d-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-9bxfs\" (UID: \"c6aab1e4-77bc-415b-bb29-d2885f6edb0d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9bxfs"
Sep 09 02:43:16.445087 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:43:16.444951    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjhgb\" (UniqueName: \"kubernetes.io/projected/c6aab1e4-77bc-415b-bb29-d2885f6edb0d-kube-api-access-tjhgb\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-9bxfs\" (UID: \"c6aab1e4-77bc-415b-bb29-d2885f6edb0d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9bxfs"
Sep 09 02:43:16.445301 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:43:16.445223    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6aab1e4-77bc-415b-bb29-d2885f6edb0d-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-9bxfs\" (UID: \"c6aab1e4-77bc-415b-bb29-d2885f6edb0d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9bxfs"
Sep 09 02:43:16.445373 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:43:16.445353    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6aab1e4-77bc-415b-bb29-d2885f6edb0d-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-9bxfs\" (UID: \"c6aab1e4-77bc-415b-bb29-d2885f6edb0d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9bxfs"
Sep 09 02:43:16.445423 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:43:16.445407    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6aab1e4-77bc-415b-bb29-d2885f6edb0d-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-9bxfs\" (UID: \"c6aab1e4-77bc-415b-bb29-d2885f6edb0d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9bxfs"
Sep 09 02:43:16.447324 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:43:16.447293    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6aab1e4-77bc-415b-bb29-d2885f6edb0d-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-9bxfs\" (UID: \"c6aab1e4-77bc-415b-bb29-d2885f6edb0d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9bxfs"
Sep 09 02:43:16.447473 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:43:16.447453    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6aab1e4-77bc-415b-bb29-d2885f6edb0d-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-9bxfs\" (UID: \"c6aab1e4-77bc-415b-bb29-d2885f6edb0d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9bxfs"
Sep 09 02:43:16.471922 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:43:16.471892    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjhgb\" (UniqueName: \"kubernetes.io/projected/c6aab1e4-77bc-415b-bb29-d2885f6edb0d-kube-api-access-tjhgb\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-9bxfs\" (UID: \"c6aab1e4-77bc-415b-bb29-d2885f6edb0d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9bxfs"
Sep 09 02:43:16.566564 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:43:16.566496    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9bxfs"
Sep 09 02:43:16.838725 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:43:16.838699    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-9bxfs"]
Sep 09 02:43:16.840706 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:43:16.840683    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc6aab1e4_77bc_415b_bb29_d2885f6edb0d.slice/crio-25eb6b2479f4434a628127d003c7728c8ce1fa4f9362a483d54116cafbfcb649 WatchSource:0}: Error finding container 25eb6b2479f4434a628127d003c7728c8ce1fa4f9362a483d54116cafbfcb649: Status 404 returned error can't find the container with id 25eb6b2479f4434a628127d003c7728c8ce1fa4f9362a483d54116cafbfcb649
Sep 09 02:43:17.499042 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:43:17.498935    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9bxfs" event={"ID":"c6aab1e4-77bc-415b-bb29-d2885f6edb0d","Type":"ContainerStarted","Data":"8f0690b275a8b2f015d8cf8247a567449b3f22eee1ac5d7f2e7eb5537942d1f7"}
Sep 09 02:43:17.499042 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:43:17.498993    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9bxfs" event={"ID":"c6aab1e4-77bc-415b-bb29-d2885f6edb0d","Type":"ContainerStarted","Data":"25eb6b2479f4434a628127d003c7728c8ce1fa4f9362a483d54116cafbfcb649"}
Sep 09 02:43:17.517521 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:43:17.517473    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9bxfs" podStartSLOduration=1.328409703 podStartE2EDuration="1.517458153s" totalImagesPullingTime="189.04845ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:43:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:43:17.516022257 +0000 UTC m=+779.770325892" watchObservedRunningTime="2026-09-09 02:43:17.517458153 +0000 UTC m=+779.771761788"
Sep 09 02:43:18.542088 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:43:18.542051    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-99qjf"]
Sep 09 02:43:26.567615 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:43:26.567566    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9bxfs"
Sep 09 02:43:26.568101 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:43:26.567635    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9bxfs"
Sep 09 02:43:26.579927 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:43:26.579900    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9bxfs"
Sep 09 02:43:27.557946 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:43:27.557915    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9bxfs"
Sep 09 02:44:26.123510 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:44:26.123478    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59dbcfbf54-bsb94"]
Sep 09 02:44:26.126770 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:44:26.126751    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59dbcfbf54-bsb94"
Sep 09 02:44:26.130519 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:44:26.130500    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 09 02:44:26.130607 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:44:26.130500    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-nlct4\""
Sep 09 02:44:26.135087 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:44:26.135064    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59dbcfbf54-bsb94"]
Sep 09 02:44:26.257084 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:44:26.257050    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwsrp\" (UniqueName: \"kubernetes.io/projected/30658216-f113-4e7a-b255-233300d755a9-kube-api-access-hwsrp\") pod \"authorino-59dbcfbf54-bsb94\" (UID: \"30658216-f113-4e7a-b255-233300d755a9\") " pod="kuadrant-system/authorino-59dbcfbf54-bsb94"
Sep 09 02:44:26.257239 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:44:26.257092    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/30658216-f113-4e7a-b255-233300d755a9-tls-cert\") pod \"authorino-59dbcfbf54-bsb94\" (UID: \"30658216-f113-4e7a-b255-233300d755a9\") " pod="kuadrant-system/authorino-59dbcfbf54-bsb94"
Sep 09 02:44:26.358067 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:44:26.358034    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwsrp\" (UniqueName: \"kubernetes.io/projected/30658216-f113-4e7a-b255-233300d755a9-kube-api-access-hwsrp\") pod \"authorino-59dbcfbf54-bsb94\" (UID: \"30658216-f113-4e7a-b255-233300d755a9\") " pod="kuadrant-system/authorino-59dbcfbf54-bsb94"
Sep 09 02:44:26.358222 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:44:26.358080    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/30658216-f113-4e7a-b255-233300d755a9-tls-cert\") pod \"authorino-59dbcfbf54-bsb94\" (UID: \"30658216-f113-4e7a-b255-233300d755a9\") " pod="kuadrant-system/authorino-59dbcfbf54-bsb94"
Sep 09 02:44:26.360443 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:44:26.360420    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/30658216-f113-4e7a-b255-233300d755a9-tls-cert\") pod \"authorino-59dbcfbf54-bsb94\" (UID: \"30658216-f113-4e7a-b255-233300d755a9\") " pod="kuadrant-system/authorino-59dbcfbf54-bsb94"
Sep 09 02:44:26.365834 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:44:26.365814    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwsrp\" (UniqueName: \"kubernetes.io/projected/30658216-f113-4e7a-b255-233300d755a9-kube-api-access-hwsrp\") pod \"authorino-59dbcfbf54-bsb94\" (UID: \"30658216-f113-4e7a-b255-233300d755a9\") " pod="kuadrant-system/authorino-59dbcfbf54-bsb94"
Sep 09 02:44:26.437307 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:44:26.437250    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59dbcfbf54-bsb94"
Sep 09 02:44:26.707778 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:44:26.707754    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59dbcfbf54-bsb94"]
Sep 09 02:44:26.709833 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:44:26.709805    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod30658216_f113_4e7a_b255_233300d755a9.slice/crio-08f734cd2ab68fd2aa09b16f73e3aee3c952c5fe742e6e98b25b33b9b77541f9 WatchSource:0}: Error finding container 08f734cd2ab68fd2aa09b16f73e3aee3c952c5fe742e6e98b25b33b9b77541f9: Status 404 returned error can't find the container with id 08f734cd2ab68fd2aa09b16f73e3aee3c952c5fe742e6e98b25b33b9b77541f9
Sep 09 02:44:26.824514 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:44:26.824477    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59dbcfbf54-bsb94" event={"ID":"30658216-f113-4e7a-b255-233300d755a9","Type":"ContainerStarted","Data":"08f734cd2ab68fd2aa09b16f73e3aee3c952c5fe742e6e98b25b33b9b77541f9"}
Sep 09 02:44:44.907435 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:44:44.907399    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59dbcfbf54-bsb94" event={"ID":"30658216-f113-4e7a-b255-233300d755a9","Type":"ContainerStarted","Data":"6320edd138c02ab3bf3c9e9ce59cf799f43e4c6d7b8acbbbda84c0511e707183"}
Sep 09 02:44:44.945444 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:44:44.945338    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59dbcfbf54-bsb94" podStartSLOduration=0.969244786 podStartE2EDuration="18.945322773s" totalImagesPullingTime="17.976077987s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:44:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:44:44.94338575 +0000 UTC m=+867.197689387" watchObservedRunningTime="2026-09-09 02:44:44.945322773 +0000 UTC m=+867.199626409"
Sep 09 02:45:06.848838 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:06.848797    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-99qjf"]
Sep 09 02:45:11.962938 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:11.962846    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-79a2702e-openshift-default-6dc4c95574-g2jlk"]
Sep 09 02:45:11.974695 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:11.974667    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-79a2702e-openshift-default-6dc4c95574-g2jlk"
Sep 09 02:45:11.976931 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:11.976897    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-79a2702e-openshift-default-6dc4c95574-g2jlk"]
Sep 09 02:45:11.980087 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:11.979265    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-79a2702e-openshift-default-dockercfg-4wrds\""
Sep 09 02:45:12.067937 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:12.067883    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe-istio-data\") pod \"e2e-ait-79a2702e-openshift-default-6dc4c95574-g2jlk\" (UID: \"bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe\") " pod="openshift-ingress/e2e-ait-79a2702e-openshift-default-6dc4c95574-g2jlk"
Sep 09 02:45:12.068130 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:12.067967    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe-istio-token\") pod \"e2e-ait-79a2702e-openshift-default-6dc4c95574-g2jlk\" (UID: \"bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe\") " pod="openshift-ingress/e2e-ait-79a2702e-openshift-default-6dc4c95574-g2jlk"
Sep 09 02:45:12.068130 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:12.068070    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe-istiod-ca-cert\") pod \"e2e-ait-79a2702e-openshift-default-6dc4c95574-g2jlk\" (UID: \"bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe\") " pod="openshift-ingress/e2e-ait-79a2702e-openshift-default-6dc4c95574-g2jlk"
Sep 09 02:45:12.068268 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:12.068153    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe-istio-envoy\") pod \"e2e-ait-79a2702e-openshift-default-6dc4c95574-g2jlk\" (UID: \"bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe\") " pod="openshift-ingress/e2e-ait-79a2702e-openshift-default-6dc4c95574-g2jlk"
Sep 09 02:45:12.068268 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:12.068216    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe-credential-socket\") pod \"e2e-ait-79a2702e-openshift-default-6dc4c95574-g2jlk\" (UID: \"bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe\") " pod="openshift-ingress/e2e-ait-79a2702e-openshift-default-6dc4c95574-g2jlk"
Sep 09 02:45:12.068268 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:12.068246    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe-workload-socket\") pod \"e2e-ait-79a2702e-openshift-default-6dc4c95574-g2jlk\" (UID: \"bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe\") " pod="openshift-ingress/e2e-ait-79a2702e-openshift-default-6dc4c95574-g2jlk"
Sep 09 02:45:12.068422 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:12.068325    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe-istio-podinfo\") pod \"e2e-ait-79a2702e-openshift-default-6dc4c95574-g2jlk\" (UID: \"bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe\") " pod="openshift-ingress/e2e-ait-79a2702e-openshift-default-6dc4c95574-g2jlk"
Sep 09 02:45:12.068422 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:12.068386    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe-workload-certs\") pod \"e2e-ait-79a2702e-openshift-default-6dc4c95574-g2jlk\" (UID: \"bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe\") " pod="openshift-ingress/e2e-ait-79a2702e-openshift-default-6dc4c95574-g2jlk"
Sep 09 02:45:12.068422 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:12.068414    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8tg2\" (UniqueName: \"kubernetes.io/projected/bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe-kube-api-access-v8tg2\") pod \"e2e-ait-79a2702e-openshift-default-6dc4c95574-g2jlk\" (UID: \"bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe\") " pod="openshift-ingress/e2e-ait-79a2702e-openshift-default-6dc4c95574-g2jlk"
Sep 09 02:45:12.169568 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:12.169535    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe-istio-token\") pod \"e2e-ait-79a2702e-openshift-default-6dc4c95574-g2jlk\" (UID: \"bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe\") " pod="openshift-ingress/e2e-ait-79a2702e-openshift-default-6dc4c95574-g2jlk"
Sep 09 02:45:12.169743 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:12.169572    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe-istiod-ca-cert\") pod \"e2e-ait-79a2702e-openshift-default-6dc4c95574-g2jlk\" (UID: \"bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe\") " pod="openshift-ingress/e2e-ait-79a2702e-openshift-default-6dc4c95574-g2jlk"
Sep 09 02:45:12.169743 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:12.169615    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe-istio-envoy\") pod \"e2e-ait-79a2702e-openshift-default-6dc4c95574-g2jlk\" (UID: \"bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe\") " pod="openshift-ingress/e2e-ait-79a2702e-openshift-default-6dc4c95574-g2jlk"
Sep 09 02:45:12.169743 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:12.169645    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe-credential-socket\") pod \"e2e-ait-79a2702e-openshift-default-6dc4c95574-g2jlk\" (UID: \"bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe\") " pod="openshift-ingress/e2e-ait-79a2702e-openshift-default-6dc4c95574-g2jlk"
Sep 09 02:45:12.169743 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:12.169673    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe-workload-socket\") pod \"e2e-ait-79a2702e-openshift-default-6dc4c95574-g2jlk\" (UID: \"bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe\") " pod="openshift-ingress/e2e-ait-79a2702e-openshift-default-6dc4c95574-g2jlk"
Sep 09 02:45:12.169743 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:12.169719    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe-istio-podinfo\") pod \"e2e-ait-79a2702e-openshift-default-6dc4c95574-g2jlk\" (UID: \"bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe\") " pod="openshift-ingress/e2e-ait-79a2702e-openshift-default-6dc4c95574-g2jlk"
Sep 09 02:45:12.170056 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:12.169760    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe-workload-certs\") pod \"e2e-ait-79a2702e-openshift-default-6dc4c95574-g2jlk\" (UID: \"bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe\") " pod="openshift-ingress/e2e-ait-79a2702e-openshift-default-6dc4c95574-g2jlk"
Sep 09 02:45:12.170056 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:12.169788    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8tg2\" (UniqueName: \"kubernetes.io/projected/bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe-kube-api-access-v8tg2\") pod \"e2e-ait-79a2702e-openshift-default-6dc4c95574-g2jlk\" (UID: \"bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe\") " pod="openshift-ingress/e2e-ait-79a2702e-openshift-default-6dc4c95574-g2jlk"
Sep 09 02:45:12.170056 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:12.169839    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe-istio-data\") pod \"e2e-ait-79a2702e-openshift-default-6dc4c95574-g2jlk\" (UID: \"bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe\") " pod="openshift-ingress/e2e-ait-79a2702e-openshift-default-6dc4c95574-g2jlk"
Sep 09 02:45:12.170211 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:12.170108    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe-credential-socket\") pod \"e2e-ait-79a2702e-openshift-default-6dc4c95574-g2jlk\" (UID: \"bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe\") " pod="openshift-ingress/e2e-ait-79a2702e-openshift-default-6dc4c95574-g2jlk"
Sep 09 02:45:12.170211 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:12.170150    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe-workload-socket\") pod \"e2e-ait-79a2702e-openshift-default-6dc4c95574-g2jlk\" (UID: \"bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe\") " pod="openshift-ingress/e2e-ait-79a2702e-openshift-default-6dc4c95574-g2jlk"
Sep 09 02:45:12.170316 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:12.170269    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe-istio-data\") pod \"e2e-ait-79a2702e-openshift-default-6dc4c95574-g2jlk\" (UID: \"bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe\") " pod="openshift-ingress/e2e-ait-79a2702e-openshift-default-6dc4c95574-g2jlk"
Sep 09 02:45:12.170316 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:12.170292    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe-workload-certs\") pod \"e2e-ait-79a2702e-openshift-default-6dc4c95574-g2jlk\" (UID: \"bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe\") " pod="openshift-ingress/e2e-ait-79a2702e-openshift-default-6dc4c95574-g2jlk"
Sep 09 02:45:12.170585 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:12.170550    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe-istiod-ca-cert\") pod \"e2e-ait-79a2702e-openshift-default-6dc4c95574-g2jlk\" (UID: \"bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe\") " pod="openshift-ingress/e2e-ait-79a2702e-openshift-default-6dc4c95574-g2jlk"
Sep 09 02:45:12.172186 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:12.172160    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe-istio-envoy\") pod \"e2e-ait-79a2702e-openshift-default-6dc4c95574-g2jlk\" (UID: \"bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe\") " pod="openshift-ingress/e2e-ait-79a2702e-openshift-default-6dc4c95574-g2jlk"
Sep 09 02:45:12.172471 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:12.172450    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe-istio-podinfo\") pod \"e2e-ait-79a2702e-openshift-default-6dc4c95574-g2jlk\" (UID: \"bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe\") " pod="openshift-ingress/e2e-ait-79a2702e-openshift-default-6dc4c95574-g2jlk"
Sep 09 02:45:12.177939 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:12.177912    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe-istio-token\") pod \"e2e-ait-79a2702e-openshift-default-6dc4c95574-g2jlk\" (UID: \"bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe\") " pod="openshift-ingress/e2e-ait-79a2702e-openshift-default-6dc4c95574-g2jlk"
Sep 09 02:45:12.178135 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:12.178119    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8tg2\" (UniqueName: \"kubernetes.io/projected/bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe-kube-api-access-v8tg2\") pod \"e2e-ait-79a2702e-openshift-default-6dc4c95574-g2jlk\" (UID: \"bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe\") " pod="openshift-ingress/e2e-ait-79a2702e-openshift-default-6dc4c95574-g2jlk"
Sep 09 02:45:12.298421 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:12.298341    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-79a2702e-openshift-default-6dc4c95574-g2jlk"
Sep 09 02:45:12.582588 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:12.582562    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-79a2702e-openshift-default-6dc4c95574-g2jlk"]
Sep 09 02:45:12.584777 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:45:12.584752    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd54615b_f6c5_446b_8d7c_9f2ec5a5fcbe.slice/crio-272a9c780b02e7819c1e9fe4b824827290366569452c19b54d00656b9766f6a2 WatchSource:0}: Error finding container 272a9c780b02e7819c1e9fe4b824827290366569452c19b54d00656b9766f6a2: Status 404 returned error can't find the container with id 272a9c780b02e7819c1e9fe4b824827290366569452c19b54d00656b9766f6a2
Sep 09 02:45:12.586822 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:12.586795    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 02:45:12.586879 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:12.586855    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 02:45:12.586914 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:12.586883    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 02:45:13.037218 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:13.037181    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-79a2702e-openshift-default-6dc4c95574-g2jlk" event={"ID":"bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe","Type":"ContainerStarted","Data":"7bd7f190651b692fab4ba95629e85ec8c537dbf4dd40d58d42ab875e6dcc7083"}
Sep 09 02:45:13.037588 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:13.037236    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-79a2702e-openshift-default-6dc4c95574-g2jlk" event={"ID":"bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe","Type":"ContainerStarted","Data":"272a9c780b02e7819c1e9fe4b824827290366569452c19b54d00656b9766f6a2"}
Sep 09 02:45:13.058468 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:13.058427    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-79a2702e-openshift-default-6dc4c95574-g2jlk" podStartSLOduration=2.05841364 podStartE2EDuration="2.05841364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:45:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:45:13.056664657 +0000 UTC m=+895.310968293" watchObservedRunningTime="2026-09-09 02:45:13.05841364 +0000 UTC m=+895.312717273"
Sep 09 02:45:13.299253 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:13.299172    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-79a2702e-openshift-default-6dc4c95574-g2jlk"
Sep 09 02:45:13.305768 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:13.305741    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-79a2702e-openshift-default-6dc4c95574-g2jlk"
Sep 09 02:45:14.043044 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:14.043006    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-79a2702e-openshift-default-6dc4c95574-g2jlk"
Sep 09 02:45:14.043994 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:14.043956    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-79a2702e-openshift-default-6dc4c95574-g2jlk"
Sep 09 02:45:16.474460 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:16.474422    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-99qjf"]
Sep 09 02:45:27.527519 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:27.527482    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-a-80fff954-kserve-5d5dd55697-nrzf6"]
Sep 09 02:45:27.531045 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:27.531023    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-80fff954-kserve-5d5dd55697-nrzf6"
Sep 09 02:45:27.533685 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:27.533663    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-a-80fff954-kserve-self-signed-certs\""
Sep 09 02:45:27.540471 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:27.540441    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-80fff954-kserve-5d5dd55697-nrzf6"]
Sep 09 02:45:27.614071 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:27.614037    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fcf320b2-3158-4fef-8335-a7f311712377-model-cache\") pod \"e2e-conflict-a-80fff954-kserve-5d5dd55697-nrzf6\" (UID: \"fcf320b2-3158-4fef-8335-a7f311712377\") " pod="llm/e2e-conflict-a-80fff954-kserve-5d5dd55697-nrzf6"
Sep 09 02:45:27.614241 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:27.614086    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fcf320b2-3158-4fef-8335-a7f311712377-tls-certs\") pod \"e2e-conflict-a-80fff954-kserve-5d5dd55697-nrzf6\" (UID: \"fcf320b2-3158-4fef-8335-a7f311712377\") " pod="llm/e2e-conflict-a-80fff954-kserve-5d5dd55697-nrzf6"
Sep 09 02:45:27.614241 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:27.614156    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fcf320b2-3158-4fef-8335-a7f311712377-home\") pod \"e2e-conflict-a-80fff954-kserve-5d5dd55697-nrzf6\" (UID: \"fcf320b2-3158-4fef-8335-a7f311712377\") " pod="llm/e2e-conflict-a-80fff954-kserve-5d5dd55697-nrzf6"
Sep 09 02:45:27.614241 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:27.614206    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fcf320b2-3158-4fef-8335-a7f311712377-dshm\") pod \"e2e-conflict-a-80fff954-kserve-5d5dd55697-nrzf6\" (UID: \"fcf320b2-3158-4fef-8335-a7f311712377\") " pod="llm/e2e-conflict-a-80fff954-kserve-5d5dd55697-nrzf6"
Sep 09 02:45:27.614357 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:27.614241    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fcf320b2-3158-4fef-8335-a7f311712377-tmp-dir\") pod \"e2e-conflict-a-80fff954-kserve-5d5dd55697-nrzf6\" (UID: \"fcf320b2-3158-4fef-8335-a7f311712377\") " pod="llm/e2e-conflict-a-80fff954-kserve-5d5dd55697-nrzf6"
Sep 09 02:45:27.614357 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:27.614257    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fhsv\" (UniqueName: \"kubernetes.io/projected/fcf320b2-3158-4fef-8335-a7f311712377-kube-api-access-8fhsv\") pod \"e2e-conflict-a-80fff954-kserve-5d5dd55697-nrzf6\" (UID: \"fcf320b2-3158-4fef-8335-a7f311712377\") " pod="llm/e2e-conflict-a-80fff954-kserve-5d5dd55697-nrzf6"
Sep 09 02:45:27.715708 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:27.715669    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fcf320b2-3158-4fef-8335-a7f311712377-model-cache\") pod \"e2e-conflict-a-80fff954-kserve-5d5dd55697-nrzf6\" (UID: \"fcf320b2-3158-4fef-8335-a7f311712377\") " pod="llm/e2e-conflict-a-80fff954-kserve-5d5dd55697-nrzf6"
Sep 09 02:45:27.715872 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:27.715719    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fcf320b2-3158-4fef-8335-a7f311712377-tls-certs\") pod \"e2e-conflict-a-80fff954-kserve-5d5dd55697-nrzf6\" (UID: \"fcf320b2-3158-4fef-8335-a7f311712377\") " pod="llm/e2e-conflict-a-80fff954-kserve-5d5dd55697-nrzf6"
Sep 09 02:45:27.715872 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:27.715771    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fcf320b2-3158-4fef-8335-a7f311712377-home\") pod \"e2e-conflict-a-80fff954-kserve-5d5dd55697-nrzf6\" (UID: \"fcf320b2-3158-4fef-8335-a7f311712377\") " pod="llm/e2e-conflict-a-80fff954-kserve-5d5dd55697-nrzf6"
Sep 09 02:45:27.715872 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:27.715797    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fcf320b2-3158-4fef-8335-a7f311712377-dshm\") pod \"e2e-conflict-a-80fff954-kserve-5d5dd55697-nrzf6\" (UID: \"fcf320b2-3158-4fef-8335-a7f311712377\") " pod="llm/e2e-conflict-a-80fff954-kserve-5d5dd55697-nrzf6"
Sep 09 02:45:27.716008 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:27.715927    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fcf320b2-3158-4fef-8335-a7f311712377-tmp-dir\") pod \"e2e-conflict-a-80fff954-kserve-5d5dd55697-nrzf6\" (UID: \"fcf320b2-3158-4fef-8335-a7f311712377\") " pod="llm/e2e-conflict-a-80fff954-kserve-5d5dd55697-nrzf6"
Sep 09 02:45:27.716008 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:27.715960    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fhsv\" (UniqueName: \"kubernetes.io/projected/fcf320b2-3158-4fef-8335-a7f311712377-kube-api-access-8fhsv\") pod \"e2e-conflict-a-80fff954-kserve-5d5dd55697-nrzf6\" (UID: \"fcf320b2-3158-4fef-8335-a7f311712377\") " pod="llm/e2e-conflict-a-80fff954-kserve-5d5dd55697-nrzf6"
Sep 09 02:45:27.716213 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:27.716188    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fcf320b2-3158-4fef-8335-a7f311712377-model-cache\") pod \"e2e-conflict-a-80fff954-kserve-5d5dd55697-nrzf6\" (UID: \"fcf320b2-3158-4fef-8335-a7f311712377\") " pod="llm/e2e-conflict-a-80fff954-kserve-5d5dd55697-nrzf6"
Sep 09 02:45:27.716291 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:27.716191    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fcf320b2-3158-4fef-8335-a7f311712377-home\") pod \"e2e-conflict-a-80fff954-kserve-5d5dd55697-nrzf6\" (UID: \"fcf320b2-3158-4fef-8335-a7f311712377\") " pod="llm/e2e-conflict-a-80fff954-kserve-5d5dd55697-nrzf6"
Sep 09 02:45:27.716291 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:27.716278    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fcf320b2-3158-4fef-8335-a7f311712377-tmp-dir\") pod \"e2e-conflict-a-80fff954-kserve-5d5dd55697-nrzf6\" (UID: \"fcf320b2-3158-4fef-8335-a7f311712377\") " pod="llm/e2e-conflict-a-80fff954-kserve-5d5dd55697-nrzf6"
Sep 09 02:45:27.718188 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:27.718165    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fcf320b2-3158-4fef-8335-a7f311712377-dshm\") pod \"e2e-conflict-a-80fff954-kserve-5d5dd55697-nrzf6\" (UID: \"fcf320b2-3158-4fef-8335-a7f311712377\") " pod="llm/e2e-conflict-a-80fff954-kserve-5d5dd55697-nrzf6"
Sep 09 02:45:27.718391 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:27.718374    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fcf320b2-3158-4fef-8335-a7f311712377-tls-certs\") pod \"e2e-conflict-a-80fff954-kserve-5d5dd55697-nrzf6\" (UID: \"fcf320b2-3158-4fef-8335-a7f311712377\") " pod="llm/e2e-conflict-a-80fff954-kserve-5d5dd55697-nrzf6"
Sep 09 02:45:27.723805 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:27.723782    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fhsv\" (UniqueName: \"kubernetes.io/projected/fcf320b2-3158-4fef-8335-a7f311712377-kube-api-access-8fhsv\") pod \"e2e-conflict-a-80fff954-kserve-5d5dd55697-nrzf6\" (UID: \"fcf320b2-3158-4fef-8335-a7f311712377\") " pod="llm/e2e-conflict-a-80fff954-kserve-5d5dd55697-nrzf6"
Sep 09 02:45:27.840640 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:27.840599    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-80fff954-kserve-5d5dd55697-nrzf6"
Sep 09 02:45:28.131652 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:28.131620    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-80fff954-kserve-5d5dd55697-nrzf6"]
Sep 09 02:45:28.133666 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:45:28.133637    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfcf320b2_3158_4fef_8335_a7f311712377.slice/crio-6dc402244a57b36b9027c7c499d1b0a936e1b2aae8e7d47892347f1f23e991ce WatchSource:0}: Error finding container 6dc402244a57b36b9027c7c499d1b0a936e1b2aae8e7d47892347f1f23e991ce: Status 404 returned error can't find the container with id 6dc402244a57b36b9027c7c499d1b0a936e1b2aae8e7d47892347f1f23e991ce
Sep 09 02:45:28.664340 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:28.664301    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-99qjf"]
Sep 09 02:45:29.117682 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:29.117646    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-80fff954-kserve-5d5dd55697-nrzf6" event={"ID":"fcf320b2-3158-4fef-8335-a7f311712377","Type":"ContainerStarted","Data":"e6816cde2d273a26e55398c3bcc3cc4a4dd58e1b173c612f67617a00b0208b5f"}
Sep 09 02:45:29.117878 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:29.117689    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-80fff954-kserve-5d5dd55697-nrzf6" event={"ID":"fcf320b2-3158-4fef-8335-a7f311712377","Type":"ContainerStarted","Data":"6dc402244a57b36b9027c7c499d1b0a936e1b2aae8e7d47892347f1f23e991ce"}
Sep 09 02:45:29.136700 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:29.136656    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-a-80fff954-kserve-5d5dd55697-nrzf6" podStartSLOduration=2.136639221 podStartE2EDuration="2.136639221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:45:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:45:29.135195773 +0000 UTC m=+911.389499410" watchObservedRunningTime="2026-09-09 02:45:29.136639221 +0000 UTC m=+911.390942856"
Sep 09 02:45:33.136769 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:33.136737    2570 generic.go:356] "Generic (PLEG): container finished" podID="bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe" containerID="7bd7f190651b692fab4ba95629e85ec8c537dbf4dd40d58d42ab875e6dcc7083" exitCode=0
Sep 09 02:45:33.137209 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:33.136802    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-79a2702e-openshift-default-6dc4c95574-g2jlk" event={"ID":"bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe","Type":"ContainerDied","Data":"7bd7f190651b692fab4ba95629e85ec8c537dbf4dd40d58d42ab875e6dcc7083"}
Sep 09 02:45:33.137253 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:33.137218    2570 scope.go:117] "RemoveContainer" containerID="7bd7f190651b692fab4ba95629e85ec8c537dbf4dd40d58d42ab875e6dcc7083"
Sep 09 02:45:33.138281 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:33.138247    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 02:45:33.138413 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:33.138329    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 02:45:33.138413 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:33.138381    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 02:45:34.146434 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:34.146384    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-79a2702e-openshift-default-6dc4c95574-g2jlk" event={"ID":"bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe","Type":"ContainerStarted","Data":"86f7228cfe198a21d5302e7800b83518717c97f9a8932ae8094904550b0ffc08"}
Sep 09 02:45:34.299218 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:34.299182    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-79a2702e-openshift-default-6dc4c95574-g2jlk"
Sep 09 02:45:34.299399 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:34.299360    2570 patch_prober.go:28] interesting pod/e2e-ait-79a2702e-openshift-default-6dc4c95574-g2jlk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Sep 09 02:45:34.299475 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:34.299428    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-79a2702e-openshift-default-6dc4c95574-g2jlk" podUID="bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Sep 09 02:45:35.299149 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:35.299113    2570 patch_prober.go:28] interesting pod/e2e-ait-79a2702e-openshift-default-6dc4c95574-g2jlk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Sep 09 02:45:35.299521 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:35.299167    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-79a2702e-openshift-default-6dc4c95574-g2jlk" podUID="bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Sep 09 02:45:36.236063 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:36.236031    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-6b11a2-openshift-default-5cfbd4b94-fql7q"]
Sep 09 02:45:36.238583 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:36.238562    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6b11a2-openshift-default-5cfbd4b94-fql7q"
Sep 09 02:45:36.256501 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:36.256457    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-6b11a2-openshift-default-dockercfg-dtsdj\""
Sep 09 02:45:36.265207 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:36.265140    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-6b11a2-openshift-default-5cfbd4b94-fql7q"]
Sep 09 02:45:36.298863 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:36.298834    2570 patch_prober.go:28] interesting pod/e2e-ait-79a2702e-openshift-default-6dc4c95574-g2jlk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Sep 09 02:45:36.298966 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:36.298877    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-79a2702e-openshift-default-6dc4c95574-g2jlk" podUID="bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Sep 09 02:45:36.400401 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:36.400366    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hzlc\" (UniqueName: \"kubernetes.io/projected/2a7cadb9-da51-457d-8c72-007236378887-kube-api-access-5hzlc\") pod \"e2e-inf-b-6b11a2-openshift-default-5cfbd4b94-fql7q\" (UID: \"2a7cadb9-da51-457d-8c72-007236378887\") " pod="openshift-ingress/e2e-inf-b-6b11a2-openshift-default-5cfbd4b94-fql7q"
Sep 09 02:45:36.400762 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:36.400407    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a7cadb9-da51-457d-8c72-007236378887-istiod-ca-cert\") pod \"e2e-inf-b-6b11a2-openshift-default-5cfbd4b94-fql7q\" (UID: \"2a7cadb9-da51-457d-8c72-007236378887\") " pod="openshift-ingress/e2e-inf-b-6b11a2-openshift-default-5cfbd4b94-fql7q"
Sep 09 02:45:36.400762 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:36.400431    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a7cadb9-da51-457d-8c72-007236378887-workload-certs\") pod \"e2e-inf-b-6b11a2-openshift-default-5cfbd4b94-fql7q\" (UID: \"2a7cadb9-da51-457d-8c72-007236378887\") " pod="openshift-ingress/e2e-inf-b-6b11a2-openshift-default-5cfbd4b94-fql7q"
Sep 09 02:45:36.400762 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:36.400490    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a7cadb9-da51-457d-8c72-007236378887-istio-token\") pod \"e2e-inf-b-6b11a2-openshift-default-5cfbd4b94-fql7q\" (UID: \"2a7cadb9-da51-457d-8c72-007236378887\") " pod="openshift-ingress/e2e-inf-b-6b11a2-openshift-default-5cfbd4b94-fql7q"
Sep 09 02:45:36.400762 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:36.400508    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a7cadb9-da51-457d-8c72-007236378887-credential-socket\") pod \"e2e-inf-b-6b11a2-openshift-default-5cfbd4b94-fql7q\" (UID: \"2a7cadb9-da51-457d-8c72-007236378887\") " pod="openshift-ingress/e2e-inf-b-6b11a2-openshift-default-5cfbd4b94-fql7q"
Sep 09 02:45:36.400762 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:36.400561    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a7cadb9-da51-457d-8c72-007236378887-istio-data\") pod \"e2e-inf-b-6b11a2-openshift-default-5cfbd4b94-fql7q\" (UID: \"2a7cadb9-da51-457d-8c72-007236378887\") " pod="openshift-ingress/e2e-inf-b-6b11a2-openshift-default-5cfbd4b94-fql7q"
Sep 09 02:45:36.400762 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:36.400602    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a7cadb9-da51-457d-8c72-007236378887-istio-podinfo\") pod \"e2e-inf-b-6b11a2-openshift-default-5cfbd4b94-fql7q\" (UID: \"2a7cadb9-da51-457d-8c72-007236378887\") " pod="openshift-ingress/e2e-inf-b-6b11a2-openshift-default-5cfbd4b94-fql7q"
Sep 09 02:45:36.400762 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:36.400650    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a7cadb9-da51-457d-8c72-007236378887-workload-socket\") pod \"e2e-inf-b-6b11a2-openshift-default-5cfbd4b94-fql7q\" (UID: \"2a7cadb9-da51-457d-8c72-007236378887\") " pod="openshift-ingress/e2e-inf-b-6b11a2-openshift-default-5cfbd4b94-fql7q"
Sep 09 02:45:36.400762 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:36.400667    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a7cadb9-da51-457d-8c72-007236378887-istio-envoy\") pod \"e2e-inf-b-6b11a2-openshift-default-5cfbd4b94-fql7q\" (UID: \"2a7cadb9-da51-457d-8c72-007236378887\") " pod="openshift-ingress/e2e-inf-b-6b11a2-openshift-default-5cfbd4b94-fql7q"
Sep 09 02:45:36.501786 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:36.501694    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a7cadb9-da51-457d-8c72-007236378887-credential-socket\") pod \"e2e-inf-b-6b11a2-openshift-default-5cfbd4b94-fql7q\" (UID: \"2a7cadb9-da51-457d-8c72-007236378887\") " pod="openshift-ingress/e2e-inf-b-6b11a2-openshift-default-5cfbd4b94-fql7q"
Sep 09 02:45:36.501786 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:36.501756    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a7cadb9-da51-457d-8c72-007236378887-istio-data\") pod \"e2e-inf-b-6b11a2-openshift-default-5cfbd4b94-fql7q\" (UID: \"2a7cadb9-da51-457d-8c72-007236378887\") " pod="openshift-ingress/e2e-inf-b-6b11a2-openshift-default-5cfbd4b94-fql7q"
Sep 09 02:45:36.502060 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:36.501798    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a7cadb9-da51-457d-8c72-007236378887-istio-podinfo\") pod \"e2e-inf-b-6b11a2-openshift-default-5cfbd4b94-fql7q\" (UID: \"2a7cadb9-da51-457d-8c72-007236378887\") " pod="openshift-ingress/e2e-inf-b-6b11a2-openshift-default-5cfbd4b94-fql7q"
Sep 09 02:45:36.502060 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:36.501862    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a7cadb9-da51-457d-8c72-007236378887-workload-socket\") pod \"e2e-inf-b-6b11a2-openshift-default-5cfbd4b94-fql7q\" (UID: \"2a7cadb9-da51-457d-8c72-007236378887\") " pod="openshift-ingress/e2e-inf-b-6b11a2-openshift-default-5cfbd4b94-fql7q"
Sep 09 02:45:36.502060 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:36.501883    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a7cadb9-da51-457d-8c72-007236378887-istio-envoy\") pod \"e2e-inf-b-6b11a2-openshift-default-5cfbd4b94-fql7q\" (UID: \"2a7cadb9-da51-457d-8c72-007236378887\") " pod="openshift-ingress/e2e-inf-b-6b11a2-openshift-default-5cfbd4b94-fql7q"
Sep 09 02:45:36.502060 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:36.501929    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hzlc\" (UniqueName: \"kubernetes.io/projected/2a7cadb9-da51-457d-8c72-007236378887-kube-api-access-5hzlc\") pod \"e2e-inf-b-6b11a2-openshift-default-5cfbd4b94-fql7q\" (UID: \"2a7cadb9-da51-457d-8c72-007236378887\") " pod="openshift-ingress/e2e-inf-b-6b11a2-openshift-default-5cfbd4b94-fql7q"
Sep 09 02:45:36.502060 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:36.501968    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a7cadb9-da51-457d-8c72-007236378887-istiod-ca-cert\") pod \"e2e-inf-b-6b11a2-openshift-default-5cfbd4b94-fql7q\" (UID: \"2a7cadb9-da51-457d-8c72-007236378887\") " pod="openshift-ingress/e2e-inf-b-6b11a2-openshift-default-5cfbd4b94-fql7q"
Sep 09 02:45:36.502060 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:36.502036    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a7cadb9-da51-457d-8c72-007236378887-workload-certs\") pod \"e2e-inf-b-6b11a2-openshift-default-5cfbd4b94-fql7q\" (UID: \"2a7cadb9-da51-457d-8c72-007236378887\") " pod="openshift-ingress/e2e-inf-b-6b11a2-openshift-default-5cfbd4b94-fql7q"
Sep 09 02:45:36.502383 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:36.502076    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a7cadb9-da51-457d-8c72-007236378887-istio-token\") pod \"e2e-inf-b-6b11a2-openshift-default-5cfbd4b94-fql7q\" (UID: \"2a7cadb9-da51-457d-8c72-007236378887\") " pod="openshift-ingress/e2e-inf-b-6b11a2-openshift-default-5cfbd4b94-fql7q"
Sep 09 02:45:36.502383 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:36.502179    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a7cadb9-da51-457d-8c72-007236378887-credential-socket\") pod \"e2e-inf-b-6b11a2-openshift-default-5cfbd4b94-fql7q\" (UID: \"2a7cadb9-da51-457d-8c72-007236378887\") " pod="openshift-ingress/e2e-inf-b-6b11a2-openshift-default-5cfbd4b94-fql7q"
Sep 09 02:45:36.502383 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:36.502242    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a7cadb9-da51-457d-8c72-007236378887-istio-data\") pod \"e2e-inf-b-6b11a2-openshift-default-5cfbd4b94-fql7q\" (UID: \"2a7cadb9-da51-457d-8c72-007236378887\") " pod="openshift-ingress/e2e-inf-b-6b11a2-openshift-default-5cfbd4b94-fql7q"
Sep 09 02:45:36.502383 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:36.502263    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a7cadb9-da51-457d-8c72-007236378887-workload-socket\") pod \"e2e-inf-b-6b11a2-openshift-default-5cfbd4b94-fql7q\" (UID: \"2a7cadb9-da51-457d-8c72-007236378887\") " pod="openshift-ingress/e2e-inf-b-6b11a2-openshift-default-5cfbd4b94-fql7q"
Sep 09 02:45:36.502610 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:36.502556    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a7cadb9-da51-457d-8c72-007236378887-workload-certs\") pod \"e2e-inf-b-6b11a2-openshift-default-5cfbd4b94-fql7q\" (UID: \"2a7cadb9-da51-457d-8c72-007236378887\") " pod="openshift-ingress/e2e-inf-b-6b11a2-openshift-default-5cfbd4b94-fql7q"
Sep 09 02:45:36.502785 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:36.502764    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a7cadb9-da51-457d-8c72-007236378887-istiod-ca-cert\") pod \"e2e-inf-b-6b11a2-openshift-default-5cfbd4b94-fql7q\" (UID: \"2a7cadb9-da51-457d-8c72-007236378887\") " pod="openshift-ingress/e2e-inf-b-6b11a2-openshift-default-5cfbd4b94-fql7q"
Sep 09 02:45:36.504231 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:36.504211    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a7cadb9-da51-457d-8c72-007236378887-istio-envoy\") pod \"e2e-inf-b-6b11a2-openshift-default-5cfbd4b94-fql7q\" (UID: \"2a7cadb9-da51-457d-8c72-007236378887\") " pod="openshift-ingress/e2e-inf-b-6b11a2-openshift-default-5cfbd4b94-fql7q"
Sep 09 02:45:36.504545 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:36.504525    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a7cadb9-da51-457d-8c72-007236378887-istio-podinfo\") pod \"e2e-inf-b-6b11a2-openshift-default-5cfbd4b94-fql7q\" (UID: \"2a7cadb9-da51-457d-8c72-007236378887\") " pod="openshift-ingress/e2e-inf-b-6b11a2-openshift-default-5cfbd4b94-fql7q"
Sep 09 02:45:36.510139 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:36.510109    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hzlc\" (UniqueName: \"kubernetes.io/projected/2a7cadb9-da51-457d-8c72-007236378887-kube-api-access-5hzlc\") pod \"e2e-inf-b-6b11a2-openshift-default-5cfbd4b94-fql7q\" (UID: \"2a7cadb9-da51-457d-8c72-007236378887\") " pod="openshift-ingress/e2e-inf-b-6b11a2-openshift-default-5cfbd4b94-fql7q"
Sep 09 02:45:36.510693 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:36.510671    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a7cadb9-da51-457d-8c72-007236378887-istio-token\") pod \"e2e-inf-b-6b11a2-openshift-default-5cfbd4b94-fql7q\" (UID: \"2a7cadb9-da51-457d-8c72-007236378887\") " pod="openshift-ingress/e2e-inf-b-6b11a2-openshift-default-5cfbd4b94-fql7q"
Sep 09 02:45:36.574575 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:36.574541    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6b11a2-openshift-default-5cfbd4b94-fql7q"
Sep 09 02:45:36.853148 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:36.853065    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-6b11a2-openshift-default-5cfbd4b94-fql7q"]
Sep 09 02:45:36.867838 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:45:36.867806    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a7cadb9_da51_457d_8c72_007236378887.slice/crio-6c2b0754b95df1935288f568067eeb9fda0b38ef433569d961eba66e307fc040 WatchSource:0}: Error finding container 6c2b0754b95df1935288f568067eeb9fda0b38ef433569d961eba66e307fc040: Status 404 returned error can't find the container with id 6c2b0754b95df1935288f568067eeb9fda0b38ef433569d961eba66e307fc040
Sep 09 02:45:36.870954 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:36.870922    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 02:45:36.871084 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:36.871013    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 02:45:36.871084 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:36.871044    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 02:45:37.169109 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:37.168096    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6b11a2-openshift-default-5cfbd4b94-fql7q" event={"ID":"2a7cadb9-da51-457d-8c72-007236378887","Type":"ContainerStarted","Data":"a87e064bd39e86b069a49c445832c8eccd611de6b23454ca5c136842685b86c4"}
Sep 09 02:45:37.169109 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:37.168161    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6b11a2-openshift-default-5cfbd4b94-fql7q" event={"ID":"2a7cadb9-da51-457d-8c72-007236378887","Type":"ContainerStarted","Data":"6c2b0754b95df1935288f568067eeb9fda0b38ef433569d961eba66e307fc040"}
Sep 09 02:45:37.213699 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:37.213634    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-6b11a2-openshift-default-5cfbd4b94-fql7q" podStartSLOduration=1.213613722 podStartE2EDuration="1.213613722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:45:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:45:37.212009941 +0000 UTC m=+919.466313577" watchObservedRunningTime="2026-09-09 02:45:37.213613722 +0000 UTC m=+919.467917361"
Sep 09 02:45:37.299196 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:37.299148    2570 patch_prober.go:28] interesting pod/e2e-ait-79a2702e-openshift-default-6dc4c95574-g2jlk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Sep 09 02:45:37.299385 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:37.299212    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-79a2702e-openshift-default-6dc4c95574-g2jlk" podUID="bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Sep 09 02:45:37.575509 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:37.575419    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-6b11a2-openshift-default-5cfbd4b94-fql7q"
Sep 09 02:45:37.577680 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:37.577552    2570 patch_prober.go:28] interesting pod/e2e-inf-b-6b11a2-openshift-default-5cfbd4b94-fql7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Sep 09 02:45:37.577680 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:37.577601    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6b11a2-openshift-default-5cfbd4b94-fql7q" podUID="2a7cadb9-da51-457d-8c72-007236378887" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Sep 09 02:45:37.840937 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:37.840905    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-a-80fff954-kserve-5d5dd55697-nrzf6"
Sep 09 02:45:37.841188 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:37.841170    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-a-80fff954-kserve-5d5dd55697-nrzf6"
Sep 09 02:45:37.875383 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:37.875355    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-a-80fff954-kserve-5d5dd55697-nrzf6"
Sep 09 02:45:38.185250 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:38.185168    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-a-80fff954-kserve-5d5dd55697-nrzf6"
Sep 09 02:45:38.299106 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:38.299073    2570 patch_prober.go:28] interesting pod/e2e-ait-79a2702e-openshift-default-6dc4c95574-g2jlk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Sep 09 02:45:38.299262 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:38.299145    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-79a2702e-openshift-default-6dc4c95574-g2jlk" podUID="bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Sep 09 02:45:38.581213 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:38.581116    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-6b11a2-openshift-default-5cfbd4b94-fql7q"
Sep 09 02:45:39.179624 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:39.179587    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-6b11a2-openshift-default-5cfbd4b94-fql7q"
Sep 09 02:45:39.180418 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:39.180400    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-6b11a2-openshift-default-5cfbd4b94-fql7q"
Sep 09 02:45:39.299476 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:39.299440    2570 patch_prober.go:28] interesting pod/e2e-ait-79a2702e-openshift-default-6dc4c95574-g2jlk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Sep 09 02:45:39.299629 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:39.299519    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-79a2702e-openshift-default-6dc4c95574-g2jlk" podUID="bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Sep 09 02:45:39.950134 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:39.950107    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-79a2702e-f8t2h"]
Sep 09 02:45:39.952306 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:39.952291    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-79a2702e-f8t2h"
Sep 09 02:45:39.955113 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:39.955089    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vbr52\""
Sep 09 02:45:39.955437 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:39.955420    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 02:45:39.964922 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:39.964897    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-79a2702e-f8t2h"]
Sep 09 02:45:40.040965 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:40.040926    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6a91f20-b0a2-476a-b3e1-48cd91071d8e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-79a2702e-f8t2h\" (UID: \"c6a91f20-b0a2-476a-b3e1-48cd91071d8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-79a2702e-f8t2h"
Sep 09 02:45:40.142379 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:40.142339    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6a91f20-b0a2-476a-b3e1-48cd91071d8e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-79a2702e-f8t2h\" (UID: \"c6a91f20-b0a2-476a-b3e1-48cd91071d8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-79a2702e-f8t2h"
Sep 09 02:45:40.142989 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:40.142953    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6a91f20-b0a2-476a-b3e1-48cd91071d8e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-79a2702e-f8t2h\" (UID: \"c6a91f20-b0a2-476a-b3e1-48cd91071d8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-79a2702e-f8t2h"
Sep 09 02:45:40.263171 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:40.263080    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-79a2702e-f8t2h"
Sep 09 02:45:40.304836 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:40.304804    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-79a2702e-openshift-default-6dc4c95574-g2jlk"
Sep 09 02:45:40.305222 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:40.305201    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-79a2702e-openshift-default-6dc4c95574-g2jlk"
Sep 09 02:45:40.305843 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:40.305825    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-79a2702e-openshift-default-6dc4c95574-g2jlk"
Sep 09 02:45:40.534841 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:40.534819    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-79a2702e-f8t2h"]
Sep 09 02:45:40.537160 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:45:40.537123    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6a91f20_b0a2_476a_b3e1_48cd91071d8e.slice/crio-b04ca9024a443d50c33defe107a26dd28159646711341b824356c29e57c57b62 WatchSource:0}: Error finding container b04ca9024a443d50c33defe107a26dd28159646711341b824356c29e57c57b62: Status 404 returned error can't find the container with id b04ca9024a443d50c33defe107a26dd28159646711341b824356c29e57c57b62
Sep 09 02:45:41.189810 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:41.189773    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-79a2702e-f8t2h" event={"ID":"c6a91f20-b0a2-476a-b3e1-48cd91071d8e","Type":"ContainerStarted","Data":"b04ca9024a443d50c33defe107a26dd28159646711341b824356c29e57c57b62"}
Sep 09 02:45:43.202023 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:43.201989    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-79a2702e-f8t2h" event={"ID":"c6a91f20-b0a2-476a-b3e1-48cd91071d8e","Type":"ContainerStarted","Data":"1cc27679f17e7002a33fee9802c7533bd40b3e96a4f83900b646b8d4b9232149"}
Sep 09 02:45:43.218575 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:43.218527    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-79a2702e-f8t2h" podStartSLOduration=2.087909372 podStartE2EDuration="4.218511875s" totalImagesPullingTime="2.130602503s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:45:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:45:43.21694597 +0000 UTC m=+925.471249608" watchObservedRunningTime="2026-09-09 02:45:43.218511875 +0000 UTC m=+925.472815511"
Sep 09 02:45:44.207846 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:44.207813    2570 generic.go:356] "Generic (PLEG): container finished" podID="c6a91f20-b0a2-476a-b3e1-48cd91071d8e" containerID="1cc27679f17e7002a33fee9802c7533bd40b3e96a4f83900b646b8d4b9232149" exitCode=0
Sep 09 02:45:44.208281 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:44.207900    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-79a2702e-f8t2h" event={"ID":"c6a91f20-b0a2-476a-b3e1-48cd91071d8e","Type":"ContainerDied","Data":"1cc27679f17e7002a33fee9802c7533bd40b3e96a4f83900b646b8d4b9232149"}
Sep 09 02:45:45.050054 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:45.050018    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-99qjf"]
Sep 09 02:45:45.351298 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:45.351274    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-79a2702e-f8t2h"
Sep 09 02:45:45.391766 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:45.391738    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6a91f20-b0a2-476a-b3e1-48cd91071d8e-maas-api-service-ca\") pod \"c6a91f20-b0a2-476a-b3e1-48cd91071d8e\" (UID: \"c6a91f20-b0a2-476a-b3e1-48cd91071d8e\") "
Sep 09 02:45:45.392082 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:45.392060    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6a91f20-b0a2-476a-b3e1-48cd91071d8e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c6a91f20-b0a2-476a-b3e1-48cd91071d8e" (UID: "c6a91f20-b0a2-476a-b3e1-48cd91071d8e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 02:45:45.493360 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:45.493327    2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6a91f20-b0a2-476a-b3e1-48cd91071d8e-maas-api-service-ca\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:45:46.221914 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:46.221885    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-79a2702e-f8t2h"
Sep 09 02:45:46.223106 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:46.223078    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-79a2702e-f8t2h" event={"ID":"c6a91f20-b0a2-476a-b3e1-48cd91071d8e","Type":"ContainerDied","Data":"b04ca9024a443d50c33defe107a26dd28159646711341b824356c29e57c57b62"}
Sep 09 02:45:46.223192 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:46.223133    2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b04ca9024a443d50c33defe107a26dd28159646711341b824356c29e57c57b62"
Sep 09 02:45:55.221150 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:55.221102    2570 patch_prober.go:28] interesting pod/e2e-inf-b-6b11a2-openshift-default-5cfbd4b94-fql7q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 09 02:45:55.221622 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:55.221165    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-6b11a2-openshift-default-5cfbd4b94-fql7q" podUID="2a7cadb9-da51-457d-8c72-007236378887" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 09 02:45:56.654793 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:56.654767    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-79a2702e-f8t2h"]
Sep 09 02:45:56.657158 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:56.657137    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-79a2702e-f8t2h"]
Sep 09 02:45:56.782119 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:56.782049    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-79a2702e-openshift-default-6dc4c95574-g2jlk"]
Sep 09 02:45:56.782362 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:56.782306    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-79a2702e-openshift-default-6dc4c95574-g2jlk" podUID="bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe" containerName="istio-proxy" containerID="cri-o://86f7228cfe198a21d5302e7800b83518717c97f9a8932ae8094904550b0ffc08" gracePeriod=30
Sep 09 02:45:57.281234 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:57.281201    2570 generic.go:356] "Generic (PLEG): container finished" podID="2a7cadb9-da51-457d-8c72-007236378887" containerID="a87e064bd39e86b069a49c445832c8eccd611de6b23454ca5c136842685b86c4" exitCode=0
Sep 09 02:45:57.281393 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:57.281277    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6b11a2-openshift-default-5cfbd4b94-fql7q" event={"ID":"2a7cadb9-da51-457d-8c72-007236378887","Type":"ContainerDied","Data":"a87e064bd39e86b069a49c445832c8eccd611de6b23454ca5c136842685b86c4"}
Sep 09 02:45:57.281660 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:57.281646    2570 scope.go:117] "RemoveContainer" containerID="a87e064bd39e86b069a49c445832c8eccd611de6b23454ca5c136842685b86c4"
Sep 09 02:45:57.282750 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:57.282714    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 02:45:57.282862 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:57.282787    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 02:45:57.282862 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:57.282821    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 02:45:57.712048 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:57.712017    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-b0df6c69-gw-openshift-default-86797cc876-lklw2"]
Sep 09 02:45:57.712509 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:57.712434    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6a91f20-b0a2-476a-b3e1-48cd91071d8e" containerName="revoke-keys"
Sep 09 02:45:57.712509 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:57.712446    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6a91f20-b0a2-476a-b3e1-48cd91071d8e" containerName="revoke-keys"
Sep 09 02:45:57.712618 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:57.712513    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6a91f20-b0a2-476a-b3e1-48cd91071d8e" containerName="revoke-keys"
Sep 09 02:45:57.718482 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:57.718460    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-b0df6c69-gw-openshift-default-86797cc876-lklw2"
Sep 09 02:45:57.722128 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:57.722100    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-b0df6c69-gw-openshift-default-dockercfg-f7rll\""
Sep 09 02:45:57.732912 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:57.729491    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-b0df6c69-gw-openshift-default-86797cc876-lklw2"]
Sep 09 02:45:57.833782 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:57.833747    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znstk\" (UniqueName: \"kubernetes.io/projected/f0d61ced-695f-49b3-a86b-732843026f8b-kube-api-access-znstk\") pod \"e2e-migrate-b0df6c69-gw-openshift-default-86797cc876-lklw2\" (UID: \"f0d61ced-695f-49b3-a86b-732843026f8b\") " pod="openshift-ingress/e2e-migrate-b0df6c69-gw-openshift-default-86797cc876-lklw2"
Sep 09 02:45:57.834011 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:57.833798    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0d61ced-695f-49b3-a86b-732843026f8b-istio-data\") pod \"e2e-migrate-b0df6c69-gw-openshift-default-86797cc876-lklw2\" (UID: \"f0d61ced-695f-49b3-a86b-732843026f8b\") " pod="openshift-ingress/e2e-migrate-b0df6c69-gw-openshift-default-86797cc876-lklw2"
Sep 09 02:45:57.834011 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:57.833879    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0d61ced-695f-49b3-a86b-732843026f8b-istio-podinfo\") pod \"e2e-migrate-b0df6c69-gw-openshift-default-86797cc876-lklw2\" (UID: \"f0d61ced-695f-49b3-a86b-732843026f8b\") " pod="openshift-ingress/e2e-migrate-b0df6c69-gw-openshift-default-86797cc876-lklw2"
Sep 09 02:45:57.834011 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:57.833918    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0d61ced-695f-49b3-a86b-732843026f8b-credential-socket\") pod \"e2e-migrate-b0df6c69-gw-openshift-default-86797cc876-lklw2\" (UID: \"f0d61ced-695f-49b3-a86b-732843026f8b\") " pod="openshift-ingress/e2e-migrate-b0df6c69-gw-openshift-default-86797cc876-lklw2"
Sep 09 02:45:57.834011 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:57.833947    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0d61ced-695f-49b3-a86b-732843026f8b-workload-socket\") pod \"e2e-migrate-b0df6c69-gw-openshift-default-86797cc876-lklw2\" (UID: \"f0d61ced-695f-49b3-a86b-732843026f8b\") " pod="openshift-ingress/e2e-migrate-b0df6c69-gw-openshift-default-86797cc876-lklw2"
Sep 09 02:45:57.834276 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:57.834036    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0d61ced-695f-49b3-a86b-732843026f8b-istio-token\") pod \"e2e-migrate-b0df6c69-gw-openshift-default-86797cc876-lklw2\" (UID: \"f0d61ced-695f-49b3-a86b-732843026f8b\") " pod="openshift-ingress/e2e-migrate-b0df6c69-gw-openshift-default-86797cc876-lklw2"
Sep 09 02:45:57.834276 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:57.834090    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0d61ced-695f-49b3-a86b-732843026f8b-workload-certs\") pod \"e2e-migrate-b0df6c69-gw-openshift-default-86797cc876-lklw2\" (UID: \"f0d61ced-695f-49b3-a86b-732843026f8b\") " pod="openshift-ingress/e2e-migrate-b0df6c69-gw-openshift-default-86797cc876-lklw2"
Sep 09 02:45:57.834276 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:57.834126    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0d61ced-695f-49b3-a86b-732843026f8b-istiod-ca-cert\") pod \"e2e-migrate-b0df6c69-gw-openshift-default-86797cc876-lklw2\" (UID: \"f0d61ced-695f-49b3-a86b-732843026f8b\") " pod="openshift-ingress/e2e-migrate-b0df6c69-gw-openshift-default-86797cc876-lklw2"
Sep 09 02:45:57.834276 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:57.834158    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0d61ced-695f-49b3-a86b-732843026f8b-istio-envoy\") pod \"e2e-migrate-b0df6c69-gw-openshift-default-86797cc876-lklw2\" (UID: \"f0d61ced-695f-49b3-a86b-732843026f8b\") " pod="openshift-ingress/e2e-migrate-b0df6c69-gw-openshift-default-86797cc876-lklw2"
Sep 09 02:45:57.934816 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:57.934780    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0d61ced-695f-49b3-a86b-732843026f8b-istiod-ca-cert\") pod \"e2e-migrate-b0df6c69-gw-openshift-default-86797cc876-lklw2\" (UID: \"f0d61ced-695f-49b3-a86b-732843026f8b\") " pod="openshift-ingress/e2e-migrate-b0df6c69-gw-openshift-default-86797cc876-lklw2"
Sep 09 02:45:57.934816 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:57.934820    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0d61ced-695f-49b3-a86b-732843026f8b-istio-envoy\") pod \"e2e-migrate-b0df6c69-gw-openshift-default-86797cc876-lklw2\" (UID: \"f0d61ced-695f-49b3-a86b-732843026f8b\") " pod="openshift-ingress/e2e-migrate-b0df6c69-gw-openshift-default-86797cc876-lklw2"
Sep 09 02:45:57.935069 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:57.934863    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znstk\" (UniqueName: \"kubernetes.io/projected/f0d61ced-695f-49b3-a86b-732843026f8b-kube-api-access-znstk\") pod \"e2e-migrate-b0df6c69-gw-openshift-default-86797cc876-lklw2\" (UID: \"f0d61ced-695f-49b3-a86b-732843026f8b\") " pod="openshift-ingress/e2e-migrate-b0df6c69-gw-openshift-default-86797cc876-lklw2"
Sep 09 02:45:57.935069 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:57.934882    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0d61ced-695f-49b3-a86b-732843026f8b-istio-data\") pod \"e2e-migrate-b0df6c69-gw-openshift-default-86797cc876-lklw2\" (UID: \"f0d61ced-695f-49b3-a86b-732843026f8b\") " pod="openshift-ingress/e2e-migrate-b0df6c69-gw-openshift-default-86797cc876-lklw2"
Sep 09 02:45:57.935069 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:57.934908    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0d61ced-695f-49b3-a86b-732843026f8b-istio-podinfo\") pod \"e2e-migrate-b0df6c69-gw-openshift-default-86797cc876-lklw2\" (UID: \"f0d61ced-695f-49b3-a86b-732843026f8b\") " pod="openshift-ingress/e2e-migrate-b0df6c69-gw-openshift-default-86797cc876-lklw2"
Sep 09 02:45:57.935069 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:57.934934    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0d61ced-695f-49b3-a86b-732843026f8b-credential-socket\") pod \"e2e-migrate-b0df6c69-gw-openshift-default-86797cc876-lklw2\" (UID: \"f0d61ced-695f-49b3-a86b-732843026f8b\") " pod="openshift-ingress/e2e-migrate-b0df6c69-gw-openshift-default-86797cc876-lklw2"
Sep 09 02:45:57.935269 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:57.935097    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0d61ced-695f-49b3-a86b-732843026f8b-workload-socket\") pod \"e2e-migrate-b0df6c69-gw-openshift-default-86797cc876-lklw2\" (UID: \"f0d61ced-695f-49b3-a86b-732843026f8b\") " pod="openshift-ingress/e2e-migrate-b0df6c69-gw-openshift-default-86797cc876-lklw2"
Sep 09 02:45:57.935269 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:57.935163    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0d61ced-695f-49b3-a86b-732843026f8b-istio-token\") pod \"e2e-migrate-b0df6c69-gw-openshift-default-86797cc876-lklw2\" (UID: \"f0d61ced-695f-49b3-a86b-732843026f8b\") " pod="openshift-ingress/e2e-migrate-b0df6c69-gw-openshift-default-86797cc876-lklw2"
Sep 09 02:45:57.935269 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:57.935218    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0d61ced-695f-49b3-a86b-732843026f8b-workload-certs\") pod \"e2e-migrate-b0df6c69-gw-openshift-default-86797cc876-lklw2\" (UID: \"f0d61ced-695f-49b3-a86b-732843026f8b\") " pod="openshift-ingress/e2e-migrate-b0df6c69-gw-openshift-default-86797cc876-lklw2"
Sep 09 02:45:57.935706 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:57.935644    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0d61ced-695f-49b3-a86b-732843026f8b-credential-socket\") pod \"e2e-migrate-b0df6c69-gw-openshift-default-86797cc876-lklw2\" (UID: \"f0d61ced-695f-49b3-a86b-732843026f8b\") " pod="openshift-ingress/e2e-migrate-b0df6c69-gw-openshift-default-86797cc876-lklw2"
Sep 09 02:45:57.935706 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:57.935696    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0d61ced-695f-49b3-a86b-732843026f8b-istio-data\") pod \"e2e-migrate-b0df6c69-gw-openshift-default-86797cc876-lklw2\" (UID: \"f0d61ced-695f-49b3-a86b-732843026f8b\") " pod="openshift-ingress/e2e-migrate-b0df6c69-gw-openshift-default-86797cc876-lklw2"
Sep 09 02:45:57.935880 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:57.935709    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0d61ced-695f-49b3-a86b-732843026f8b-istiod-ca-cert\") pod \"e2e-migrate-b0df6c69-gw-openshift-default-86797cc876-lklw2\" (UID: \"f0d61ced-695f-49b3-a86b-732843026f8b\") " pod="openshift-ingress/e2e-migrate-b0df6c69-gw-openshift-default-86797cc876-lklw2"
Sep 09 02:45:57.935880 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:57.935796    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0d61ced-695f-49b3-a86b-732843026f8b-workload-socket\") pod \"e2e-migrate-b0df6c69-gw-openshift-default-86797cc876-lklw2\" (UID: \"f0d61ced-695f-49b3-a86b-732843026f8b\") " pod="openshift-ingress/e2e-migrate-b0df6c69-gw-openshift-default-86797cc876-lklw2"
Sep 09 02:45:57.935880 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:57.935827    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0d61ced-695f-49b3-a86b-732843026f8b-workload-certs\") pod \"e2e-migrate-b0df6c69-gw-openshift-default-86797cc876-lklw2\" (UID: \"f0d61ced-695f-49b3-a86b-732843026f8b\") " pod="openshift-ingress/e2e-migrate-b0df6c69-gw-openshift-default-86797cc876-lklw2"
Sep 09 02:45:57.937652 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:57.937627    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0d61ced-695f-49b3-a86b-732843026f8b-istio-envoy\") pod \"e2e-migrate-b0df6c69-gw-openshift-default-86797cc876-lklw2\" (UID: \"f0d61ced-695f-49b3-a86b-732843026f8b\") " pod="openshift-ingress/e2e-migrate-b0df6c69-gw-openshift-default-86797cc876-lklw2"
Sep 09 02:45:57.937752 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:57.937738    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0d61ced-695f-49b3-a86b-732843026f8b-istio-podinfo\") pod \"e2e-migrate-b0df6c69-gw-openshift-default-86797cc876-lklw2\" (UID: \"f0d61ced-695f-49b3-a86b-732843026f8b\") " pod="openshift-ingress/e2e-migrate-b0df6c69-gw-openshift-default-86797cc876-lklw2"
Sep 09 02:45:57.943711 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:57.943690    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0d61ced-695f-49b3-a86b-732843026f8b-istio-token\") pod \"e2e-migrate-b0df6c69-gw-openshift-default-86797cc876-lklw2\" (UID: \"f0d61ced-695f-49b3-a86b-732843026f8b\") " pod="openshift-ingress/e2e-migrate-b0df6c69-gw-openshift-default-86797cc876-lklw2"
Sep 09 02:45:57.944113 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:57.944093    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-znstk\" (UniqueName: \"kubernetes.io/projected/f0d61ced-695f-49b3-a86b-732843026f8b-kube-api-access-znstk\") pod \"e2e-migrate-b0df6c69-gw-openshift-default-86797cc876-lklw2\" (UID: \"f0d61ced-695f-49b3-a86b-732843026f8b\") " pod="openshift-ingress/e2e-migrate-b0df6c69-gw-openshift-default-86797cc876-lklw2"
Sep 09 02:45:58.035725 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:58.035658    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-b0df6c69-gw-openshift-default-86797cc876-lklw2"
Sep 09 02:45:58.223062 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:58.223022    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6a91f20-b0a2-476a-b3e1-48cd91071d8e" path="/var/lib/kubelet/pods/c6a91f20-b0a2-476a-b3e1-48cd91071d8e/volumes"
Sep 09 02:45:58.288052 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:58.287953    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6b11a2-openshift-default-5cfbd4b94-fql7q" event={"ID":"2a7cadb9-da51-457d-8c72-007236378887","Type":"ContainerStarted","Data":"71a36d8103aa77732e1409b7531eace16cb2840c4c9676bdbcc7bfc95dcce12d"}
Sep 09 02:45:58.347684 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:58.340644    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-b0df6c69-gw-openshift-default-86797cc876-lklw2"]
Sep 09 02:45:58.351252 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:58.351207    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 02:45:58.351469 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:58.351443    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 02:45:58.351628 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:58.351607    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 02:45:58.378544 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:58.378514    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-99qjf"]
Sep 09 02:45:58.575628 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:58.575528    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-6b11a2-openshift-default-5cfbd4b94-fql7q"
Sep 09 02:45:58.575804 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:58.575700    2570 patch_prober.go:28] interesting pod/e2e-inf-b-6b11a2-openshift-default-5cfbd4b94-fql7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Sep 09 02:45:58.575804 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:58.575743    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6b11a2-openshift-default-5cfbd4b94-fql7q" podUID="2a7cadb9-da51-457d-8c72-007236378887" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Sep 09 02:45:59.294443 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:59.294398    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-b0df6c69-gw-openshift-default-86797cc876-lklw2" event={"ID":"f0d61ced-695f-49b3-a86b-732843026f8b","Type":"ContainerStarted","Data":"36a53bfd02f96d8eba83887a6f15ade7867a2205fae0b6fa9f4083a7216c32a6"}
Sep 09 02:45:59.294845 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:59.294476    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-b0df6c69-gw-openshift-default-86797cc876-lklw2" event={"ID":"f0d61ced-695f-49b3-a86b-732843026f8b","Type":"ContainerStarted","Data":"16eebc82b9db8ef052a09cc4a91bfb718d0ffaabe3c8df53e31b2b5335ff3805"}
Sep 09 02:45:59.318475 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:59.318414    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-b0df6c69-gw-openshift-default-86797cc876-lklw2" podStartSLOduration=2.318384315 podStartE2EDuration="2.318384315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:45:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:45:59.315427579 +0000 UTC m=+941.569731216" watchObservedRunningTime="2026-09-09 02:45:59.318384315 +0000 UTC m=+941.572687952"
Sep 09 02:45:59.575884 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:59.575795    2570 patch_prober.go:28] interesting pod/e2e-inf-b-6b11a2-openshift-default-5cfbd4b94-fql7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Sep 09 02:45:59.575884 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:45:59.575868    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6b11a2-openshift-default-5cfbd4b94-fql7q" podUID="2a7cadb9-da51-457d-8c72-007236378887" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Sep 09 02:46:00.036718 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:00.036683    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-b0df6c69-gw-openshift-default-86797cc876-lklw2"
Sep 09 02:46:00.048485 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:00.048445    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-b0df6c69-gw-openshift-default-86797cc876-lklw2"
Sep 09 02:46:00.299691 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:00.299599    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-b0df6c69-gw-openshift-default-86797cc876-lklw2"
Sep 09 02:46:00.300503 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:00.300482    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-b0df6c69-gw-openshift-default-86797cc876-lklw2"
Sep 09 02:46:00.575620 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:00.575531    2570 patch_prober.go:28] interesting pod/e2e-inf-b-6b11a2-openshift-default-5cfbd4b94-fql7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Sep 09 02:46:00.575620 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:00.575586    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6b11a2-openshift-default-5cfbd4b94-fql7q" podUID="2a7cadb9-da51-457d-8c72-007236378887" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Sep 09 02:46:01.575030 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:01.574961    2570 patch_prober.go:28] interesting pod/e2e-inf-b-6b11a2-openshift-default-5cfbd4b94-fql7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Sep 09 02:46:01.575388 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:01.575070    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6b11a2-openshift-default-5cfbd4b94-fql7q" podUID="2a7cadb9-da51-457d-8c72-007236378887" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Sep 09 02:46:02.087706 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:02.087681    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-79a2702e-openshift-default-6dc4c95574-g2jlk"
Sep 09 02:46:02.192206 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:02.192117    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe-istio-data\") pod \"bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe\" (UID: \"bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe\") "
Sep 09 02:46:02.192206 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:02.192161    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe-istio-envoy\") pod \"bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe\" (UID: \"bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe\") "
Sep 09 02:46:02.192206 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:02.192187    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe-istio-token\") pod \"bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe\" (UID: \"bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe\") "
Sep 09 02:46:02.192477 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:02.192217    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v8tg2\" (UniqueName: \"kubernetes.io/projected/bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe-kube-api-access-v8tg2\") pod \"bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe\" (UID: \"bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe\") "
Sep 09 02:46:02.192477 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:02.192238    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe-istiod-ca-cert\") pod \"bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe\" (UID: \"bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe\") "
Sep 09 02:46:02.192477 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:02.192261    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe-credential-socket\") pod \"bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe\" (UID: \"bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe\") "
Sep 09 02:46:02.192477 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:02.192299    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe-workload-certs\") pod \"bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe\" (UID: \"bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe\") "
Sep 09 02:46:02.192477 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:02.192414    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe-istio-data" (OuterVolumeSpecName: "istio-data") pod "bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe" (UID: "bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 02:46:02.192731 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:02.192484    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe-workload-socket\") pod \"bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe\" (UID: \"bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe\") "
Sep 09 02:46:02.192731 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:02.192590    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe" (UID: "bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 02:46:02.192731 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:02.192631    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe-istio-podinfo\") pod \"bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe\" (UID: \"bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe\") "
Sep 09 02:46:02.192731 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:02.192663    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe" (UID: "bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 02:46:02.192914 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:02.192749    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe" (UID: "bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 02:46:02.192914 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:02.192810    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe" (UID: "bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 02:46:02.193058 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:02.193003    2570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe-istio-data\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:46:02.193058 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:02.193022    2570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe-istiod-ca-cert\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:46:02.193058 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:02.193038    2570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe-credential-socket\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:46:02.193058 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:02.193050    2570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe-workload-certs\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:46:02.193058 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:02.193057    2570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe-workload-socket\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:46:02.194656 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:02.194622    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe-kube-api-access-v8tg2" (OuterVolumeSpecName: "kube-api-access-v8tg2") pod "bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe" (UID: "bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe"). InnerVolumeSpecName "kube-api-access-v8tg2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 02:46:02.194775 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:02.194709    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe" (UID: "bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 02:46:02.194775 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:02.194721    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe-istio-token" (OuterVolumeSpecName: "istio-token") pod "bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe" (UID: "bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 02:46:02.194775 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:02.194745    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe" (UID: "bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 02:46:02.294165 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:02.294123    2570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe-istio-podinfo\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:46:02.294165 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:02.294156    2570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe-istio-envoy\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:46:02.294165 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:02.294165    2570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe-istio-token\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:46:02.294165 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:02.294174    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8tg2\" (UniqueName: \"kubernetes.io/projected/bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe-kube-api-access-v8tg2\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:46:02.311114 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:02.311077    2570 generic.go:356] "Generic (PLEG): container finished" podID="bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe" containerID="86f7228cfe198a21d5302e7800b83518717c97f9a8932ae8094904550b0ffc08" exitCode=0
Sep 09 02:46:02.311282 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:02.311135    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-79a2702e-openshift-default-6dc4c95574-g2jlk"
Sep 09 02:46:02.311282 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:02.311157    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-79a2702e-openshift-default-6dc4c95574-g2jlk" event={"ID":"bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe","Type":"ContainerDied","Data":"86f7228cfe198a21d5302e7800b83518717c97f9a8932ae8094904550b0ffc08"}
Sep 09 02:46:02.311282 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:02.311208    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-79a2702e-openshift-default-6dc4c95574-g2jlk" event={"ID":"bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe","Type":"ContainerDied","Data":"272a9c780b02e7819c1e9fe4b824827290366569452c19b54d00656b9766f6a2"}
Sep 09 02:46:02.311282 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:02.311231    2570 scope.go:117] "RemoveContainer" containerID="86f7228cfe198a21d5302e7800b83518717c97f9a8932ae8094904550b0ffc08"
Sep 09 02:46:02.322611 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:02.322590    2570 scope.go:117] "RemoveContainer" containerID="7bd7f190651b692fab4ba95629e85ec8c537dbf4dd40d58d42ab875e6dcc7083"
Sep 09 02:46:02.330735 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:02.330710    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-79a2702e-openshift-default-6dc4c95574-g2jlk"]
Sep 09 02:46:02.331021 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:02.331008    2570 scope.go:117] "RemoveContainer" containerID="86f7228cfe198a21d5302e7800b83518717c97f9a8932ae8094904550b0ffc08"
Sep 09 02:46:02.331255 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:46:02.331239    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86f7228cfe198a21d5302e7800b83518717c97f9a8932ae8094904550b0ffc08\": container with ID starting with 86f7228cfe198a21d5302e7800b83518717c97f9a8932ae8094904550b0ffc08 not found: ID does not exist" containerID="86f7228cfe198a21d5302e7800b83518717c97f9a8932ae8094904550b0ffc08"
Sep 09 02:46:02.331298 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:02.331264    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86f7228cfe198a21d5302e7800b83518717c97f9a8932ae8094904550b0ffc08"} err="failed to get container status \"86f7228cfe198a21d5302e7800b83518717c97f9a8932ae8094904550b0ffc08\": rpc error: code = NotFound desc = could not find container \"86f7228cfe198a21d5302e7800b83518717c97f9a8932ae8094904550b0ffc08\": container with ID starting with 86f7228cfe198a21d5302e7800b83518717c97f9a8932ae8094904550b0ffc08 not found: ID does not exist"
Sep 09 02:46:02.331298 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:02.331279    2570 scope.go:117] "RemoveContainer" containerID="7bd7f190651b692fab4ba95629e85ec8c537dbf4dd40d58d42ab875e6dcc7083"
Sep 09 02:46:02.331491 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:46:02.331475    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bd7f190651b692fab4ba95629e85ec8c537dbf4dd40d58d42ab875e6dcc7083\": container with ID starting with 7bd7f190651b692fab4ba95629e85ec8c537dbf4dd40d58d42ab875e6dcc7083 not found: ID does not exist" containerID="7bd7f190651b692fab4ba95629e85ec8c537dbf4dd40d58d42ab875e6dcc7083"
Sep 09 02:46:02.331525 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:02.331495    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bd7f190651b692fab4ba95629e85ec8c537dbf4dd40d58d42ab875e6dcc7083"} err="failed to get container status \"7bd7f190651b692fab4ba95629e85ec8c537dbf4dd40d58d42ab875e6dcc7083\": rpc error: code = NotFound desc = could not find container \"7bd7f190651b692fab4ba95629e85ec8c537dbf4dd40d58d42ab875e6dcc7083\": container with ID starting with 7bd7f190651b692fab4ba95629e85ec8c537dbf4dd40d58d42ab875e6dcc7083 not found: ID does not exist"
Sep 09 02:46:02.335826 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:02.335804    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-79a2702e-openshift-default-6dc4c95574-g2jlk"]
Sep 09 02:46:02.576077 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:02.575959    2570 patch_prober.go:28] interesting pod/e2e-inf-b-6b11a2-openshift-default-5cfbd4b94-fql7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Sep 09 02:46:02.576512 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:02.576071    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6b11a2-openshift-default-5cfbd4b94-fql7q" podUID="2a7cadb9-da51-457d-8c72-007236378887" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Sep 09 02:46:03.575397 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:03.575358    2570 patch_prober.go:28] interesting pod/e2e-inf-b-6b11a2-openshift-default-5cfbd4b94-fql7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Sep 09 02:46:03.575576 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:03.575431    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6b11a2-openshift-default-5cfbd4b94-fql7q" podUID="2a7cadb9-da51-457d-8c72-007236378887" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Sep 09 02:46:04.224273 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:04.224236    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe" path="/var/lib/kubelet/pods/bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe/volumes"
Sep 09 02:46:04.578656 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:04.578586    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-6b11a2-openshift-default-5cfbd4b94-fql7q"
Sep 09 02:46:04.578874 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:04.578845    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-6b11a2-openshift-default-5cfbd4b94-fql7q"
Sep 09 02:46:04.579599 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:04.579582    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-6b11a2-openshift-default-5cfbd4b94-fql7q"
Sep 09 02:46:13.346152 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:13.346108    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-99qjf"]
Sep 09 02:46:16.340162 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:16.340109    2570 patch_prober.go:28] interesting pod/e2e-migrate-b0df6c69-gw-openshift-default-86797cc876-lklw2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 09 02:46:16.340628 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:16.340183    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-b0df6c69-gw-openshift-default-86797cc876-lklw2" podUID="f0d61ced-695f-49b3-a86b-732843026f8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 09 02:46:17.399179 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:17.399146    2570 generic.go:356] "Generic (PLEG): container finished" podID="f0d61ced-695f-49b3-a86b-732843026f8b" containerID="36a53bfd02f96d8eba83887a6f15ade7867a2205fae0b6fa9f4083a7216c32a6" exitCode=0
Sep 09 02:46:17.399651 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:17.399221    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-b0df6c69-gw-openshift-default-86797cc876-lklw2" event={"ID":"f0d61ced-695f-49b3-a86b-732843026f8b","Type":"ContainerDied","Data":"36a53bfd02f96d8eba83887a6f15ade7867a2205fae0b6fa9f4083a7216c32a6"}
Sep 09 02:46:17.399651 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:17.399623    2570 scope.go:117] "RemoveContainer" containerID="36a53bfd02f96d8eba83887a6f15ade7867a2205fae0b6fa9f4083a7216c32a6"
Sep 09 02:46:17.400681 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:17.400656    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 02:46:17.400739 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:17.400730    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 02:46:17.400794 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:17.400760    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 02:46:18.406834 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:18.406791    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-b0df6c69-gw-openshift-default-86797cc876-lklw2" event={"ID":"f0d61ced-695f-49b3-a86b-732843026f8b","Type":"ContainerStarted","Data":"b226dcbed47c4f0f96b48a5e27bf4b4f338aab14387df43ab7c522a223a24c19"}
Sep 09 02:46:19.036285 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:19.036246    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-b0df6c69-gw-openshift-default-86797cc876-lklw2"
Sep 09 02:46:19.036484 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:19.036437    2570 patch_prober.go:28] interesting pod/e2e-migrate-b0df6c69-gw-openshift-default-86797cc876-lklw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Sep 09 02:46:19.036557 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:19.036492    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-b0df6c69-gw-openshift-default-86797cc876-lklw2" podUID="f0d61ced-695f-49b3-a86b-732843026f8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Sep 09 02:46:20.036342 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:20.036308    2570 patch_prober.go:28] interesting pod/e2e-migrate-b0df6c69-gw-openshift-default-86797cc876-lklw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Sep 09 02:46:20.036720 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:20.036377    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-b0df6c69-gw-openshift-default-86797cc876-lklw2" podUID="f0d61ced-695f-49b3-a86b-732843026f8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Sep 09 02:46:21.036708 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:21.036664    2570 patch_prober.go:28] interesting pod/e2e-migrate-b0df6c69-gw-openshift-default-86797cc876-lklw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Sep 09 02:46:21.037125 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:21.036729    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-b0df6c69-gw-openshift-default-86797cc876-lklw2" podUID="f0d61ced-695f-49b3-a86b-732843026f8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Sep 09 02:46:22.037065 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:22.037029    2570 patch_prober.go:28] interesting pod/e2e-migrate-b0df6c69-gw-openshift-default-86797cc876-lklw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Sep 09 02:46:22.037443 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:22.037101    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-b0df6c69-gw-openshift-default-86797cc876-lklw2" podUID="f0d61ced-695f-49b3-a86b-732843026f8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Sep 09 02:46:23.036259 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:23.036219    2570 patch_prober.go:28] interesting pod/e2e-migrate-b0df6c69-gw-openshift-default-86797cc876-lklw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Sep 09 02:46:23.036444 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:23.036291    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-b0df6c69-gw-openshift-default-86797cc876-lklw2" podUID="f0d61ced-695f-49b3-a86b-732843026f8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Sep 09 02:46:24.051333 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:24.051288    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-b0df6c69-gw-openshift-default-86797cc876-lklw2"
Sep 09 02:46:24.051819 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:24.051738    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-b0df6c69-gw-openshift-default-86797cc876-lklw2"
Sep 09 02:46:24.052465 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:24.052446    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-b0df6c69-gw-openshift-default-86797cc876-lklw2"
Sep 09 02:46:25.662476 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:25.662441    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b0df6c69-84fhh"]
Sep 09 02:46:25.662998 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:25.662963    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe" containerName="istio-proxy"
Sep 09 02:46:25.663073 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:25.663002    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe" containerName="istio-proxy"
Sep 09 02:46:25.663073 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:25.663034    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe" containerName="istio-proxy"
Sep 09 02:46:25.663073 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:25.663043    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe" containerName="istio-proxy"
Sep 09 02:46:25.663227 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:25.663147    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe" containerName="istio-proxy"
Sep 09 02:46:25.663227 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:25.663162    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd54615b-f6c5-446b-8d7c-9f2ec5a5fcbe" containerName="istio-proxy"
Sep 09 02:46:25.667561 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:25.667542    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b0df6c69-84fhh"
Sep 09 02:46:25.670839 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:25.670812    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 02:46:25.670839 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:25.670824    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vbr52\""
Sep 09 02:46:25.676384 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:25.676357    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b0df6c69-84fhh"]
Sep 09 02:46:25.723930 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:25.723888    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce41f45e-d729-40e0-86f4-b22631a6ee05-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b0df6c69-84fhh\" (UID: \"ce41f45e-d729-40e0-86f4-b22631a6ee05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b0df6c69-84fhh"
Sep 09 02:46:25.825287 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:25.825241    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce41f45e-d729-40e0-86f4-b22631a6ee05-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b0df6c69-84fhh\" (UID: \"ce41f45e-d729-40e0-86f4-b22631a6ee05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b0df6c69-84fhh"
Sep 09 02:46:25.825895 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:25.825871    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce41f45e-d729-40e0-86f4-b22631a6ee05-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b0df6c69-84fhh\" (UID: \"ce41f45e-d729-40e0-86f4-b22631a6ee05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b0df6c69-84fhh"
Sep 09 02:46:25.978396 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:25.978305    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b0df6c69-84fhh"
Sep 09 02:46:26.249364 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:26.249339    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b0df6c69-84fhh"]
Sep 09 02:46:26.251308 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:46:26.251281    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce41f45e_d729_40e0_86f4_b22631a6ee05.slice/crio-4a78d6474206d85cf26880a94c1715b1ce79ee3d158043cf805e0860dbc683e3 WatchSource:0}: Error finding container 4a78d6474206d85cf26880a94c1715b1ce79ee3d158043cf805e0860dbc683e3: Status 404 returned error can't find the container with id 4a78d6474206d85cf26880a94c1715b1ce79ee3d158043cf805e0860dbc683e3
Sep 09 02:46:26.446242 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:26.446212    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b0df6c69-84fhh" event={"ID":"ce41f45e-d729-40e0-86f4-b22631a6ee05","Type":"ContainerStarted","Data":"4a78d6474206d85cf26880a94c1715b1ce79ee3d158043cf805e0860dbc683e3"}
Sep 09 02:46:27.452631 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:27.452598    2570 generic.go:356] "Generic (PLEG): container finished" podID="ce41f45e-d729-40e0-86f4-b22631a6ee05" containerID="15e3d6983dcf0946f0613690a09acc259da489e3af8711c93d87cf738259f0da" exitCode=0
Sep 09 02:46:27.453026 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:27.452665    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b0df6c69-84fhh" event={"ID":"ce41f45e-d729-40e0-86f4-b22631a6ee05","Type":"ContainerDied","Data":"15e3d6983dcf0946f0613690a09acc259da489e3af8711c93d87cf738259f0da"}
Sep 09 02:46:27.641306 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:27.641264    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-b-80fff954-kserve-5cf6f4f767-pp45n"]
Sep 09 02:46:27.651340 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:27.651311    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-80fff954-kserve-5cf6f4f767-pp45n"
Sep 09 02:46:27.654260 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:27.654233    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-b-80fff954-kserve-self-signed-certs\""
Sep 09 02:46:27.655558 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:27.655529    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-80fff954-kserve-5cf6f4f767-pp45n"]
Sep 09 02:46:27.744288 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:27.744203    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d685t\" (UniqueName: \"kubernetes.io/projected/f8a5bc9c-eb60-4978-88a4-ebf506f37ad2-kube-api-access-d685t\") pod \"e2e-conflict-b-80fff954-kserve-5cf6f4f767-pp45n\" (UID: \"f8a5bc9c-eb60-4978-88a4-ebf506f37ad2\") " pod="llm/e2e-conflict-b-80fff954-kserve-5cf6f4f767-pp45n"
Sep 09 02:46:27.744433 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:27.744289    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8a5bc9c-eb60-4978-88a4-ebf506f37ad2-model-cache\") pod \"e2e-conflict-b-80fff954-kserve-5cf6f4f767-pp45n\" (UID: \"f8a5bc9c-eb60-4978-88a4-ebf506f37ad2\") " pod="llm/e2e-conflict-b-80fff954-kserve-5cf6f4f767-pp45n"
Sep 09 02:46:27.744433 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:27.744344    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8a5bc9c-eb60-4978-88a4-ebf506f37ad2-tls-certs\") pod \"e2e-conflict-b-80fff954-kserve-5cf6f4f767-pp45n\" (UID: \"f8a5bc9c-eb60-4978-88a4-ebf506f37ad2\") " pod="llm/e2e-conflict-b-80fff954-kserve-5cf6f4f767-pp45n"
Sep 09 02:46:27.744433 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:27.744390    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8a5bc9c-eb60-4978-88a4-ebf506f37ad2-dshm\") pod \"e2e-conflict-b-80fff954-kserve-5cf6f4f767-pp45n\" (UID: \"f8a5bc9c-eb60-4978-88a4-ebf506f37ad2\") " pod="llm/e2e-conflict-b-80fff954-kserve-5cf6f4f767-pp45n"
Sep 09 02:46:27.744543 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:27.744434    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8a5bc9c-eb60-4978-88a4-ebf506f37ad2-home\") pod \"e2e-conflict-b-80fff954-kserve-5cf6f4f767-pp45n\" (UID: \"f8a5bc9c-eb60-4978-88a4-ebf506f37ad2\") " pod="llm/e2e-conflict-b-80fff954-kserve-5cf6f4f767-pp45n"
Sep 09 02:46:27.744543 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:27.744476    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8a5bc9c-eb60-4978-88a4-ebf506f37ad2-tmp-dir\") pod \"e2e-conflict-b-80fff954-kserve-5cf6f4f767-pp45n\" (UID: \"f8a5bc9c-eb60-4978-88a4-ebf506f37ad2\") " pod="llm/e2e-conflict-b-80fff954-kserve-5cf6f4f767-pp45n"
Sep 09 02:46:27.845645 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:27.845612    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d685t\" (UniqueName: \"kubernetes.io/projected/f8a5bc9c-eb60-4978-88a4-ebf506f37ad2-kube-api-access-d685t\") pod \"e2e-conflict-b-80fff954-kserve-5cf6f4f767-pp45n\" (UID: \"f8a5bc9c-eb60-4978-88a4-ebf506f37ad2\") " pod="llm/e2e-conflict-b-80fff954-kserve-5cf6f4f767-pp45n"
Sep 09 02:46:27.845831 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:27.845683    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8a5bc9c-eb60-4978-88a4-ebf506f37ad2-model-cache\") pod \"e2e-conflict-b-80fff954-kserve-5cf6f4f767-pp45n\" (UID: \"f8a5bc9c-eb60-4978-88a4-ebf506f37ad2\") " pod="llm/e2e-conflict-b-80fff954-kserve-5cf6f4f767-pp45n"
Sep 09 02:46:27.845831 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:27.845712    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8a5bc9c-eb60-4978-88a4-ebf506f37ad2-tls-certs\") pod \"e2e-conflict-b-80fff954-kserve-5cf6f4f767-pp45n\" (UID: \"f8a5bc9c-eb60-4978-88a4-ebf506f37ad2\") " pod="llm/e2e-conflict-b-80fff954-kserve-5cf6f4f767-pp45n"
Sep 09 02:46:27.845831 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:27.845748    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8a5bc9c-eb60-4978-88a4-ebf506f37ad2-dshm\") pod \"e2e-conflict-b-80fff954-kserve-5cf6f4f767-pp45n\" (UID: \"f8a5bc9c-eb60-4978-88a4-ebf506f37ad2\") " pod="llm/e2e-conflict-b-80fff954-kserve-5cf6f4f767-pp45n"
Sep 09 02:46:27.845831 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:27.845785    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8a5bc9c-eb60-4978-88a4-ebf506f37ad2-home\") pod \"e2e-conflict-b-80fff954-kserve-5cf6f4f767-pp45n\" (UID: \"f8a5bc9c-eb60-4978-88a4-ebf506f37ad2\") " pod="llm/e2e-conflict-b-80fff954-kserve-5cf6f4f767-pp45n"
Sep 09 02:46:27.845831 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:27.845824    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8a5bc9c-eb60-4978-88a4-ebf506f37ad2-tmp-dir\") pod \"e2e-conflict-b-80fff954-kserve-5cf6f4f767-pp45n\" (UID: \"f8a5bc9c-eb60-4978-88a4-ebf506f37ad2\") " pod="llm/e2e-conflict-b-80fff954-kserve-5cf6f4f767-pp45n"
Sep 09 02:46:27.846212 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:27.846190    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8a5bc9c-eb60-4978-88a4-ebf506f37ad2-model-cache\") pod \"e2e-conflict-b-80fff954-kserve-5cf6f4f767-pp45n\" (UID: \"f8a5bc9c-eb60-4978-88a4-ebf506f37ad2\") " pod="llm/e2e-conflict-b-80fff954-kserve-5cf6f4f767-pp45n"
Sep 09 02:46:27.846212 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:27.846205    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8a5bc9c-eb60-4978-88a4-ebf506f37ad2-tmp-dir\") pod \"e2e-conflict-b-80fff954-kserve-5cf6f4f767-pp45n\" (UID: \"f8a5bc9c-eb60-4978-88a4-ebf506f37ad2\") " pod="llm/e2e-conflict-b-80fff954-kserve-5cf6f4f767-pp45n"
Sep 09 02:46:27.846343 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:27.846283    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8a5bc9c-eb60-4978-88a4-ebf506f37ad2-home\") pod \"e2e-conflict-b-80fff954-kserve-5cf6f4f767-pp45n\" (UID: \"f8a5bc9c-eb60-4978-88a4-ebf506f37ad2\") " pod="llm/e2e-conflict-b-80fff954-kserve-5cf6f4f767-pp45n"
Sep 09 02:46:27.848156 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:27.848134    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8a5bc9c-eb60-4978-88a4-ebf506f37ad2-dshm\") pod \"e2e-conflict-b-80fff954-kserve-5cf6f4f767-pp45n\" (UID: \"f8a5bc9c-eb60-4978-88a4-ebf506f37ad2\") " pod="llm/e2e-conflict-b-80fff954-kserve-5cf6f4f767-pp45n"
Sep 09 02:46:27.848500 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:27.848480    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8a5bc9c-eb60-4978-88a4-ebf506f37ad2-tls-certs\") pod \"e2e-conflict-b-80fff954-kserve-5cf6f4f767-pp45n\" (UID: \"f8a5bc9c-eb60-4978-88a4-ebf506f37ad2\") " pod="llm/e2e-conflict-b-80fff954-kserve-5cf6f4f767-pp45n"
Sep 09 02:46:27.854480 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:27.854456    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d685t\" (UniqueName: \"kubernetes.io/projected/f8a5bc9c-eb60-4978-88a4-ebf506f37ad2-kube-api-access-d685t\") pod \"e2e-conflict-b-80fff954-kserve-5cf6f4f767-pp45n\" (UID: \"f8a5bc9c-eb60-4978-88a4-ebf506f37ad2\") " pod="llm/e2e-conflict-b-80fff954-kserve-5cf6f4f767-pp45n"
Sep 09 02:46:27.962448 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:27.962408    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-80fff954-kserve-5cf6f4f767-pp45n"
Sep 09 02:46:28.238111 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:28.238088    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-80fff954-kserve-5cf6f4f767-pp45n"]
Sep 09 02:46:28.240640 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:46:28.240610    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf8a5bc9c_eb60_4978_88a4_ebf506f37ad2.slice/crio-bc91d543c06c09c626698192ae08ed4ac0a7303b40630c651c839cfb91bb1d2f WatchSource:0}: Error finding container bc91d543c06c09c626698192ae08ed4ac0a7303b40630c651c839cfb91bb1d2f: Status 404 returned error can't find the container with id bc91d543c06c09c626698192ae08ed4ac0a7303b40630c651c839cfb91bb1d2f
Sep 09 02:46:28.458961 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:28.458922    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-80fff954-kserve-5cf6f4f767-pp45n" event={"ID":"f8a5bc9c-eb60-4978-88a4-ebf506f37ad2","Type":"ContainerStarted","Data":"9d96181c007e5c13ec1fd18ee600c03ceeaad99ebb2369b64bebf908e1eb90ee"}
Sep 09 02:46:28.458961 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:28.458984    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-80fff954-kserve-5cf6f4f767-pp45n" event={"ID":"f8a5bc9c-eb60-4978-88a4-ebf506f37ad2","Type":"ContainerStarted","Data":"bc91d543c06c09c626698192ae08ed4ac0a7303b40630c651c839cfb91bb1d2f"}
Sep 09 02:46:28.479484 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:28.479425    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-b-80fff954-kserve-5cf6f4f767-pp45n" podStartSLOduration=1.479406644 podStartE2EDuration="1.479406644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:46:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:46:28.478000183 +0000 UTC m=+970.732303810" watchObservedRunningTime="2026-09-09 02:46:28.479406644 +0000 UTC m=+970.733710281"
Sep 09 02:46:28.587692 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:28.587662    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b0df6c69-84fhh"
Sep 09 02:46:28.653680 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:28.653649    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce41f45e-d729-40e0-86f4-b22631a6ee05-maas-api-service-ca\") pod \"ce41f45e-d729-40e0-86f4-b22631a6ee05\" (UID: \"ce41f45e-d729-40e0-86f4-b22631a6ee05\") "
Sep 09 02:46:28.654036 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:28.654013    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce41f45e-d729-40e0-86f4-b22631a6ee05-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ce41f45e-d729-40e0-86f4-b22631a6ee05" (UID: "ce41f45e-d729-40e0-86f4-b22631a6ee05"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 02:46:28.755068 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:28.755025    2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce41f45e-d729-40e0-86f4-b22631a6ee05-maas-api-service-ca\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:46:28.848596 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:28.848558    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-99qjf"]
Sep 09 02:46:29.465734 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:29.465703    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b0df6c69-84fhh"
Sep 09 02:46:29.466119 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:29.465742    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b0df6c69-84fhh" event={"ID":"ce41f45e-d729-40e0-86f4-b22631a6ee05","Type":"ContainerDied","Data":"4a78d6474206d85cf26880a94c1715b1ce79ee3d158043cf805e0860dbc683e3"}
Sep 09 02:46:29.466119 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:29.465788    2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4a78d6474206d85cf26880a94c1715b1ce79ee3d158043cf805e0860dbc683e3"
Sep 09 02:46:31.883472 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:31.883427    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b0df6c69-84fhh"]
Sep 09 02:46:31.889873 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:31.889846    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b0df6c69-84fhh"]
Sep 09 02:46:32.006621 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:32.006586    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-b0df6c69-gw-openshift-default-86797cc876-lklw2"]
Sep 09 02:46:32.006838 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:32.006803    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-b0df6c69-gw-openshift-default-86797cc876-lklw2" podUID="f0d61ced-695f-49b3-a86b-732843026f8b" containerName="istio-proxy" containerID="cri-o://b226dcbed47c4f0f96b48a5e27bf4b4f338aab14387df43ab7c522a223a24c19" gracePeriod=30
Sep 09 02:46:32.222443 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:32.222351    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce41f45e-d729-40e0-86f4-b22631a6ee05" path="/var/lib/kubelet/pods/ce41f45e-d729-40e0-86f4-b22631a6ee05/volumes"
Sep 09 02:46:32.728022 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:32.727986    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-7e17db81-openshift-default-794f5f8566-cp7m4"]
Sep 09 02:46:32.728355 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:32.728343    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce41f45e-d729-40e0-86f4-b22631a6ee05" containerName="revoke-keys"
Sep 09 02:46:32.728397 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:32.728357    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce41f45e-d729-40e0-86f4-b22631a6ee05" containerName="revoke-keys"
Sep 09 02:46:32.728463 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:32.728454    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce41f45e-d729-40e0-86f4-b22631a6ee05" containerName="revoke-keys"
Sep 09 02:46:32.731851 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:32.731830    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7e17db81-openshift-default-794f5f8566-cp7m4"
Sep 09 02:46:32.735102 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:32.735084    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-7e17db81-openshift-default-dockercfg-7rp4w\""
Sep 09 02:46:32.747472 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:32.747450    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-7e17db81-openshift-default-794f5f8566-cp7m4"]
Sep 09 02:46:32.804234 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:32.804193    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39b1c8a7-44a0-4a11-b079-f398788e7102-istio-envoy\") pod \"e2e-ait-7e17db81-openshift-default-794f5f8566-cp7m4\" (UID: \"39b1c8a7-44a0-4a11-b079-f398788e7102\") " pod="openshift-ingress/e2e-ait-7e17db81-openshift-default-794f5f8566-cp7m4"
Sep 09 02:46:32.804407 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:32.804258    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlr8p\" (UniqueName: \"kubernetes.io/projected/39b1c8a7-44a0-4a11-b079-f398788e7102-kube-api-access-tlr8p\") pod \"e2e-ait-7e17db81-openshift-default-794f5f8566-cp7m4\" (UID: \"39b1c8a7-44a0-4a11-b079-f398788e7102\") " pod="openshift-ingress/e2e-ait-7e17db81-openshift-default-794f5f8566-cp7m4"
Sep 09 02:46:32.804407 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:32.804354    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39b1c8a7-44a0-4a11-b079-f398788e7102-workload-certs\") pod \"e2e-ait-7e17db81-openshift-default-794f5f8566-cp7m4\" (UID: \"39b1c8a7-44a0-4a11-b079-f398788e7102\") " pod="openshift-ingress/e2e-ait-7e17db81-openshift-default-794f5f8566-cp7m4"
Sep 09 02:46:32.804407 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:32.804379    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39b1c8a7-44a0-4a11-b079-f398788e7102-istio-token\") pod \"e2e-ait-7e17db81-openshift-default-794f5f8566-cp7m4\" (UID: \"39b1c8a7-44a0-4a11-b079-f398788e7102\") " pod="openshift-ingress/e2e-ait-7e17db81-openshift-default-794f5f8566-cp7m4"
Sep 09 02:46:32.804407 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:32.804396    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39b1c8a7-44a0-4a11-b079-f398788e7102-istiod-ca-cert\") pod \"e2e-ait-7e17db81-openshift-default-794f5f8566-cp7m4\" (UID: \"39b1c8a7-44a0-4a11-b079-f398788e7102\") " pod="openshift-ingress/e2e-ait-7e17db81-openshift-default-794f5f8566-cp7m4"
Sep 09 02:46:32.804561 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:32.804414    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39b1c8a7-44a0-4a11-b079-f398788e7102-credential-socket\") pod \"e2e-ait-7e17db81-openshift-default-794f5f8566-cp7m4\" (UID: \"39b1c8a7-44a0-4a11-b079-f398788e7102\") " pod="openshift-ingress/e2e-ait-7e17db81-openshift-default-794f5f8566-cp7m4"
Sep 09 02:46:32.804561 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:32.804495    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39b1c8a7-44a0-4a11-b079-f398788e7102-workload-socket\") pod \"e2e-ait-7e17db81-openshift-default-794f5f8566-cp7m4\" (UID: \"39b1c8a7-44a0-4a11-b079-f398788e7102\") " pod="openshift-ingress/e2e-ait-7e17db81-openshift-default-794f5f8566-cp7m4"
Sep 09 02:46:32.804629 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:32.804558    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39b1c8a7-44a0-4a11-b079-f398788e7102-istio-data\") pod \"e2e-ait-7e17db81-openshift-default-794f5f8566-cp7m4\" (UID: \"39b1c8a7-44a0-4a11-b079-f398788e7102\") " pod="openshift-ingress/e2e-ait-7e17db81-openshift-default-794f5f8566-cp7m4"
Sep 09 02:46:32.804629 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:32.804584    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39b1c8a7-44a0-4a11-b079-f398788e7102-istio-podinfo\") pod \"e2e-ait-7e17db81-openshift-default-794f5f8566-cp7m4\" (UID: \"39b1c8a7-44a0-4a11-b079-f398788e7102\") " pod="openshift-ingress/e2e-ait-7e17db81-openshift-default-794f5f8566-cp7m4"
Sep 09 02:46:32.905660 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:32.905620    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39b1c8a7-44a0-4a11-b079-f398788e7102-istio-envoy\") pod \"e2e-ait-7e17db81-openshift-default-794f5f8566-cp7m4\" (UID: \"39b1c8a7-44a0-4a11-b079-f398788e7102\") " pod="openshift-ingress/e2e-ait-7e17db81-openshift-default-794f5f8566-cp7m4"
Sep 09 02:46:32.906103 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:32.905676    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlr8p\" (UniqueName: \"kubernetes.io/projected/39b1c8a7-44a0-4a11-b079-f398788e7102-kube-api-access-tlr8p\") pod \"e2e-ait-7e17db81-openshift-default-794f5f8566-cp7m4\" (UID: \"39b1c8a7-44a0-4a11-b079-f398788e7102\") " pod="openshift-ingress/e2e-ait-7e17db81-openshift-default-794f5f8566-cp7m4"
Sep 09 02:46:32.906103 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:32.905733    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39b1c8a7-44a0-4a11-b079-f398788e7102-workload-certs\") pod \"e2e-ait-7e17db81-openshift-default-794f5f8566-cp7m4\" (UID: \"39b1c8a7-44a0-4a11-b079-f398788e7102\") " pod="openshift-ingress/e2e-ait-7e17db81-openshift-default-794f5f8566-cp7m4"
Sep 09 02:46:32.906103 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:32.905750    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39b1c8a7-44a0-4a11-b079-f398788e7102-istio-token\") pod \"e2e-ait-7e17db81-openshift-default-794f5f8566-cp7m4\" (UID: \"39b1c8a7-44a0-4a11-b079-f398788e7102\") " pod="openshift-ingress/e2e-ait-7e17db81-openshift-default-794f5f8566-cp7m4"
Sep 09 02:46:32.906103 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:32.905887    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39b1c8a7-44a0-4a11-b079-f398788e7102-istiod-ca-cert\") pod \"e2e-ait-7e17db81-openshift-default-794f5f8566-cp7m4\" (UID: \"39b1c8a7-44a0-4a11-b079-f398788e7102\") " pod="openshift-ingress/e2e-ait-7e17db81-openshift-default-794f5f8566-cp7m4"
Sep 09 02:46:32.906103 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:32.905937    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39b1c8a7-44a0-4a11-b079-f398788e7102-credential-socket\") pod \"e2e-ait-7e17db81-openshift-default-794f5f8566-cp7m4\" (UID: \"39b1c8a7-44a0-4a11-b079-f398788e7102\") " pod="openshift-ingress/e2e-ait-7e17db81-openshift-default-794f5f8566-cp7m4"
Sep 09 02:46:32.906103 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:32.906002    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39b1c8a7-44a0-4a11-b079-f398788e7102-workload-socket\") pod \"e2e-ait-7e17db81-openshift-default-794f5f8566-cp7m4\" (UID: \"39b1c8a7-44a0-4a11-b079-f398788e7102\") " pod="openshift-ingress/e2e-ait-7e17db81-openshift-default-794f5f8566-cp7m4"
Sep 09 02:46:32.906103 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:32.906073    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39b1c8a7-44a0-4a11-b079-f398788e7102-istio-data\") pod \"e2e-ait-7e17db81-openshift-default-794f5f8566-cp7m4\" (UID: \"39b1c8a7-44a0-4a11-b079-f398788e7102\") " pod="openshift-ingress/e2e-ait-7e17db81-openshift-default-794f5f8566-cp7m4"
Sep 09 02:46:32.906497 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:32.906111    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39b1c8a7-44a0-4a11-b079-f398788e7102-istio-podinfo\") pod \"e2e-ait-7e17db81-openshift-default-794f5f8566-cp7m4\" (UID: \"39b1c8a7-44a0-4a11-b079-f398788e7102\") " pod="openshift-ingress/e2e-ait-7e17db81-openshift-default-794f5f8566-cp7m4"
Sep 09 02:46:32.906497 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:32.906170    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39b1c8a7-44a0-4a11-b079-f398788e7102-workload-certs\") pod \"e2e-ait-7e17db81-openshift-default-794f5f8566-cp7m4\" (UID: \"39b1c8a7-44a0-4a11-b079-f398788e7102\") " pod="openshift-ingress/e2e-ait-7e17db81-openshift-default-794f5f8566-cp7m4"
Sep 09 02:46:32.906497 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:32.906331    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39b1c8a7-44a0-4a11-b079-f398788e7102-credential-socket\") pod \"e2e-ait-7e17db81-openshift-default-794f5f8566-cp7m4\" (UID: \"39b1c8a7-44a0-4a11-b079-f398788e7102\") " pod="openshift-ingress/e2e-ait-7e17db81-openshift-default-794f5f8566-cp7m4"
Sep 09 02:46:32.906497 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:32.906458    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39b1c8a7-44a0-4a11-b079-f398788e7102-workload-socket\") pod \"e2e-ait-7e17db81-openshift-default-794f5f8566-cp7m4\" (UID: \"39b1c8a7-44a0-4a11-b079-f398788e7102\") " pod="openshift-ingress/e2e-ait-7e17db81-openshift-default-794f5f8566-cp7m4"
Sep 09 02:46:32.906497 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:32.906461    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39b1c8a7-44a0-4a11-b079-f398788e7102-istio-data\") pod \"e2e-ait-7e17db81-openshift-default-794f5f8566-cp7m4\" (UID: \"39b1c8a7-44a0-4a11-b079-f398788e7102\") " pod="openshift-ingress/e2e-ait-7e17db81-openshift-default-794f5f8566-cp7m4"
Sep 09 02:46:32.906665 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:32.906605    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39b1c8a7-44a0-4a11-b079-f398788e7102-istiod-ca-cert\") pod \"e2e-ait-7e17db81-openshift-default-794f5f8566-cp7m4\" (UID: \"39b1c8a7-44a0-4a11-b079-f398788e7102\") " pod="openshift-ingress/e2e-ait-7e17db81-openshift-default-794f5f8566-cp7m4"
Sep 09 02:46:32.907986 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:32.907961    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39b1c8a7-44a0-4a11-b079-f398788e7102-istio-envoy\") pod \"e2e-ait-7e17db81-openshift-default-794f5f8566-cp7m4\" (UID: \"39b1c8a7-44a0-4a11-b079-f398788e7102\") " pod="openshift-ingress/e2e-ait-7e17db81-openshift-default-794f5f8566-cp7m4"
Sep 09 02:46:32.908474 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:32.908458    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39b1c8a7-44a0-4a11-b079-f398788e7102-istio-podinfo\") pod \"e2e-ait-7e17db81-openshift-default-794f5f8566-cp7m4\" (UID: \"39b1c8a7-44a0-4a11-b079-f398788e7102\") " pod="openshift-ingress/e2e-ait-7e17db81-openshift-default-794f5f8566-cp7m4"
Sep 09 02:46:32.914994 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:32.914949    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlr8p\" (UniqueName: \"kubernetes.io/projected/39b1c8a7-44a0-4a11-b079-f398788e7102-kube-api-access-tlr8p\") pod \"e2e-ait-7e17db81-openshift-default-794f5f8566-cp7m4\" (UID: \"39b1c8a7-44a0-4a11-b079-f398788e7102\") " pod="openshift-ingress/e2e-ait-7e17db81-openshift-default-794f5f8566-cp7m4"
Sep 09 02:46:32.915156 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:32.915139    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39b1c8a7-44a0-4a11-b079-f398788e7102-istio-token\") pod \"e2e-ait-7e17db81-openshift-default-794f5f8566-cp7m4\" (UID: \"39b1c8a7-44a0-4a11-b079-f398788e7102\") " pod="openshift-ingress/e2e-ait-7e17db81-openshift-default-794f5f8566-cp7m4"
Sep 09 02:46:33.043676 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:33.043586    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7e17db81-openshift-default-794f5f8566-cp7m4"
Sep 09 02:46:33.323320 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:33.323292    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-7e17db81-openshift-default-794f5f8566-cp7m4"]
Sep 09 02:46:33.325024 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:46:33.324992    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39b1c8a7_44a0_4a11_b079_f398788e7102.slice/crio-95a54d26c3a3482bcfedc71b9d21e86aa01ccb79c15656ca3ee660dce9e2f3fb WatchSource:0}: Error finding container 95a54d26c3a3482bcfedc71b9d21e86aa01ccb79c15656ca3ee660dce9e2f3fb: Status 404 returned error can't find the container with id 95a54d26c3a3482bcfedc71b9d21e86aa01ccb79c15656ca3ee660dce9e2f3fb
Sep 09 02:46:33.327249 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:33.327205    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 02:46:33.327365 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:33.327277    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 02:46:33.327365 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:33.327305    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 02:46:33.487312 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:33.487271    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7e17db81-openshift-default-794f5f8566-cp7m4" event={"ID":"39b1c8a7-44a0-4a11-b079-f398788e7102","Type":"ContainerStarted","Data":"c056c3a08bf3b6aa65cc0bde42374cc5380191bf95e3427b82c051c12918b902"}
Sep 09 02:46:33.487708 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:33.487330    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7e17db81-openshift-default-794f5f8566-cp7m4" event={"ID":"39b1c8a7-44a0-4a11-b079-f398788e7102","Type":"ContainerStarted","Data":"95a54d26c3a3482bcfedc71b9d21e86aa01ccb79c15656ca3ee660dce9e2f3fb"}
Sep 09 02:46:33.509042 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:33.508939    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-7e17db81-openshift-default-794f5f8566-cp7m4" podStartSLOduration=1.50891789 podStartE2EDuration="1.50891789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:46:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:46:33.507272065 +0000 UTC m=+975.761575700" watchObservedRunningTime="2026-09-09 02:46:33.50891789 +0000 UTC m=+975.763221527"
Sep 09 02:46:34.044153 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:34.044110    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-7e17db81-openshift-default-794f5f8566-cp7m4"
Sep 09 02:46:34.049069 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:34.049043    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-7e17db81-openshift-default-794f5f8566-cp7m4"
Sep 09 02:46:34.492682 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:34.492654    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-7e17db81-openshift-default-794f5f8566-cp7m4"
Sep 09 02:46:34.493631 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:34.493611    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-7e17db81-openshift-default-794f5f8566-cp7m4"
Sep 09 02:46:37.261081 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:37.261056    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-b0df6c69-gw-openshift-default-86797cc876-lklw2"
Sep 09 02:46:37.354037 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:37.353995    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0d61ced-695f-49b3-a86b-732843026f8b-istio-token\") pod \"f0d61ced-695f-49b3-a86b-732843026f8b\" (UID: \"f0d61ced-695f-49b3-a86b-732843026f8b\") "
Sep 09 02:46:37.354227 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:37.354048    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0d61ced-695f-49b3-a86b-732843026f8b-workload-certs\") pod \"f0d61ced-695f-49b3-a86b-732843026f8b\" (UID: \"f0d61ced-695f-49b3-a86b-732843026f8b\") "
Sep 09 02:46:37.354227 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:37.354076    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0d61ced-695f-49b3-a86b-732843026f8b-istiod-ca-cert\") pod \"f0d61ced-695f-49b3-a86b-732843026f8b\" (UID: \"f0d61ced-695f-49b3-a86b-732843026f8b\") "
Sep 09 02:46:37.354227 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:37.354163    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0d61ced-695f-49b3-a86b-732843026f8b-workload-socket\") pod \"f0d61ced-695f-49b3-a86b-732843026f8b\" (UID: \"f0d61ced-695f-49b3-a86b-732843026f8b\") "
Sep 09 02:46:37.354227 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:37.354180    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0d61ced-695f-49b3-a86b-732843026f8b-istio-data\") pod \"f0d61ced-695f-49b3-a86b-732843026f8b\" (UID: \"f0d61ced-695f-49b3-a86b-732843026f8b\") "
Sep 09 02:46:37.354444 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:37.354239    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-znstk\" (UniqueName: \"kubernetes.io/projected/f0d61ced-695f-49b3-a86b-732843026f8b-kube-api-access-znstk\") pod \"f0d61ced-695f-49b3-a86b-732843026f8b\" (UID: \"f0d61ced-695f-49b3-a86b-732843026f8b\") "
Sep 09 02:46:37.354444 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:37.354274    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0d61ced-695f-49b3-a86b-732843026f8b-credential-socket\") pod \"f0d61ced-695f-49b3-a86b-732843026f8b\" (UID: \"f0d61ced-695f-49b3-a86b-732843026f8b\") "
Sep 09 02:46:37.354444 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:37.354308    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0d61ced-695f-49b3-a86b-732843026f8b-istio-envoy\") pod \"f0d61ced-695f-49b3-a86b-732843026f8b\" (UID: \"f0d61ced-695f-49b3-a86b-732843026f8b\") "
Sep 09 02:46:37.354444 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:37.354356    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0d61ced-695f-49b3-a86b-732843026f8b-istio-podinfo\") pod \"f0d61ced-695f-49b3-a86b-732843026f8b\" (UID: \"f0d61ced-695f-49b3-a86b-732843026f8b\") "
Sep 09 02:46:37.354444 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:37.354425    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0d61ced-695f-49b3-a86b-732843026f8b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f0d61ced-695f-49b3-a86b-732843026f8b" (UID: "f0d61ced-695f-49b3-a86b-732843026f8b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 02:46:37.354637 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:37.354493    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0d61ced-695f-49b3-a86b-732843026f8b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f0d61ced-695f-49b3-a86b-732843026f8b" (UID: "f0d61ced-695f-49b3-a86b-732843026f8b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 02:46:37.354637 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:37.354511    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0d61ced-695f-49b3-a86b-732843026f8b-istio-data" (OuterVolumeSpecName: "istio-data") pod "f0d61ced-695f-49b3-a86b-732843026f8b" (UID: "f0d61ced-695f-49b3-a86b-732843026f8b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 02:46:37.354637 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:37.354587    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0d61ced-695f-49b3-a86b-732843026f8b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f0d61ced-695f-49b3-a86b-732843026f8b" (UID: "f0d61ced-695f-49b3-a86b-732843026f8b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 02:46:37.354944 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:37.354859    2570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0d61ced-695f-49b3-a86b-732843026f8b-workload-certs\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:46:37.354944 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:37.354884    2570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0d61ced-695f-49b3-a86b-732843026f8b-istiod-ca-cert\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:46:37.354944 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:37.354898    2570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0d61ced-695f-49b3-a86b-732843026f8b-workload-socket\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:46:37.354944 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:37.354911    2570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0d61ced-695f-49b3-a86b-732843026f8b-istio-data\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:46:37.354944 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:37.354914    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0d61ced-695f-49b3-a86b-732843026f8b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f0d61ced-695f-49b3-a86b-732843026f8b" (UID: "f0d61ced-695f-49b3-a86b-732843026f8b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 02:46:37.356373 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:37.356351    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0d61ced-695f-49b3-a86b-732843026f8b-istio-token" (OuterVolumeSpecName: "istio-token") pod "f0d61ced-695f-49b3-a86b-732843026f8b" (UID: "f0d61ced-695f-49b3-a86b-732843026f8b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 02:46:37.356660 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:37.356642    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f0d61ced-695f-49b3-a86b-732843026f8b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f0d61ced-695f-49b3-a86b-732843026f8b" (UID: "f0d61ced-695f-49b3-a86b-732843026f8b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 02:46:37.356890 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:37.356868    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0d61ced-695f-49b3-a86b-732843026f8b-kube-api-access-znstk" (OuterVolumeSpecName: "kube-api-access-znstk") pod "f0d61ced-695f-49b3-a86b-732843026f8b" (UID: "f0d61ced-695f-49b3-a86b-732843026f8b"). InnerVolumeSpecName "kube-api-access-znstk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 02:46:37.357064 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:37.357049    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0d61ced-695f-49b3-a86b-732843026f8b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f0d61ced-695f-49b3-a86b-732843026f8b" (UID: "f0d61ced-695f-49b3-a86b-732843026f8b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 02:46:37.456346 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:37.456307    2570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0d61ced-695f-49b3-a86b-732843026f8b-credential-socket\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:46:37.456346 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:37.456340    2570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0d61ced-695f-49b3-a86b-732843026f8b-istio-envoy\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:46:37.456346 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:37.456350    2570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0d61ced-695f-49b3-a86b-732843026f8b-istio-podinfo\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:46:37.456346 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:37.456359    2570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0d61ced-695f-49b3-a86b-732843026f8b-istio-token\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:46:37.456622 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:37.456368    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znstk\" (UniqueName: \"kubernetes.io/projected/f0d61ced-695f-49b3-a86b-732843026f8b-kube-api-access-znstk\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:46:37.510379 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:37.510346    2570 generic.go:356] "Generic (PLEG): container finished" podID="f0d61ced-695f-49b3-a86b-732843026f8b" containerID="b226dcbed47c4f0f96b48a5e27bf4b4f338aab14387df43ab7c522a223a24c19" exitCode=0
Sep 09 02:46:37.510562 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:37.510416    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-b0df6c69-gw-openshift-default-86797cc876-lklw2"
Sep 09 02:46:37.510562 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:37.510440    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-b0df6c69-gw-openshift-default-86797cc876-lklw2" event={"ID":"f0d61ced-695f-49b3-a86b-732843026f8b","Type":"ContainerDied","Data":"b226dcbed47c4f0f96b48a5e27bf4b4f338aab14387df43ab7c522a223a24c19"}
Sep 09 02:46:37.510562 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:37.510525    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-b0df6c69-gw-openshift-default-86797cc876-lklw2" event={"ID":"f0d61ced-695f-49b3-a86b-732843026f8b","Type":"ContainerDied","Data":"16eebc82b9db8ef052a09cc4a91bfb718d0ffaabe3c8df53e31b2b5335ff3805"}
Sep 09 02:46:37.510562 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:37.510549    2570 scope.go:117] "RemoveContainer" containerID="b226dcbed47c4f0f96b48a5e27bf4b4f338aab14387df43ab7c522a223a24c19"
Sep 09 02:46:37.520358 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:37.520338    2570 scope.go:117] "RemoveContainer" containerID="36a53bfd02f96d8eba83887a6f15ade7867a2205fae0b6fa9f4083a7216c32a6"
Sep 09 02:46:37.528984 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:37.528950    2570 scope.go:117] "RemoveContainer" containerID="b226dcbed47c4f0f96b48a5e27bf4b4f338aab14387df43ab7c522a223a24c19"
Sep 09 02:46:37.529276 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:46:37.529255    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b226dcbed47c4f0f96b48a5e27bf4b4f338aab14387df43ab7c522a223a24c19\": container with ID starting with b226dcbed47c4f0f96b48a5e27bf4b4f338aab14387df43ab7c522a223a24c19 not found: ID does not exist" containerID="b226dcbed47c4f0f96b48a5e27bf4b4f338aab14387df43ab7c522a223a24c19"
Sep 09 02:46:37.529317 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:37.529288    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b226dcbed47c4f0f96b48a5e27bf4b4f338aab14387df43ab7c522a223a24c19"} err="failed to get container status \"b226dcbed47c4f0f96b48a5e27bf4b4f338aab14387df43ab7c522a223a24c19\": rpc error: code = NotFound desc = could not find container \"b226dcbed47c4f0f96b48a5e27bf4b4f338aab14387df43ab7c522a223a24c19\": container with ID starting with b226dcbed47c4f0f96b48a5e27bf4b4f338aab14387df43ab7c522a223a24c19 not found: ID does not exist"
Sep 09 02:46:37.529317 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:37.529307    2570 scope.go:117] "RemoveContainer" containerID="36a53bfd02f96d8eba83887a6f15ade7867a2205fae0b6fa9f4083a7216c32a6"
Sep 09 02:46:37.529556 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:46:37.529541    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36a53bfd02f96d8eba83887a6f15ade7867a2205fae0b6fa9f4083a7216c32a6\": container with ID starting with 36a53bfd02f96d8eba83887a6f15ade7867a2205fae0b6fa9f4083a7216c32a6 not found: ID does not exist" containerID="36a53bfd02f96d8eba83887a6f15ade7867a2205fae0b6fa9f4083a7216c32a6"
Sep 09 02:46:37.529593 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:37.529560    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36a53bfd02f96d8eba83887a6f15ade7867a2205fae0b6fa9f4083a7216c32a6"} err="failed to get container status \"36a53bfd02f96d8eba83887a6f15ade7867a2205fae0b6fa9f4083a7216c32a6\": rpc error: code = NotFound desc = could not find container \"36a53bfd02f96d8eba83887a6f15ade7867a2205fae0b6fa9f4083a7216c32a6\": container with ID starting with 36a53bfd02f96d8eba83887a6f15ade7867a2205fae0b6fa9f4083a7216c32a6 not found: ID does not exist"
Sep 09 02:46:37.535561 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:37.535539    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-b0df6c69-gw-openshift-default-86797cc876-lklw2"]
Sep 09 02:46:37.540841 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:37.540822    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-b0df6c69-gw-openshift-default-86797cc876-lklw2"]
Sep 09 02:46:37.963019 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:37.962958    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-b-80fff954-kserve-5cf6f4f767-pp45n"
Sep 09 02:46:37.963208 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:37.963036    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-b-80fff954-kserve-5cf6f4f767-pp45n"
Sep 09 02:46:37.975820 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:37.975793    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-b-80fff954-kserve-5cf6f4f767-pp45n"
Sep 09 02:46:38.033938 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:38.033901    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-07b2d4/autores-model-07b2d4-kserve-58b9c97855-mdwg4"]
Sep 09 02:46:38.034294 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:38.034281    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0d61ced-695f-49b3-a86b-732843026f8b" containerName="istio-proxy"
Sep 09 02:46:38.034294 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:38.034295    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0d61ced-695f-49b3-a86b-732843026f8b" containerName="istio-proxy"
Sep 09 02:46:38.034378 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:38.034304    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0d61ced-695f-49b3-a86b-732843026f8b" containerName="istio-proxy"
Sep 09 02:46:38.034378 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:38.034309    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0d61ced-695f-49b3-a86b-732843026f8b" containerName="istio-proxy"
Sep 09 02:46:38.034378 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:38.034361    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0d61ced-695f-49b3-a86b-732843026f8b" containerName="istio-proxy"
Sep 09 02:46:38.034378 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:38.034369    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0d61ced-695f-49b3-a86b-732843026f8b" containerName="istio-proxy"
Sep 09 02:46:38.038816 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:38.038800    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-07b2d4/autores-model-07b2d4-kserve-58b9c97855-mdwg4"
Sep 09 02:46:38.041648 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:38.041617    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-07b2d4\"/\"default-dockercfg-2f9kv\""
Sep 09 02:46:38.041648 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:38.041625    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-07b2d4\"/\"kube-root-ca.crt\""
Sep 09 02:46:38.041894 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:38.041673    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-07b2d4\"/\"autores-model-07b2d4-kserve-self-signed-certs\""
Sep 09 02:46:38.041894 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:38.041625    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-07b2d4\"/\"openshift-service-ca.crt\""
Sep 09 02:46:38.047125 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:38.047103    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-07b2d4/autores-model-07b2d4-kserve-58b9c97855-mdwg4"]
Sep 09 02:46:38.164708 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:38.164671    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a6f0048-279c-4a78-ab65-290c6c8beece-tls-certs\") pod \"autores-model-07b2d4-kserve-58b9c97855-mdwg4\" (UID: \"7a6f0048-279c-4a78-ab65-290c6c8beece\") " pod="ai-tenant-e2e-autores-07b2d4/autores-model-07b2d4-kserve-58b9c97855-mdwg4"
Sep 09 02:46:38.164708 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:38.164712    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a6f0048-279c-4a78-ab65-290c6c8beece-tmp-dir\") pod \"autores-model-07b2d4-kserve-58b9c97855-mdwg4\" (UID: \"7a6f0048-279c-4a78-ab65-290c6c8beece\") " pod="ai-tenant-e2e-autores-07b2d4/autores-model-07b2d4-kserve-58b9c97855-mdwg4"
Sep 09 02:46:38.164922 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:38.164728    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a6f0048-279c-4a78-ab65-290c6c8beece-dshm\") pod \"autores-model-07b2d4-kserve-58b9c97855-mdwg4\" (UID: \"7a6f0048-279c-4a78-ab65-290c6c8beece\") " pod="ai-tenant-e2e-autores-07b2d4/autores-model-07b2d4-kserve-58b9c97855-mdwg4"
Sep 09 02:46:38.164922 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:38.164865    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a6f0048-279c-4a78-ab65-290c6c8beece-model-cache\") pod \"autores-model-07b2d4-kserve-58b9c97855-mdwg4\" (UID: \"7a6f0048-279c-4a78-ab65-290c6c8beece\") " pod="ai-tenant-e2e-autores-07b2d4/autores-model-07b2d4-kserve-58b9c97855-mdwg4"
Sep 09 02:46:38.165025 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:38.164963    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szxq7\" (UniqueName: \"kubernetes.io/projected/7a6f0048-279c-4a78-ab65-290c6c8beece-kube-api-access-szxq7\") pod \"autores-model-07b2d4-kserve-58b9c97855-mdwg4\" (UID: \"7a6f0048-279c-4a78-ab65-290c6c8beece\") " pod="ai-tenant-e2e-autores-07b2d4/autores-model-07b2d4-kserve-58b9c97855-mdwg4"
Sep 09 02:46:38.165066 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:38.165044    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a6f0048-279c-4a78-ab65-290c6c8beece-home\") pod \"autores-model-07b2d4-kserve-58b9c97855-mdwg4\" (UID: \"7a6f0048-279c-4a78-ab65-290c6c8beece\") " pod="ai-tenant-e2e-autores-07b2d4/autores-model-07b2d4-kserve-58b9c97855-mdwg4"
Sep 09 02:46:38.222945 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:38.222864    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0d61ced-695f-49b3-a86b-732843026f8b" path="/var/lib/kubelet/pods/f0d61ced-695f-49b3-a86b-732843026f8b/volumes"
Sep 09 02:46:38.265887 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:38.265856    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a6f0048-279c-4a78-ab65-290c6c8beece-model-cache\") pod \"autores-model-07b2d4-kserve-58b9c97855-mdwg4\" (UID: \"7a6f0048-279c-4a78-ab65-290c6c8beece\") " pod="ai-tenant-e2e-autores-07b2d4/autores-model-07b2d4-kserve-58b9c97855-mdwg4"
Sep 09 02:46:38.266321 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:38.265931    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szxq7\" (UniqueName: \"kubernetes.io/projected/7a6f0048-279c-4a78-ab65-290c6c8beece-kube-api-access-szxq7\") pod \"autores-model-07b2d4-kserve-58b9c97855-mdwg4\" (UID: \"7a6f0048-279c-4a78-ab65-290c6c8beece\") " pod="ai-tenant-e2e-autores-07b2d4/autores-model-07b2d4-kserve-58b9c97855-mdwg4"
Sep 09 02:46:38.266321 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:38.265965    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a6f0048-279c-4a78-ab65-290c6c8beece-home\") pod \"autores-model-07b2d4-kserve-58b9c97855-mdwg4\" (UID: \"7a6f0048-279c-4a78-ab65-290c6c8beece\") " pod="ai-tenant-e2e-autores-07b2d4/autores-model-07b2d4-kserve-58b9c97855-mdwg4"
Sep 09 02:46:38.266321 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:38.266011    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a6f0048-279c-4a78-ab65-290c6c8beece-tls-certs\") pod \"autores-model-07b2d4-kserve-58b9c97855-mdwg4\" (UID: \"7a6f0048-279c-4a78-ab65-290c6c8beece\") " pod="ai-tenant-e2e-autores-07b2d4/autores-model-07b2d4-kserve-58b9c97855-mdwg4"
Sep 09 02:46:38.266321 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:38.266035    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a6f0048-279c-4a78-ab65-290c6c8beece-tmp-dir\") pod \"autores-model-07b2d4-kserve-58b9c97855-mdwg4\" (UID: \"7a6f0048-279c-4a78-ab65-290c6c8beece\") " pod="ai-tenant-e2e-autores-07b2d4/autores-model-07b2d4-kserve-58b9c97855-mdwg4"
Sep 09 02:46:38.266321 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:38.266058    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a6f0048-279c-4a78-ab65-290c6c8beece-dshm\") pod \"autores-model-07b2d4-kserve-58b9c97855-mdwg4\" (UID: \"7a6f0048-279c-4a78-ab65-290c6c8beece\") " pod="ai-tenant-e2e-autores-07b2d4/autores-model-07b2d4-kserve-58b9c97855-mdwg4"
Sep 09 02:46:38.266570 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:38.266350    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a6f0048-279c-4a78-ab65-290c6c8beece-model-cache\") pod \"autores-model-07b2d4-kserve-58b9c97855-mdwg4\" (UID: \"7a6f0048-279c-4a78-ab65-290c6c8beece\") " pod="ai-tenant-e2e-autores-07b2d4/autores-model-07b2d4-kserve-58b9c97855-mdwg4"
Sep 09 02:46:38.266570 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:38.266411    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a6f0048-279c-4a78-ab65-290c6c8beece-tmp-dir\") pod \"autores-model-07b2d4-kserve-58b9c97855-mdwg4\" (UID: \"7a6f0048-279c-4a78-ab65-290c6c8beece\") " pod="ai-tenant-e2e-autores-07b2d4/autores-model-07b2d4-kserve-58b9c97855-mdwg4"
Sep 09 02:46:38.266570 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:38.266427    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a6f0048-279c-4a78-ab65-290c6c8beece-home\") pod \"autores-model-07b2d4-kserve-58b9c97855-mdwg4\" (UID: \"7a6f0048-279c-4a78-ab65-290c6c8beece\") " pod="ai-tenant-e2e-autores-07b2d4/autores-model-07b2d4-kserve-58b9c97855-mdwg4"
Sep 09 02:46:38.268354 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:38.268324    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a6f0048-279c-4a78-ab65-290c6c8beece-dshm\") pod \"autores-model-07b2d4-kserve-58b9c97855-mdwg4\" (UID: \"7a6f0048-279c-4a78-ab65-290c6c8beece\") " pod="ai-tenant-e2e-autores-07b2d4/autores-model-07b2d4-kserve-58b9c97855-mdwg4"
Sep 09 02:46:38.268681 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:38.268663    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a6f0048-279c-4a78-ab65-290c6c8beece-tls-certs\") pod \"autores-model-07b2d4-kserve-58b9c97855-mdwg4\" (UID: \"7a6f0048-279c-4a78-ab65-290c6c8beece\") " pod="ai-tenant-e2e-autores-07b2d4/autores-model-07b2d4-kserve-58b9c97855-mdwg4"
Sep 09 02:46:38.275968 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:38.275945    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-szxq7\" (UniqueName: \"kubernetes.io/projected/7a6f0048-279c-4a78-ab65-290c6c8beece-kube-api-access-szxq7\") pod \"autores-model-07b2d4-kserve-58b9c97855-mdwg4\" (UID: \"7a6f0048-279c-4a78-ab65-290c6c8beece\") " pod="ai-tenant-e2e-autores-07b2d4/autores-model-07b2d4-kserve-58b9c97855-mdwg4"
Sep 09 02:46:38.349936 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:38.349897    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-07b2d4/autores-model-07b2d4-kserve-58b9c97855-mdwg4"
Sep 09 02:46:38.528466 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:38.528380    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-b-80fff954-kserve-5cf6f4f767-pp45n"
Sep 09 02:46:38.622418 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:38.622383    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-07b2d4/autores-model-07b2d4-kserve-58b9c97855-mdwg4"]
Sep 09 02:46:38.624691 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:46:38.624664    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7a6f0048_279c_4a78_ab65_290c6c8beece.slice/crio-7de0a2cf65ae7f3c435a0d24256a783379f6faf9cafdc60cda6831378cce6958 WatchSource:0}: Error finding container 7de0a2cf65ae7f3c435a0d24256a783379f6faf9cafdc60cda6831378cce6958: Status 404 returned error can't find the container with id 7de0a2cf65ae7f3c435a0d24256a783379f6faf9cafdc60cda6831378cce6958
Sep 09 02:46:39.523612 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:39.523577    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-07b2d4/autores-model-07b2d4-kserve-58b9c97855-mdwg4" event={"ID":"7a6f0048-279c-4a78-ab65-290c6c8beece","Type":"ContainerStarted","Data":"5c6cec996a720694d3b708c75eb17d8cfd7fc8f84b7014b708deeb4850e06a78"}
Sep 09 02:46:39.523612 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:39.523619    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-07b2d4/autores-model-07b2d4-kserve-58b9c97855-mdwg4" event={"ID":"7a6f0048-279c-4a78-ab65-290c6c8beece","Type":"ContainerStarted","Data":"7de0a2cf65ae7f3c435a0d24256a783379f6faf9cafdc60cda6831378cce6958"}
Sep 09 02:46:39.541913 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:39.541855    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-07b2d4/autores-model-07b2d4-kserve-58b9c97855-mdwg4" podStartSLOduration=1.541837489 podStartE2EDuration="1.541837489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:46:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:46:39.541186701 +0000 UTC m=+981.795490338" watchObservedRunningTime="2026-09-09 02:46:39.541837489 +0000 UTC m=+981.796141125"
Sep 09 02:46:44.061317 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:44.061226    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-99qjf"]
Sep 09 02:46:48.350344 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:48.350303    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-07b2d4/autores-model-07b2d4-kserve-58b9c97855-mdwg4"
Sep 09 02:46:48.350952 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:48.350365    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-07b2d4/autores-model-07b2d4-kserve-58b9c97855-mdwg4"
Sep 09 02:46:48.374836 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:48.374803    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-07b2d4/autores-model-07b2d4-kserve-58b9c97855-mdwg4"
Sep 09 02:46:48.584046 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:48.584010    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-07b2d4/autores-model-07b2d4-kserve-58b9c97855-mdwg4"
Sep 09 02:46:54.604031 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:54.603999    2570 generic.go:356] "Generic (PLEG): container finished" podID="39b1c8a7-44a0-4a11-b079-f398788e7102" containerID="c056c3a08bf3b6aa65cc0bde42374cc5380191bf95e3427b82c051c12918b902" exitCode=0
Sep 09 02:46:54.604432 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:54.604073    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7e17db81-openshift-default-794f5f8566-cp7m4" event={"ID":"39b1c8a7-44a0-4a11-b079-f398788e7102","Type":"ContainerDied","Data":"c056c3a08bf3b6aa65cc0bde42374cc5380191bf95e3427b82c051c12918b902"}
Sep 09 02:46:54.604513 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:54.604496    2570 scope.go:117] "RemoveContainer" containerID="c056c3a08bf3b6aa65cc0bde42374cc5380191bf95e3427b82c051c12918b902"
Sep 09 02:46:54.605593 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:54.605563    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 02:46:54.605659 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:54.605634    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 02:46:54.605701 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:54.605687    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 02:46:55.610905 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:55.610872    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7e17db81-openshift-default-794f5f8566-cp7m4" event={"ID":"39b1c8a7-44a0-4a11-b079-f398788e7102","Type":"ContainerStarted","Data":"6c8d98002f0c73d57ceff35024cbac68b984c32886c375896fd6ff1aab69ee96"}
Sep 09 02:46:56.044595 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:56.044499    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-7e17db81-openshift-default-794f5f8566-cp7m4"
Sep 09 02:46:56.045040 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:56.044893    2570 patch_prober.go:28] interesting pod/e2e-ait-7e17db81-openshift-default-794f5f8566-cp7m4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Sep 09 02:46:56.045040 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:56.044961    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7e17db81-openshift-default-794f5f8566-cp7m4" podUID="39b1c8a7-44a0-4a11-b079-f398788e7102" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Sep 09 02:46:57.044049 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:57.044004    2570 patch_prober.go:28] interesting pod/e2e-ait-7e17db81-openshift-default-794f5f8566-cp7m4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Sep 09 02:46:57.044538 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:57.044080    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7e17db81-openshift-default-794f5f8566-cp7m4" podUID="39b1c8a7-44a0-4a11-b079-f398788e7102" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Sep 09 02:46:58.046735 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:58.046682    2570 patch_prober.go:28] interesting pod/e2e-ait-7e17db81-openshift-default-794f5f8566-cp7m4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Sep 09 02:46:58.047264 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:58.046759    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7e17db81-openshift-default-794f5f8566-cp7m4" podUID="39b1c8a7-44a0-4a11-b079-f398788e7102" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Sep 09 02:46:59.044952 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:59.044907    2570 patch_prober.go:28] interesting pod/e2e-ait-7e17db81-openshift-default-794f5f8566-cp7m4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Sep 09 02:46:59.045286 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:46:59.045252    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7e17db81-openshift-default-794f5f8566-cp7m4" podUID="39b1c8a7-44a0-4a11-b079-f398788e7102" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Sep 09 02:47:00.045002 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:00.044942    2570 patch_prober.go:28] interesting pod/e2e-ait-7e17db81-openshift-default-794f5f8566-cp7m4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Sep 09 02:47:00.045541 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:00.045036    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7e17db81-openshift-default-794f5f8566-cp7m4" podUID="39b1c8a7-44a0-4a11-b079-f398788e7102" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Sep 09 02:47:00.543417 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:00.543380    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7e17db81-sjwng"]
Sep 09 02:47:00.547102 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:00.547081    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7e17db81-sjwng"
Sep 09 02:47:00.549847 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:00.549828    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 02:47:00.550232 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:00.550209    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vbr52\""
Sep 09 02:47:00.557822 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:00.557794    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7e17db81-sjwng"]
Sep 09 02:47:00.608663 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:00.608615    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e7fc16b-4714-4a2e-94f1-f280172d1cc8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7e17db81-sjwng\" (UID: \"8e7fc16b-4714-4a2e-94f1-f280172d1cc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7e17db81-sjwng"
Sep 09 02:47:00.709553 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:00.709515    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e7fc16b-4714-4a2e-94f1-f280172d1cc8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7e17db81-sjwng\" (UID: \"8e7fc16b-4714-4a2e-94f1-f280172d1cc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7e17db81-sjwng"
Sep 09 02:47:00.710191 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:00.710170    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e7fc16b-4714-4a2e-94f1-f280172d1cc8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7e17db81-sjwng\" (UID: \"8e7fc16b-4714-4a2e-94f1-f280172d1cc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7e17db81-sjwng"
Sep 09 02:47:00.860037 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:00.859995    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7e17db81-sjwng"
Sep 09 02:47:01.048731 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:01.048693    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-7e17db81-openshift-default-794f5f8566-cp7m4"
Sep 09 02:47:01.049223 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:01.049090    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-7e17db81-openshift-default-794f5f8566-cp7m4"
Sep 09 02:47:01.049674 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:01.049654    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-7e17db81-openshift-default-794f5f8566-cp7m4"
Sep 09 02:47:01.134429 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:01.134394    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7e17db81-sjwng"]
Sep 09 02:47:01.136793 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:47:01.136758    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e7fc16b_4714_4a2e_94f1_f280172d1cc8.slice/crio-c29fdeecdb9f82d68ba851b2a096eed80b0b0cf1e9ecccbfd3bd6cd16a0338ce WatchSource:0}: Error finding container c29fdeecdb9f82d68ba851b2a096eed80b0b0cf1e9ecccbfd3bd6cd16a0338ce: Status 404 returned error can't find the container with id c29fdeecdb9f82d68ba851b2a096eed80b0b0cf1e9ecccbfd3bd6cd16a0338ce
Sep 09 02:47:01.643463 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:01.643427    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7e17db81-sjwng" event={"ID":"8e7fc16b-4714-4a2e-94f1-f280172d1cc8","Type":"ContainerStarted","Data":"6a68316c8a574353b77e90cc8c8181986a6ae2b8ef18d9af507a19b305a7afa8"}
Sep 09 02:47:01.643675 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:01.643482    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7e17db81-sjwng" event={"ID":"8e7fc16b-4714-4a2e-94f1-f280172d1cc8","Type":"ContainerStarted","Data":"c29fdeecdb9f82d68ba851b2a096eed80b0b0cf1e9ecccbfd3bd6cd16a0338ce"}
Sep 09 02:47:01.662242 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:01.662185    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7e17db81-sjwng" podStartSLOduration=1.662168117 podStartE2EDuration="1.662168117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:47:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:47:01.661152625 +0000 UTC m=+1003.915456261" watchObservedRunningTime="2026-09-09 02:47:01.662168117 +0000 UTC m=+1003.916471753"
Sep 09 02:47:02.650151 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:02.650110    2570 generic.go:356] "Generic (PLEG): container finished" podID="8e7fc16b-4714-4a2e-94f1-f280172d1cc8" containerID="6a68316c8a574353b77e90cc8c8181986a6ae2b8ef18d9af507a19b305a7afa8" exitCode=0
Sep 09 02:47:02.650531 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:02.650200    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7e17db81-sjwng" event={"ID":"8e7fc16b-4714-4a2e-94f1-f280172d1cc8","Type":"ContainerDied","Data":"6a68316c8a574353b77e90cc8c8181986a6ae2b8ef18d9af507a19b305a7afa8"}
Sep 09 02:47:03.443652 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:03.443615    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-99qjf"]
Sep 09 02:47:03.807955 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:03.807929    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7e17db81-sjwng"
Sep 09 02:47:03.941772 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:03.941739    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e7fc16b-4714-4a2e-94f1-f280172d1cc8-maas-api-service-ca\") pod \"8e7fc16b-4714-4a2e-94f1-f280172d1cc8\" (UID: \"8e7fc16b-4714-4a2e-94f1-f280172d1cc8\") "
Sep 09 02:47:03.942130 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:03.942104    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e7fc16b-4714-4a2e-94f1-f280172d1cc8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8e7fc16b-4714-4a2e-94f1-f280172d1cc8" (UID: "8e7fc16b-4714-4a2e-94f1-f280172d1cc8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 02:47:04.042807 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:04.042697    2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e7fc16b-4714-4a2e-94f1-f280172d1cc8-maas-api-service-ca\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:47:04.669353 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:04.669318    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7e17db81-sjwng"
Sep 09 02:47:04.669532 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:04.669345    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7e17db81-sjwng" event={"ID":"8e7fc16b-4714-4a2e-94f1-f280172d1cc8","Type":"ContainerDied","Data":"c29fdeecdb9f82d68ba851b2a096eed80b0b0cf1e9ecccbfd3bd6cd16a0338ce"}
Sep 09 02:47:04.669532 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:04.669396    2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c29fdeecdb9f82d68ba851b2a096eed80b0b0cf1e9ecccbfd3bd6cd16a0338ce"
Sep 09 02:47:05.648563 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:05.648527    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-b18109/test-model-b18109-kserve-fd9cff9c9-7pjg2"]
Sep 09 02:47:05.648941 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:05.648893    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e7fc16b-4714-4a2e-94f1-f280172d1cc8" containerName="revoke-keys"
Sep 09 02:47:05.648941 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:05.648904    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e7fc16b-4714-4a2e-94f1-f280172d1cc8" containerName="revoke-keys"
Sep 09 02:47:05.649097 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:05.649004    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e7fc16b-4714-4a2e-94f1-f280172d1cc8" containerName="revoke-keys"
Sep 09 02:47:05.652150 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:05.652135    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-b18109/test-model-b18109-kserve-fd9cff9c9-7pjg2"
Sep 09 02:47:05.656093 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:05.656071    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-b18109\"/\"kube-root-ca.crt\""
Sep 09 02:47:05.656208 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:05.656104    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-b18109\"/\"test-model-b18109-kserve-self-signed-certs\""
Sep 09 02:47:05.656208 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:05.656123    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-b18109\"/\"default-dockercfg-8fg99\""
Sep 09 02:47:05.656208 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:05.656184    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-b18109\"/\"openshift-service-ca.crt\""
Sep 09 02:47:05.662605 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:05.662577    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-b18109/test-model-b18109-kserve-fd9cff9c9-7pjg2"]
Sep 09 02:47:05.759962 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:05.759928    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec0b7347-6013-4976-8a01-977307fd97bc-tmp-dir\") pod \"test-model-b18109-kserve-fd9cff9c9-7pjg2\" (UID: \"ec0b7347-6013-4976-8a01-977307fd97bc\") " pod="ai-tenant-e2e-inf-a-b18109/test-model-b18109-kserve-fd9cff9c9-7pjg2"
Sep 09 02:47:05.760153 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:05.760000    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec0b7347-6013-4976-8a01-977307fd97bc-home\") pod \"test-model-b18109-kserve-fd9cff9c9-7pjg2\" (UID: \"ec0b7347-6013-4976-8a01-977307fd97bc\") " pod="ai-tenant-e2e-inf-a-b18109/test-model-b18109-kserve-fd9cff9c9-7pjg2"
Sep 09 02:47:05.760153 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:05.760087    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec0b7347-6013-4976-8a01-977307fd97bc-model-cache\") pod \"test-model-b18109-kserve-fd9cff9c9-7pjg2\" (UID: \"ec0b7347-6013-4976-8a01-977307fd97bc\") " pod="ai-tenant-e2e-inf-a-b18109/test-model-b18109-kserve-fd9cff9c9-7pjg2"
Sep 09 02:47:05.760153 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:05.760125    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2rqp\" (UniqueName: \"kubernetes.io/projected/ec0b7347-6013-4976-8a01-977307fd97bc-kube-api-access-t2rqp\") pod \"test-model-b18109-kserve-fd9cff9c9-7pjg2\" (UID: \"ec0b7347-6013-4976-8a01-977307fd97bc\") " pod="ai-tenant-e2e-inf-a-b18109/test-model-b18109-kserve-fd9cff9c9-7pjg2"
Sep 09 02:47:05.760260 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:05.760194    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec0b7347-6013-4976-8a01-977307fd97bc-dshm\") pod \"test-model-b18109-kserve-fd9cff9c9-7pjg2\" (UID: \"ec0b7347-6013-4976-8a01-977307fd97bc\") " pod="ai-tenant-e2e-inf-a-b18109/test-model-b18109-kserve-fd9cff9c9-7pjg2"
Sep 09 02:47:05.760260 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:05.760248    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec0b7347-6013-4976-8a01-977307fd97bc-tls-certs\") pod \"test-model-b18109-kserve-fd9cff9c9-7pjg2\" (UID: \"ec0b7347-6013-4976-8a01-977307fd97bc\") " pod="ai-tenant-e2e-inf-a-b18109/test-model-b18109-kserve-fd9cff9c9-7pjg2"
Sep 09 02:47:05.861052 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:05.861016    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec0b7347-6013-4976-8a01-977307fd97bc-home\") pod \"test-model-b18109-kserve-fd9cff9c9-7pjg2\" (UID: \"ec0b7347-6013-4976-8a01-977307fd97bc\") " pod="ai-tenant-e2e-inf-a-b18109/test-model-b18109-kserve-fd9cff9c9-7pjg2"
Sep 09 02:47:05.861052 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:05.861069    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec0b7347-6013-4976-8a01-977307fd97bc-model-cache\") pod \"test-model-b18109-kserve-fd9cff9c9-7pjg2\" (UID: \"ec0b7347-6013-4976-8a01-977307fd97bc\") " pod="ai-tenant-e2e-inf-a-b18109/test-model-b18109-kserve-fd9cff9c9-7pjg2"
Sep 09 02:47:05.861296 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:05.861091    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2rqp\" (UniqueName: \"kubernetes.io/projected/ec0b7347-6013-4976-8a01-977307fd97bc-kube-api-access-t2rqp\") pod \"test-model-b18109-kserve-fd9cff9c9-7pjg2\" (UID: \"ec0b7347-6013-4976-8a01-977307fd97bc\") " pod="ai-tenant-e2e-inf-a-b18109/test-model-b18109-kserve-fd9cff9c9-7pjg2"
Sep 09 02:47:05.861296 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:05.861122    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec0b7347-6013-4976-8a01-977307fd97bc-dshm\") pod \"test-model-b18109-kserve-fd9cff9c9-7pjg2\" (UID: \"ec0b7347-6013-4976-8a01-977307fd97bc\") " pod="ai-tenant-e2e-inf-a-b18109/test-model-b18109-kserve-fd9cff9c9-7pjg2"
Sep 09 02:47:05.861296 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:05.861160    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec0b7347-6013-4976-8a01-977307fd97bc-tls-certs\") pod \"test-model-b18109-kserve-fd9cff9c9-7pjg2\" (UID: \"ec0b7347-6013-4976-8a01-977307fd97bc\") " pod="ai-tenant-e2e-inf-a-b18109/test-model-b18109-kserve-fd9cff9c9-7pjg2"
Sep 09 02:47:05.861296 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:05.861208    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec0b7347-6013-4976-8a01-977307fd97bc-tmp-dir\") pod \"test-model-b18109-kserve-fd9cff9c9-7pjg2\" (UID: \"ec0b7347-6013-4976-8a01-977307fd97bc\") " pod="ai-tenant-e2e-inf-a-b18109/test-model-b18109-kserve-fd9cff9c9-7pjg2"
Sep 09 02:47:05.861487 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:05.861449    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec0b7347-6013-4976-8a01-977307fd97bc-model-cache\") pod \"test-model-b18109-kserve-fd9cff9c9-7pjg2\" (UID: \"ec0b7347-6013-4976-8a01-977307fd97bc\") " pod="ai-tenant-e2e-inf-a-b18109/test-model-b18109-kserve-fd9cff9c9-7pjg2"
Sep 09 02:47:05.861547 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:05.861485    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec0b7347-6013-4976-8a01-977307fd97bc-home\") pod \"test-model-b18109-kserve-fd9cff9c9-7pjg2\" (UID: \"ec0b7347-6013-4976-8a01-977307fd97bc\") " pod="ai-tenant-e2e-inf-a-b18109/test-model-b18109-kserve-fd9cff9c9-7pjg2"
Sep 09 02:47:05.861547 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:05.861530    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec0b7347-6013-4976-8a01-977307fd97bc-tmp-dir\") pod \"test-model-b18109-kserve-fd9cff9c9-7pjg2\" (UID: \"ec0b7347-6013-4976-8a01-977307fd97bc\") " pod="ai-tenant-e2e-inf-a-b18109/test-model-b18109-kserve-fd9cff9c9-7pjg2"
Sep 09 02:47:05.863423 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:05.863404    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec0b7347-6013-4976-8a01-977307fd97bc-dshm\") pod \"test-model-b18109-kserve-fd9cff9c9-7pjg2\" (UID: \"ec0b7347-6013-4976-8a01-977307fd97bc\") " pod="ai-tenant-e2e-inf-a-b18109/test-model-b18109-kserve-fd9cff9c9-7pjg2"
Sep 09 02:47:05.863719 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:05.863699    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec0b7347-6013-4976-8a01-977307fd97bc-tls-certs\") pod \"test-model-b18109-kserve-fd9cff9c9-7pjg2\" (UID: \"ec0b7347-6013-4976-8a01-977307fd97bc\") " pod="ai-tenant-e2e-inf-a-b18109/test-model-b18109-kserve-fd9cff9c9-7pjg2"
Sep 09 02:47:05.869374 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:05.869352    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2rqp\" (UniqueName: \"kubernetes.io/projected/ec0b7347-6013-4976-8a01-977307fd97bc-kube-api-access-t2rqp\") pod \"test-model-b18109-kserve-fd9cff9c9-7pjg2\" (UID: \"ec0b7347-6013-4976-8a01-977307fd97bc\") " pod="ai-tenant-e2e-inf-a-b18109/test-model-b18109-kserve-fd9cff9c9-7pjg2"
Sep 09 02:47:05.962010 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:05.961884    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-b18109/test-model-b18109-kserve-fd9cff9c9-7pjg2"
Sep 09 02:47:06.241162 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:06.241136    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-b18109/test-model-b18109-kserve-fd9cff9c9-7pjg2"]
Sep 09 02:47:06.243352 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:47:06.243322    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podec0b7347_6013_4976_8a01_977307fd97bc.slice/crio-27726124a3207ea8def64bdbd2428a43e4ecbbc56be905c9fc8368699903a0cd WatchSource:0}: Error finding container 27726124a3207ea8def64bdbd2428a43e4ecbbc56be905c9fc8368699903a0cd: Status 404 returned error can't find the container with id 27726124a3207ea8def64bdbd2428a43e4ecbbc56be905c9fc8368699903a0cd
Sep 09 02:47:06.681363 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:06.681327    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-b18109/test-model-b18109-kserve-fd9cff9c9-7pjg2" event={"ID":"ec0b7347-6013-4976-8a01-977307fd97bc","Type":"ContainerStarted","Data":"79189f8a8fb61d31e57ec7ae6d14d3d56cd91d2e53e5f0b20bc75b719e9af153"}
Sep 09 02:47:06.681768 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:06.681369    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-b18109/test-model-b18109-kserve-fd9cff9c9-7pjg2" event={"ID":"ec0b7347-6013-4976-8a01-977307fd97bc","Type":"ContainerStarted","Data":"27726124a3207ea8def64bdbd2428a43e4ecbbc56be905c9fc8368699903a0cd"}
Sep 09 02:47:06.701090 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:06.701041    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-b18109/test-model-b18109-kserve-fd9cff9c9-7pjg2" podStartSLOduration=1.7010220010000001 podStartE2EDuration="1.701022001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:47:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:47:06.699637851 +0000 UTC m=+1008.953941500" watchObservedRunningTime="2026-09-09 02:47:06.701022001 +0000 UTC m=+1008.955325636"
Sep 09 02:47:07.143772 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:07.143734    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7e17db81-sjwng"]
Sep 09 02:47:07.148710 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:07.148681    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7e17db81-sjwng"]
Sep 09 02:47:08.222951 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:08.222917    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e7fc16b-4714-4a2e-94f1-f280172d1cc8" path="/var/lib/kubelet/pods/8e7fc16b-4714-4a2e-94f1-f280172d1cc8/volumes"
Sep 09 02:47:08.274739 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:08.274702    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-7e17db81-openshift-default-794f5f8566-cp7m4"]
Sep 09 02:47:08.275000 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:08.274954    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-7e17db81-openshift-default-794f5f8566-cp7m4" podUID="39b1c8a7-44a0-4a11-b079-f398788e7102" containerName="istio-proxy" containerID="cri-o://6c8d98002f0c73d57ceff35024cbac68b984c32886c375896fd6ff1aab69ee96" gracePeriod=30
Sep 09 02:47:09.328236 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:09.328199    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-80fff954-kserve-5d5dd55697-nrzf6"]
Sep 09 02:47:12.739479 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:12.739443    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-80fff954-kserve-5cf6f4f767-pp45n"]
Sep 09 02:47:13.527570 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:13.527545    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7e17db81-openshift-default-794f5f8566-cp7m4"
Sep 09 02:47:13.639964 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:13.639925    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39b1c8a7-44a0-4a11-b079-f398788e7102-workload-socket\") pod \"39b1c8a7-44a0-4a11-b079-f398788e7102\" (UID: \"39b1c8a7-44a0-4a11-b079-f398788e7102\") "
Sep 09 02:47:13.640193 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:13.640095    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39b1c8a7-44a0-4a11-b079-f398788e7102-workload-certs\") pod \"39b1c8a7-44a0-4a11-b079-f398788e7102\" (UID: \"39b1c8a7-44a0-4a11-b079-f398788e7102\") "
Sep 09 02:47:13.640193 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:13.640129    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39b1c8a7-44a0-4a11-b079-f398788e7102-credential-socket\") pod \"39b1c8a7-44a0-4a11-b079-f398788e7102\" (UID: \"39b1c8a7-44a0-4a11-b079-f398788e7102\") "
Sep 09 02:47:13.640193 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:13.640164    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tlr8p\" (UniqueName: \"kubernetes.io/projected/39b1c8a7-44a0-4a11-b079-f398788e7102-kube-api-access-tlr8p\") pod \"39b1c8a7-44a0-4a11-b079-f398788e7102\" (UID: \"39b1c8a7-44a0-4a11-b079-f398788e7102\") "
Sep 09 02:47:13.640369 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:13.640219    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39b1c8a7-44a0-4a11-b079-f398788e7102-istiod-ca-cert\") pod \"39b1c8a7-44a0-4a11-b079-f398788e7102\" (UID: \"39b1c8a7-44a0-4a11-b079-f398788e7102\") "
Sep 09 02:47:13.640369 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:13.640259    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39b1c8a7-44a0-4a11-b079-f398788e7102-istio-data\") pod \"39b1c8a7-44a0-4a11-b079-f398788e7102\" (UID: \"39b1c8a7-44a0-4a11-b079-f398788e7102\") "
Sep 09 02:47:13.640369 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:13.640279    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39b1c8a7-44a0-4a11-b079-f398788e7102-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "39b1c8a7-44a0-4a11-b079-f398788e7102" (UID: "39b1c8a7-44a0-4a11-b079-f398788e7102"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 02:47:13.640369 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:13.640304    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39b1c8a7-44a0-4a11-b079-f398788e7102-istio-token\") pod \"39b1c8a7-44a0-4a11-b079-f398788e7102\" (UID: \"39b1c8a7-44a0-4a11-b079-f398788e7102\") "
Sep 09 02:47:13.640369 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:13.640339    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39b1c8a7-44a0-4a11-b079-f398788e7102-istio-envoy\") pod \"39b1c8a7-44a0-4a11-b079-f398788e7102\" (UID: \"39b1c8a7-44a0-4a11-b079-f398788e7102\") "
Sep 09 02:47:13.640630 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:13.640370    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39b1c8a7-44a0-4a11-b079-f398788e7102-istio-podinfo\") pod \"39b1c8a7-44a0-4a11-b079-f398788e7102\" (UID: \"39b1c8a7-44a0-4a11-b079-f398788e7102\") "
Sep 09 02:47:13.640630 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:13.640417    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39b1c8a7-44a0-4a11-b079-f398788e7102-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "39b1c8a7-44a0-4a11-b079-f398788e7102" (UID: "39b1c8a7-44a0-4a11-b079-f398788e7102"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 02:47:13.640739 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:13.640625    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39b1c8a7-44a0-4a11-b079-f398788e7102-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "39b1c8a7-44a0-4a11-b079-f398788e7102" (UID: "39b1c8a7-44a0-4a11-b079-f398788e7102"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 02:47:13.640739 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:13.640724    2570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39b1c8a7-44a0-4a11-b079-f398788e7102-workload-certs\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:47:13.640855 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:13.640732    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39b1c8a7-44a0-4a11-b079-f398788e7102-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "39b1c8a7-44a0-4a11-b079-f398788e7102" (UID: "39b1c8a7-44a0-4a11-b079-f398788e7102"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 02:47:13.640855 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:13.640746    2570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39b1c8a7-44a0-4a11-b079-f398788e7102-istiod-ca-cert\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:47:13.640855 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:13.640761    2570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39b1c8a7-44a0-4a11-b079-f398788e7102-workload-socket\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:47:13.641124 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:13.641026    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39b1c8a7-44a0-4a11-b079-f398788e7102-istio-data" (OuterVolumeSpecName: "istio-data") pod "39b1c8a7-44a0-4a11-b079-f398788e7102" (UID: "39b1c8a7-44a0-4a11-b079-f398788e7102"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 02:47:13.643304 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:13.643264    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39b1c8a7-44a0-4a11-b079-f398788e7102-istio-token" (OuterVolumeSpecName: "istio-token") pod "39b1c8a7-44a0-4a11-b079-f398788e7102" (UID: "39b1c8a7-44a0-4a11-b079-f398788e7102"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 02:47:13.643410 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:13.643389    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39b1c8a7-44a0-4a11-b079-f398788e7102-kube-api-access-tlr8p" (OuterVolumeSpecName: "kube-api-access-tlr8p") pod "39b1c8a7-44a0-4a11-b079-f398788e7102" (UID: "39b1c8a7-44a0-4a11-b079-f398788e7102"). InnerVolumeSpecName "kube-api-access-tlr8p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 02:47:13.643588 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:13.643562    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/39b1c8a7-44a0-4a11-b079-f398788e7102-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "39b1c8a7-44a0-4a11-b079-f398788e7102" (UID: "39b1c8a7-44a0-4a11-b079-f398788e7102"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 02:47:13.643698 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:13.643608    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39b1c8a7-44a0-4a11-b079-f398788e7102-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "39b1c8a7-44a0-4a11-b079-f398788e7102" (UID: "39b1c8a7-44a0-4a11-b079-f398788e7102"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 02:47:13.733860 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:13.733815    2570 generic.go:356] "Generic (PLEG): container finished" podID="39b1c8a7-44a0-4a11-b079-f398788e7102" containerID="6c8d98002f0c73d57ceff35024cbac68b984c32886c375896fd6ff1aab69ee96" exitCode=0
Sep 09 02:47:13.734042 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:13.733893    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7e17db81-openshift-default-794f5f8566-cp7m4"
Sep 09 02:47:13.734042 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:13.733896    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7e17db81-openshift-default-794f5f8566-cp7m4" event={"ID":"39b1c8a7-44a0-4a11-b079-f398788e7102","Type":"ContainerDied","Data":"6c8d98002f0c73d57ceff35024cbac68b984c32886c375896fd6ff1aab69ee96"}
Sep 09 02:47:13.734181 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:13.734044    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7e17db81-openshift-default-794f5f8566-cp7m4" event={"ID":"39b1c8a7-44a0-4a11-b079-f398788e7102","Type":"ContainerDied","Data":"95a54d26c3a3482bcfedc71b9d21e86aa01ccb79c15656ca3ee660dce9e2f3fb"}
Sep 09 02:47:13.734181 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:13.734084    2570 scope.go:117] "RemoveContainer" containerID="6c8d98002f0c73d57ceff35024cbac68b984c32886c375896fd6ff1aab69ee96"
Sep 09 02:47:13.741994 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:13.741927    2570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39b1c8a7-44a0-4a11-b079-f398788e7102-credential-socket\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:47:13.741994 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:13.741961    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlr8p\" (UniqueName: \"kubernetes.io/projected/39b1c8a7-44a0-4a11-b079-f398788e7102-kube-api-access-tlr8p\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:47:13.741994 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:13.741993    2570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39b1c8a7-44a0-4a11-b079-f398788e7102-istio-data\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:47:13.742530 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:13.742007    2570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39b1c8a7-44a0-4a11-b079-f398788e7102-istio-token\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:47:13.742530 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:13.742020    2570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39b1c8a7-44a0-4a11-b079-f398788e7102-istio-envoy\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:47:13.742530 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:13.742032    2570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39b1c8a7-44a0-4a11-b079-f398788e7102-istio-podinfo\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:47:13.747596 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:13.747571    2570 scope.go:117] "RemoveContainer" containerID="c056c3a08bf3b6aa65cc0bde42374cc5380191bf95e3427b82c051c12918b902"
Sep 09 02:47:13.758281 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:13.758257    2570 scope.go:117] "RemoveContainer" containerID="6c8d98002f0c73d57ceff35024cbac68b984c32886c375896fd6ff1aab69ee96"
Sep 09 02:47:13.758605 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:47:13.758578    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c8d98002f0c73d57ceff35024cbac68b984c32886c375896fd6ff1aab69ee96\": container with ID starting with 6c8d98002f0c73d57ceff35024cbac68b984c32886c375896fd6ff1aab69ee96 not found: ID does not exist" containerID="6c8d98002f0c73d57ceff35024cbac68b984c32886c375896fd6ff1aab69ee96"
Sep 09 02:47:13.758704 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:13.758614    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c8d98002f0c73d57ceff35024cbac68b984c32886c375896fd6ff1aab69ee96"} err="failed to get container status \"6c8d98002f0c73d57ceff35024cbac68b984c32886c375896fd6ff1aab69ee96\": rpc error: code = NotFound desc = could not find container \"6c8d98002f0c73d57ceff35024cbac68b984c32886c375896fd6ff1aab69ee96\": container with ID starting with 6c8d98002f0c73d57ceff35024cbac68b984c32886c375896fd6ff1aab69ee96 not found: ID does not exist"
Sep 09 02:47:13.758704 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:13.758639    2570 scope.go:117] "RemoveContainer" containerID="c056c3a08bf3b6aa65cc0bde42374cc5380191bf95e3427b82c051c12918b902"
Sep 09 02:47:13.758893 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:47:13.758873    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c056c3a08bf3b6aa65cc0bde42374cc5380191bf95e3427b82c051c12918b902\": container with ID starting with c056c3a08bf3b6aa65cc0bde42374cc5380191bf95e3427b82c051c12918b902 not found: ID does not exist" containerID="c056c3a08bf3b6aa65cc0bde42374cc5380191bf95e3427b82c051c12918b902"
Sep 09 02:47:13.758968 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:13.758899    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c056c3a08bf3b6aa65cc0bde42374cc5380191bf95e3427b82c051c12918b902"} err="failed to get container status \"c056c3a08bf3b6aa65cc0bde42374cc5380191bf95e3427b82c051c12918b902\": rpc error: code = NotFound desc = could not find container \"c056c3a08bf3b6aa65cc0bde42374cc5380191bf95e3427b82c051c12918b902\": container with ID starting with c056c3a08bf3b6aa65cc0bde42374cc5380191bf95e3427b82c051c12918b902 not found: ID does not exist"
Sep 09 02:47:13.765057 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:13.765030    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-7e17db81-openshift-default-794f5f8566-cp7m4"]
Sep 09 02:47:13.770416 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:13.770383    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-7e17db81-openshift-default-794f5f8566-cp7m4"]
Sep 09 02:47:14.226574 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:14.226532    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39b1c8a7-44a0-4a11-b079-f398788e7102" path="/var/lib/kubelet/pods/39b1c8a7-44a0-4a11-b079-f398788e7102/volumes"
Sep 09 02:47:14.935438 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:14.935401    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-07b2d4/autores-model-07b2d4-kserve-58b9c97855-mdwg4"]
Sep 09 02:47:15.962520 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:15.962480    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-b18109/test-model-b18109-kserve-fd9cff9c9-7pjg2"
Sep 09 02:47:15.962905 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:15.962535    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-b18109/test-model-b18109-kserve-fd9cff9c9-7pjg2"
Sep 09 02:47:15.975149 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:15.975122    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-b18109/test-model-b18109-kserve-fd9cff9c9-7pjg2"
Sep 09 02:47:16.762221 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:16.762186    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-b18109/test-model-b18109-kserve-fd9cff9c9-7pjg2"
Sep 09 02:47:22.457744 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:22.457706    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-99qjf"]
Sep 09 02:47:24.348555 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:24.348512    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-a-80fff954-kserve-5d5dd55697-nrzf6" podUID="fcf320b2-3158-4fef-8335-a7f311712377" containerName="main" containerID="cri-o://e6816cde2d273a26e55398c3bcc3cc4a4dd58e1b173c612f67617a00b0208b5f" gracePeriod=45
Sep 09 02:47:24.618440 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:24.618411    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-80fff954-kserve-5d5dd55697-nrzf6"
Sep 09 02:47:24.657448 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:24.657412    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fcf320b2-3158-4fef-8335-a7f311712377-tls-certs\") pod \"fcf320b2-3158-4fef-8335-a7f311712377\" (UID: \"fcf320b2-3158-4fef-8335-a7f311712377\") "
Sep 09 02:47:24.657638 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:24.657547    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fcf320b2-3158-4fef-8335-a7f311712377-tmp-dir\") pod \"fcf320b2-3158-4fef-8335-a7f311712377\" (UID: \"fcf320b2-3158-4fef-8335-a7f311712377\") "
Sep 09 02:47:24.657638 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:24.657601    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fcf320b2-3158-4fef-8335-a7f311712377-home\") pod \"fcf320b2-3158-4fef-8335-a7f311712377\" (UID: \"fcf320b2-3158-4fef-8335-a7f311712377\") "
Sep 09 02:47:24.657638 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:24.657628    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fcf320b2-3158-4fef-8335-a7f311712377-model-cache\") pod \"fcf320b2-3158-4fef-8335-a7f311712377\" (UID: \"fcf320b2-3158-4fef-8335-a7f311712377\") "
Sep 09 02:47:24.657812 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:24.657653    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fcf320b2-3158-4fef-8335-a7f311712377-dshm\") pod \"fcf320b2-3158-4fef-8335-a7f311712377\" (UID: \"fcf320b2-3158-4fef-8335-a7f311712377\") "
Sep 09 02:47:24.657812 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:24.657678    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8fhsv\" (UniqueName: \"kubernetes.io/projected/fcf320b2-3158-4fef-8335-a7f311712377-kube-api-access-8fhsv\") pod \"fcf320b2-3158-4fef-8335-a7f311712377\" (UID: \"fcf320b2-3158-4fef-8335-a7f311712377\") "
Sep 09 02:47:24.658517 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:24.658064    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcf320b2-3158-4fef-8335-a7f311712377-home" (OuterVolumeSpecName: "home") pod "fcf320b2-3158-4fef-8335-a7f311712377" (UID: "fcf320b2-3158-4fef-8335-a7f311712377"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 02:47:24.658517 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:24.658287    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcf320b2-3158-4fef-8335-a7f311712377-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fcf320b2-3158-4fef-8335-a7f311712377" (UID: "fcf320b2-3158-4fef-8335-a7f311712377"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 02:47:24.658517 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:24.658484    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcf320b2-3158-4fef-8335-a7f311712377-model-cache" (OuterVolumeSpecName: "model-cache") pod "fcf320b2-3158-4fef-8335-a7f311712377" (UID: "fcf320b2-3158-4fef-8335-a7f311712377"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 02:47:24.660877 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:24.660849    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcf320b2-3158-4fef-8335-a7f311712377-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fcf320b2-3158-4fef-8335-a7f311712377" (UID: "fcf320b2-3158-4fef-8335-a7f311712377"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 02:47:24.683406 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:24.683372    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcf320b2-3158-4fef-8335-a7f311712377-dshm" (OuterVolumeSpecName: "dshm") pod "fcf320b2-3158-4fef-8335-a7f311712377" (UID: "fcf320b2-3158-4fef-8335-a7f311712377"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 02:47:24.683519 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:24.683436    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcf320b2-3158-4fef-8335-a7f311712377-kube-api-access-8fhsv" (OuterVolumeSpecName: "kube-api-access-8fhsv") pod "fcf320b2-3158-4fef-8335-a7f311712377" (UID: "fcf320b2-3158-4fef-8335-a7f311712377"). InnerVolumeSpecName "kube-api-access-8fhsv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 02:47:24.759309 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:24.759274    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fcf320b2-3158-4fef-8335-a7f311712377-tls-certs\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:47:24.759309 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:24.759307    2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fcf320b2-3158-4fef-8335-a7f311712377-tmp-dir\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:47:24.759518 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:24.759317    2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fcf320b2-3158-4fef-8335-a7f311712377-home\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:47:24.759518 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:24.759332    2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fcf320b2-3158-4fef-8335-a7f311712377-model-cache\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:47:24.759518 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:24.759343    2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fcf320b2-3158-4fef-8335-a7f311712377-dshm\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:47:24.759518 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:24.759354    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fhsv\" (UniqueName: \"kubernetes.io/projected/fcf320b2-3158-4fef-8335-a7f311712377-kube-api-access-8fhsv\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:47:24.795888 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:24.795853    2570 generic.go:356] "Generic (PLEG): container finished" podID="fcf320b2-3158-4fef-8335-a7f311712377" containerID="e6816cde2d273a26e55398c3bcc3cc4a4dd58e1b173c612f67617a00b0208b5f" exitCode=0
Sep 09 02:47:24.796088 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:24.795897    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-80fff954-kserve-5d5dd55697-nrzf6" event={"ID":"fcf320b2-3158-4fef-8335-a7f311712377","Type":"ContainerDied","Data":"e6816cde2d273a26e55398c3bcc3cc4a4dd58e1b173c612f67617a00b0208b5f"}
Sep 09 02:47:24.796088 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:24.795922    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-80fff954-kserve-5d5dd55697-nrzf6"
Sep 09 02:47:24.796088 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:24.795946    2570 scope.go:117] "RemoveContainer" containerID="e6816cde2d273a26e55398c3bcc3cc4a4dd58e1b173c612f67617a00b0208b5f"
Sep 09 02:47:24.796088 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:24.795933    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-80fff954-kserve-5d5dd55697-nrzf6" event={"ID":"fcf320b2-3158-4fef-8335-a7f311712377","Type":"ContainerDied","Data":"6dc402244a57b36b9027c7c499d1b0a936e1b2aae8e7d47892347f1f23e991ce"}
Sep 09 02:47:24.806528 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:24.806349    2570 scope.go:117] "RemoveContainer" containerID="e6816cde2d273a26e55398c3bcc3cc4a4dd58e1b173c612f67617a00b0208b5f"
Sep 09 02:47:24.806753 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:47:24.806730    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6816cde2d273a26e55398c3bcc3cc4a4dd58e1b173c612f67617a00b0208b5f\": container with ID starting with e6816cde2d273a26e55398c3bcc3cc4a4dd58e1b173c612f67617a00b0208b5f not found: ID does not exist" containerID="e6816cde2d273a26e55398c3bcc3cc4a4dd58e1b173c612f67617a00b0208b5f"
Sep 09 02:47:24.806828 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:24.806763    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6816cde2d273a26e55398c3bcc3cc4a4dd58e1b173c612f67617a00b0208b5f"} err="failed to get container status \"e6816cde2d273a26e55398c3bcc3cc4a4dd58e1b173c612f67617a00b0208b5f\": rpc error: code = NotFound desc = could not find container \"e6816cde2d273a26e55398c3bcc3cc4a4dd58e1b173c612f67617a00b0208b5f\": container with ID starting with e6816cde2d273a26e55398c3bcc3cc4a4dd58e1b173c612f67617a00b0208b5f not found: ID does not exist"
Sep 09 02:47:24.822087 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:24.822064    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-80fff954-kserve-5d5dd55697-nrzf6"]
Sep 09 02:47:24.828170 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:24.828150    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-a-80fff954-kserve-5d5dd55697-nrzf6"]
Sep 09 02:47:26.223463 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:26.223421    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fcf320b2-3158-4fef-8335-a7f311712377" path="/var/lib/kubelet/pods/fcf320b2-3158-4fef-8335-a7f311712377/volumes"
Sep 09 02:47:27.760165 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:27.760106    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-b-80fff954-kserve-5cf6f4f767-pp45n" podUID="f8a5bc9c-eb60-4978-88a4-ebf506f37ad2" containerName="main" containerID="cri-o://9d96181c007e5c13ec1fd18ee600c03ceeaad99ebb2369b64bebf908e1eb90ee" gracePeriod=45
Sep 09 02:47:28.016851 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:28.016796    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-80fff954-kserve-5cf6f4f767-pp45n"
Sep 09 02:47:28.094659 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:28.094624    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8a5bc9c-eb60-4978-88a4-ebf506f37ad2-model-cache\") pod \"f8a5bc9c-eb60-4978-88a4-ebf506f37ad2\" (UID: \"f8a5bc9c-eb60-4978-88a4-ebf506f37ad2\") "
Sep 09 02:47:28.094659 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:28.094655    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8a5bc9c-eb60-4978-88a4-ebf506f37ad2-home\") pod \"f8a5bc9c-eb60-4978-88a4-ebf506f37ad2\" (UID: \"f8a5bc9c-eb60-4978-88a4-ebf506f37ad2\") "
Sep 09 02:47:28.094898 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:28.094690    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d685t\" (UniqueName: \"kubernetes.io/projected/f8a5bc9c-eb60-4978-88a4-ebf506f37ad2-kube-api-access-d685t\") pod \"f8a5bc9c-eb60-4978-88a4-ebf506f37ad2\" (UID: \"f8a5bc9c-eb60-4978-88a4-ebf506f37ad2\") "
Sep 09 02:47:28.094898 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:28.094744    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8a5bc9c-eb60-4978-88a4-ebf506f37ad2-dshm\") pod \"f8a5bc9c-eb60-4978-88a4-ebf506f37ad2\" (UID: \"f8a5bc9c-eb60-4978-88a4-ebf506f37ad2\") "
Sep 09 02:47:28.094898 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:28.094783    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8a5bc9c-eb60-4978-88a4-ebf506f37ad2-tls-certs\") pod \"f8a5bc9c-eb60-4978-88a4-ebf506f37ad2\" (UID: \"f8a5bc9c-eb60-4978-88a4-ebf506f37ad2\") "
Sep 09 02:47:28.094898 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:28.094865    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8a5bc9c-eb60-4978-88a4-ebf506f37ad2-tmp-dir\") pod \"f8a5bc9c-eb60-4978-88a4-ebf506f37ad2\" (UID: \"f8a5bc9c-eb60-4978-88a4-ebf506f37ad2\") "
Sep 09 02:47:28.095152 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:28.094954    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8a5bc9c-eb60-4978-88a4-ebf506f37ad2-model-cache" (OuterVolumeSpecName: "model-cache") pod "f8a5bc9c-eb60-4978-88a4-ebf506f37ad2" (UID: "f8a5bc9c-eb60-4978-88a4-ebf506f37ad2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 02:47:28.095152 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:28.094961    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8a5bc9c-eb60-4978-88a4-ebf506f37ad2-home" (OuterVolumeSpecName: "home") pod "f8a5bc9c-eb60-4978-88a4-ebf506f37ad2" (UID: "f8a5bc9c-eb60-4978-88a4-ebf506f37ad2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 02:47:28.095241 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:28.095198    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8a5bc9c-eb60-4978-88a4-ebf506f37ad2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f8a5bc9c-eb60-4978-88a4-ebf506f37ad2" (UID: "f8a5bc9c-eb60-4978-88a4-ebf506f37ad2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 02:47:28.095278 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:28.095259    2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8a5bc9c-eb60-4978-88a4-ebf506f37ad2-model-cache\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:47:28.095318 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:28.095278    2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8a5bc9c-eb60-4978-88a4-ebf506f37ad2-home\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:47:28.096965 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:28.096940    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8a5bc9c-eb60-4978-88a4-ebf506f37ad2-kube-api-access-d685t" (OuterVolumeSpecName: "kube-api-access-d685t") pod "f8a5bc9c-eb60-4978-88a4-ebf506f37ad2" (UID: "f8a5bc9c-eb60-4978-88a4-ebf506f37ad2"). InnerVolumeSpecName "kube-api-access-d685t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 02:47:28.096965 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:28.096947    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8a5bc9c-eb60-4978-88a4-ebf506f37ad2-dshm" (OuterVolumeSpecName: "dshm") pod "f8a5bc9c-eb60-4978-88a4-ebf506f37ad2" (UID: "f8a5bc9c-eb60-4978-88a4-ebf506f37ad2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 02:47:28.097092 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:28.097043    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8a5bc9c-eb60-4978-88a4-ebf506f37ad2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f8a5bc9c-eb60-4978-88a4-ebf506f37ad2" (UID: "f8a5bc9c-eb60-4978-88a4-ebf506f37ad2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 02:47:28.196086 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:28.196049    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d685t\" (UniqueName: \"kubernetes.io/projected/f8a5bc9c-eb60-4978-88a4-ebf506f37ad2-kube-api-access-d685t\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:47:28.196086 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:28.196081    2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8a5bc9c-eb60-4978-88a4-ebf506f37ad2-dshm\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:47:28.196086 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:28.196091    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8a5bc9c-eb60-4978-88a4-ebf506f37ad2-tls-certs\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:47:28.196317 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:28.196099    2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8a5bc9c-eb60-4978-88a4-ebf506f37ad2-tmp-dir\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:47:28.817664 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:28.817625    2570 generic.go:356] "Generic (PLEG): container finished" podID="f8a5bc9c-eb60-4978-88a4-ebf506f37ad2" containerID="9d96181c007e5c13ec1fd18ee600c03ceeaad99ebb2369b64bebf908e1eb90ee" exitCode=0
Sep 09 02:47:28.818109 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:28.817671    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-80fff954-kserve-5cf6f4f767-pp45n"
Sep 09 02:47:28.818109 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:28.817709    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-80fff954-kserve-5cf6f4f767-pp45n" event={"ID":"f8a5bc9c-eb60-4978-88a4-ebf506f37ad2","Type":"ContainerDied","Data":"9d96181c007e5c13ec1fd18ee600c03ceeaad99ebb2369b64bebf908e1eb90ee"}
Sep 09 02:47:28.818109 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:28.817758    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-80fff954-kserve-5cf6f4f767-pp45n" event={"ID":"f8a5bc9c-eb60-4978-88a4-ebf506f37ad2","Type":"ContainerDied","Data":"bc91d543c06c09c626698192ae08ed4ac0a7303b40630c651c839cfb91bb1d2f"}
Sep 09 02:47:28.818109 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:28.817773    2570 scope.go:117] "RemoveContainer" containerID="9d96181c007e5c13ec1fd18ee600c03ceeaad99ebb2369b64bebf908e1eb90ee"
Sep 09 02:47:28.826212 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:28.826190    2570 scope.go:117] "RemoveContainer" containerID="9d96181c007e5c13ec1fd18ee600c03ceeaad99ebb2369b64bebf908e1eb90ee"
Sep 09 02:47:28.826480 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:47:28.826461    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d96181c007e5c13ec1fd18ee600c03ceeaad99ebb2369b64bebf908e1eb90ee\": container with ID starting with 9d96181c007e5c13ec1fd18ee600c03ceeaad99ebb2369b64bebf908e1eb90ee not found: ID does not exist" containerID="9d96181c007e5c13ec1fd18ee600c03ceeaad99ebb2369b64bebf908e1eb90ee"
Sep 09 02:47:28.826551 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:28.826494    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d96181c007e5c13ec1fd18ee600c03ceeaad99ebb2369b64bebf908e1eb90ee"} err="failed to get container status \"9d96181c007e5c13ec1fd18ee600c03ceeaad99ebb2369b64bebf908e1eb90ee\": rpc error: code = NotFound desc = could not find container \"9d96181c007e5c13ec1fd18ee600c03ceeaad99ebb2369b64bebf908e1eb90ee\": container with ID starting with 9d96181c007e5c13ec1fd18ee600c03ceeaad99ebb2369b64bebf908e1eb90ee not found: ID does not exist"
Sep 09 02:47:28.835846 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:28.835819    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-80fff954-kserve-5cf6f4f767-pp45n"]
Sep 09 02:47:28.839655 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:28.839633    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-b-80fff954-kserve-5cf6f4f767-pp45n"]
Sep 09 02:47:29.955390 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:29.955351    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-07b2d4/autores-model-07b2d4-kserve-58b9c97855-mdwg4" podUID="7a6f0048-279c-4a78-ab65-290c6c8beece" containerName="main" containerID="cri-o://5c6cec996a720694d3b708c75eb17d8cfd7fc8f84b7014b708deeb4850e06a78" gracePeriod=45
Sep 09 02:47:30.208204 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:30.208140    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-07b2d4/autores-model-07b2d4-kserve-58b9c97855-mdwg4"
Sep 09 02:47:30.224636 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:30.224599    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8a5bc9c-eb60-4978-88a4-ebf506f37ad2" path="/var/lib/kubelet/pods/f8a5bc9c-eb60-4978-88a4-ebf506f37ad2/volumes"
Sep 09 02:47:30.316411 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:30.316379    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a6f0048-279c-4a78-ab65-290c6c8beece-tmp-dir\") pod \"7a6f0048-279c-4a78-ab65-290c6c8beece\" (UID: \"7a6f0048-279c-4a78-ab65-290c6c8beece\") "
Sep 09 02:47:30.316605 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:30.316475    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a6f0048-279c-4a78-ab65-290c6c8beece-dshm\") pod \"7a6f0048-279c-4a78-ab65-290c6c8beece\" (UID: \"7a6f0048-279c-4a78-ab65-290c6c8beece\") "
Sep 09 02:47:30.316605 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:30.316509    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a6f0048-279c-4a78-ab65-290c6c8beece-home\") pod \"7a6f0048-279c-4a78-ab65-290c6c8beece\" (UID: \"7a6f0048-279c-4a78-ab65-290c6c8beece\") "
Sep 09 02:47:30.316605 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:30.316534    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-szxq7\" (UniqueName: \"kubernetes.io/projected/7a6f0048-279c-4a78-ab65-290c6c8beece-kube-api-access-szxq7\") pod \"7a6f0048-279c-4a78-ab65-290c6c8beece\" (UID: \"7a6f0048-279c-4a78-ab65-290c6c8beece\") "
Sep 09 02:47:30.316777 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:30.316617    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a6f0048-279c-4a78-ab65-290c6c8beece-model-cache\") pod \"7a6f0048-279c-4a78-ab65-290c6c8beece\" (UID: \"7a6f0048-279c-4a78-ab65-290c6c8beece\") "
Sep 09 02:47:30.316777 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:30.316650    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a6f0048-279c-4a78-ab65-290c6c8beece-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7a6f0048-279c-4a78-ab65-290c6c8beece" (UID: "7a6f0048-279c-4a78-ab65-290c6c8beece"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 02:47:30.316777 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:30.316710    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a6f0048-279c-4a78-ab65-290c6c8beece-tls-certs\") pod \"7a6f0048-279c-4a78-ab65-290c6c8beece\" (UID: \"7a6f0048-279c-4a78-ab65-290c6c8beece\") "
Sep 09 02:47:30.316940 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:30.316853    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a6f0048-279c-4a78-ab65-290c6c8beece-model-cache" (OuterVolumeSpecName: "model-cache") pod "7a6f0048-279c-4a78-ab65-290c6c8beece" (UID: "7a6f0048-279c-4a78-ab65-290c6c8beece"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 02:47:30.317192 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:30.317121    2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a6f0048-279c-4a78-ab65-290c6c8beece-model-cache\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:47:30.317371 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:30.317355    2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a6f0048-279c-4a78-ab65-290c6c8beece-tmp-dir\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:47:30.317522 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:30.317134    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a6f0048-279c-4a78-ab65-290c6c8beece-home" (OuterVolumeSpecName: "home") pod "7a6f0048-279c-4a78-ab65-290c6c8beece" (UID: "7a6f0048-279c-4a78-ab65-290c6c8beece"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 02:47:30.319221 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:30.319194    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a6f0048-279c-4a78-ab65-290c6c8beece-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7a6f0048-279c-4a78-ab65-290c6c8beece" (UID: "7a6f0048-279c-4a78-ab65-290c6c8beece"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 02:47:30.319375 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:30.319230    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a6f0048-279c-4a78-ab65-290c6c8beece-dshm" (OuterVolumeSpecName: "dshm") pod "7a6f0048-279c-4a78-ab65-290c6c8beece" (UID: "7a6f0048-279c-4a78-ab65-290c6c8beece"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 02:47:30.319375 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:30.319229    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a6f0048-279c-4a78-ab65-290c6c8beece-kube-api-access-szxq7" (OuterVolumeSpecName: "kube-api-access-szxq7") pod "7a6f0048-279c-4a78-ab65-290c6c8beece" (UID: "7a6f0048-279c-4a78-ab65-290c6c8beece"). InnerVolumeSpecName "kube-api-access-szxq7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 02:47:30.419026 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:30.418960    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a6f0048-279c-4a78-ab65-290c6c8beece-tls-certs\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:47:30.419026 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:30.419021    2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a6f0048-279c-4a78-ab65-290c6c8beece-dshm\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:47:30.419026 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:30.419030    2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a6f0048-279c-4a78-ab65-290c6c8beece-home\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:47:30.419259 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:30.419039    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szxq7\" (UniqueName: \"kubernetes.io/projected/7a6f0048-279c-4a78-ab65-290c6c8beece-kube-api-access-szxq7\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:47:30.829991 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:30.829949    2570 generic.go:356] "Generic (PLEG): container finished" podID="7a6f0048-279c-4a78-ab65-290c6c8beece" containerID="5c6cec996a720694d3b708c75eb17d8cfd7fc8f84b7014b708deeb4850e06a78" exitCode=0
Sep 09 02:47:30.830166 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:30.829999    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-07b2d4/autores-model-07b2d4-kserve-58b9c97855-mdwg4"
Sep 09 02:47:30.830166 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:30.830035    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-07b2d4/autores-model-07b2d4-kserve-58b9c97855-mdwg4" event={"ID":"7a6f0048-279c-4a78-ab65-290c6c8beece","Type":"ContainerDied","Data":"5c6cec996a720694d3b708c75eb17d8cfd7fc8f84b7014b708deeb4850e06a78"}
Sep 09 02:47:30.830166 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:30.830079    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-07b2d4/autores-model-07b2d4-kserve-58b9c97855-mdwg4" event={"ID":"7a6f0048-279c-4a78-ab65-290c6c8beece","Type":"ContainerDied","Data":"7de0a2cf65ae7f3c435a0d24256a783379f6faf9cafdc60cda6831378cce6958"}
Sep 09 02:47:30.830166 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:30.830096    2570 scope.go:117] "RemoveContainer" containerID="5c6cec996a720694d3b708c75eb17d8cfd7fc8f84b7014b708deeb4850e06a78"
Sep 09 02:47:30.838797 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:30.838780    2570 scope.go:117] "RemoveContainer" containerID="5c6cec996a720694d3b708c75eb17d8cfd7fc8f84b7014b708deeb4850e06a78"
Sep 09 02:47:30.839072 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:47:30.839049    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c6cec996a720694d3b708c75eb17d8cfd7fc8f84b7014b708deeb4850e06a78\": container with ID starting with 5c6cec996a720694d3b708c75eb17d8cfd7fc8f84b7014b708deeb4850e06a78 not found: ID does not exist" containerID="5c6cec996a720694d3b708c75eb17d8cfd7fc8f84b7014b708deeb4850e06a78"
Sep 09 02:47:30.839150 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:30.839078    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c6cec996a720694d3b708c75eb17d8cfd7fc8f84b7014b708deeb4850e06a78"} err="failed to get container status \"5c6cec996a720694d3b708c75eb17d8cfd7fc8f84b7014b708deeb4850e06a78\": rpc error: code = NotFound desc = could not find container \"5c6cec996a720694d3b708c75eb17d8cfd7fc8f84b7014b708deeb4850e06a78\": container with ID starting with 5c6cec996a720694d3b708c75eb17d8cfd7fc8f84b7014b708deeb4850e06a78 not found: ID does not exist"
Sep 09 02:47:30.854098 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:30.854076    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-07b2d4/autores-model-07b2d4-kserve-58b9c97855-mdwg4"]
Sep 09 02:47:30.858374 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:30.858352    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-07b2d4/autores-model-07b2d4-kserve-58b9c97855-mdwg4"]
Sep 09 02:47:32.223732 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:32.223695    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a6f0048-279c-4a78-ab65-290c6c8beece" path="/var/lib/kubelet/pods/7a6f0048-279c-4a78-ab65-290c6c8beece/volumes"
Sep 09 02:47:36.684765 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:36.684728    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-b52336-openshift-default-7dd744d476-nd9gp"]
Sep 09 02:47:36.685346 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:36.685326    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a6f0048-279c-4a78-ab65-290c6c8beece" containerName="main"
Sep 09 02:47:36.685403 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:36.685350    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a6f0048-279c-4a78-ab65-290c6c8beece" containerName="main"
Sep 09 02:47:36.685403 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:36.685367    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcf320b2-3158-4fef-8335-a7f311712377" containerName="main"
Sep 09 02:47:36.685403 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:36.685376    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcf320b2-3158-4fef-8335-a7f311712377" containerName="main"
Sep 09 02:47:36.685403 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:36.685398    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39b1c8a7-44a0-4a11-b079-f398788e7102" containerName="istio-proxy"
Sep 09 02:47:36.685531 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:36.685406    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="39b1c8a7-44a0-4a11-b079-f398788e7102" containerName="istio-proxy"
Sep 09 02:47:36.685531 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:36.685423    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8a5bc9c-eb60-4978-88a4-ebf506f37ad2" containerName="main"
Sep 09 02:47:36.685531 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:36.685430    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8a5bc9c-eb60-4978-88a4-ebf506f37ad2" containerName="main"
Sep 09 02:47:36.685531 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:36.685439    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39b1c8a7-44a0-4a11-b079-f398788e7102" containerName="istio-proxy"
Sep 09 02:47:36.685531 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:36.685447    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="39b1c8a7-44a0-4a11-b079-f398788e7102" containerName="istio-proxy"
Sep 09 02:47:36.685531 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:36.685523    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="39b1c8a7-44a0-4a11-b079-f398788e7102" containerName="istio-proxy"
Sep 09 02:47:36.685703 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:36.685532    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a6f0048-279c-4a78-ab65-290c6c8beece" containerName="main"
Sep 09 02:47:36.685703 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:36.685545    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8a5bc9c-eb60-4978-88a4-ebf506f37ad2" containerName="main"
Sep 09 02:47:36.685703 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:36.685556    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="39b1c8a7-44a0-4a11-b079-f398788e7102" containerName="istio-proxy"
Sep 09 02:47:36.685703 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:36.685566    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="fcf320b2-3158-4fef-8335-a7f311712377" containerName="main"
Sep 09 02:47:36.690514 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:36.690485    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-b52336-openshift-default-7dd744d476-nd9gp"
Sep 09 02:47:36.693536 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:36.693515    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-b52336-openshift-default-dockercfg-55qbt\""
Sep 09 02:47:36.704647 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:36.704611    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-b52336-openshift-default-7dd744d476-nd9gp"]
Sep 09 02:47:36.729555 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:36.729529    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1816a546-hq6l8"]
Sep 09 02:47:36.732732 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:36.732707    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1816a546-hq6l8"
Sep 09 02:47:36.735615 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:36.735598    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vbr52\""
Sep 09 02:47:36.735699 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:36.735627    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 02:47:36.741429 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:36.741405    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1816a546-hq6l8"]
Sep 09 02:47:36.781317 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:36.781260    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mz6l\" (UniqueName: \"kubernetes.io/projected/48c61b4e-c170-4cbc-916a-efc2d6964697-kube-api-access-7mz6l\") pod \"e2e-explicit-b52336-openshift-default-7dd744d476-nd9gp\" (UID: \"48c61b4e-c170-4cbc-916a-efc2d6964697\") " pod="openshift-ingress/e2e-explicit-b52336-openshift-default-7dd744d476-nd9gp"
Sep 09 02:47:36.782076 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:36.781326    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48c61b4e-c170-4cbc-916a-efc2d6964697-istio-envoy\") pod \"e2e-explicit-b52336-openshift-default-7dd744d476-nd9gp\" (UID: \"48c61b4e-c170-4cbc-916a-efc2d6964697\") " pod="openshift-ingress/e2e-explicit-b52336-openshift-default-7dd744d476-nd9gp"
Sep 09 02:47:36.782076 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:36.781402    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c1fc464-6971-4177-b871-6077a2bfef2c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-1816a546-hq6l8\" (UID: \"1c1fc464-6971-4177-b871-6077a2bfef2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1816a546-hq6l8"
Sep 09 02:47:36.782076 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:36.781617    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48c61b4e-c170-4cbc-916a-efc2d6964697-istio-podinfo\") pod \"e2e-explicit-b52336-openshift-default-7dd744d476-nd9gp\" (UID: \"48c61b4e-c170-4cbc-916a-efc2d6964697\") " pod="openshift-ingress/e2e-explicit-b52336-openshift-default-7dd744d476-nd9gp"
Sep 09 02:47:36.782076 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:36.781765    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48c61b4e-c170-4cbc-916a-efc2d6964697-workload-certs\") pod \"e2e-explicit-b52336-openshift-default-7dd744d476-nd9gp\" (UID: \"48c61b4e-c170-4cbc-916a-efc2d6964697\") " pod="openshift-ingress/e2e-explicit-b52336-openshift-default-7dd744d476-nd9gp"
Sep 09 02:47:36.782076 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:36.781848    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48c61b4e-c170-4cbc-916a-efc2d6964697-istiod-ca-cert\") pod \"e2e-explicit-b52336-openshift-default-7dd744d476-nd9gp\" (UID: \"48c61b4e-c170-4cbc-916a-efc2d6964697\") " pod="openshift-ingress/e2e-explicit-b52336-openshift-default-7dd744d476-nd9gp"
Sep 09 02:47:36.782076 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:36.781961    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48c61b4e-c170-4cbc-916a-efc2d6964697-credential-socket\") pod \"e2e-explicit-b52336-openshift-default-7dd744d476-nd9gp\" (UID: \"48c61b4e-c170-4cbc-916a-efc2d6964697\") " pod="openshift-ingress/e2e-explicit-b52336-openshift-default-7dd744d476-nd9gp"
Sep 09 02:47:36.782076 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:36.782009    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48c61b4e-c170-4cbc-916a-efc2d6964697-istio-data\") pod \"e2e-explicit-b52336-openshift-default-7dd744d476-nd9gp\" (UID: \"48c61b4e-c170-4cbc-916a-efc2d6964697\") " pod="openshift-ingress/e2e-explicit-b52336-openshift-default-7dd744d476-nd9gp"
Sep 09 02:47:36.782541 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:36.782093    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48c61b4e-c170-4cbc-916a-efc2d6964697-workload-socket\") pod \"e2e-explicit-b52336-openshift-default-7dd744d476-nd9gp\" (UID: \"48c61b4e-c170-4cbc-916a-efc2d6964697\") " pod="openshift-ingress/e2e-explicit-b52336-openshift-default-7dd744d476-nd9gp"
Sep 09 02:47:36.782541 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:36.782122    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48c61b4e-c170-4cbc-916a-efc2d6964697-istio-token\") pod \"e2e-explicit-b52336-openshift-default-7dd744d476-nd9gp\" (UID: \"48c61b4e-c170-4cbc-916a-efc2d6964697\") " pod="openshift-ingress/e2e-explicit-b52336-openshift-default-7dd744d476-nd9gp"
Sep 09 02:47:36.883418 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:36.883384    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48c61b4e-c170-4cbc-916a-efc2d6964697-istio-podinfo\") pod \"e2e-explicit-b52336-openshift-default-7dd744d476-nd9gp\" (UID: \"48c61b4e-c170-4cbc-916a-efc2d6964697\") " pod="openshift-ingress/e2e-explicit-b52336-openshift-default-7dd744d476-nd9gp"
Sep 09 02:47:36.883418 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:36.883425    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48c61b4e-c170-4cbc-916a-efc2d6964697-workload-certs\") pod \"e2e-explicit-b52336-openshift-default-7dd744d476-nd9gp\" (UID: \"48c61b4e-c170-4cbc-916a-efc2d6964697\") " pod="openshift-ingress/e2e-explicit-b52336-openshift-default-7dd744d476-nd9gp"
Sep 09 02:47:36.883658 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:36.883444    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48c61b4e-c170-4cbc-916a-efc2d6964697-istiod-ca-cert\") pod \"e2e-explicit-b52336-openshift-default-7dd744d476-nd9gp\" (UID: \"48c61b4e-c170-4cbc-916a-efc2d6964697\") " pod="openshift-ingress/e2e-explicit-b52336-openshift-default-7dd744d476-nd9gp"
Sep 09 02:47:36.883658 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:36.883498    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48c61b4e-c170-4cbc-916a-efc2d6964697-credential-socket\") pod \"e2e-explicit-b52336-openshift-default-7dd744d476-nd9gp\" (UID: \"48c61b4e-c170-4cbc-916a-efc2d6964697\") " pod="openshift-ingress/e2e-explicit-b52336-openshift-default-7dd744d476-nd9gp"
Sep 09 02:47:36.883658 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:36.883514    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48c61b4e-c170-4cbc-916a-efc2d6964697-istio-data\") pod \"e2e-explicit-b52336-openshift-default-7dd744d476-nd9gp\" (UID: \"48c61b4e-c170-4cbc-916a-efc2d6964697\") " pod="openshift-ingress/e2e-explicit-b52336-openshift-default-7dd744d476-nd9gp"
Sep 09 02:47:36.883818 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:36.883686    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48c61b4e-c170-4cbc-916a-efc2d6964697-workload-socket\") pod \"e2e-explicit-b52336-openshift-default-7dd744d476-nd9gp\" (UID: \"48c61b4e-c170-4cbc-916a-efc2d6964697\") " pod="openshift-ingress/e2e-explicit-b52336-openshift-default-7dd744d476-nd9gp"
Sep 09 02:47:36.883818 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:36.883733    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48c61b4e-c170-4cbc-916a-efc2d6964697-istio-token\") pod \"e2e-explicit-b52336-openshift-default-7dd744d476-nd9gp\" (UID: \"48c61b4e-c170-4cbc-916a-efc2d6964697\") " pod="openshift-ingress/e2e-explicit-b52336-openshift-default-7dd744d476-nd9gp"
Sep 09 02:47:36.883818 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:36.883782    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mz6l\" (UniqueName: \"kubernetes.io/projected/48c61b4e-c170-4cbc-916a-efc2d6964697-kube-api-access-7mz6l\") pod \"e2e-explicit-b52336-openshift-default-7dd744d476-nd9gp\" (UID: \"48c61b4e-c170-4cbc-916a-efc2d6964697\") " pod="openshift-ingress/e2e-explicit-b52336-openshift-default-7dd744d476-nd9gp"
Sep 09 02:47:36.883818 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:36.883814    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48c61b4e-c170-4cbc-916a-efc2d6964697-istio-envoy\") pod \"e2e-explicit-b52336-openshift-default-7dd744d476-nd9gp\" (UID: \"48c61b4e-c170-4cbc-916a-efc2d6964697\") " pod="openshift-ingress/e2e-explicit-b52336-openshift-default-7dd744d476-nd9gp"
Sep 09 02:47:36.884070 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:36.883865    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c1fc464-6971-4177-b871-6077a2bfef2c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-1816a546-hq6l8\" (UID: \"1c1fc464-6971-4177-b871-6077a2bfef2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1816a546-hq6l8"
Sep 09 02:47:36.884070 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:36.883875    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48c61b4e-c170-4cbc-916a-efc2d6964697-istio-data\") pod \"e2e-explicit-b52336-openshift-default-7dd744d476-nd9gp\" (UID: \"48c61b4e-c170-4cbc-916a-efc2d6964697\") " pod="openshift-ingress/e2e-explicit-b52336-openshift-default-7dd744d476-nd9gp"
Sep 09 02:47:36.884181 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:36.884095    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48c61b4e-c170-4cbc-916a-efc2d6964697-credential-socket\") pod \"e2e-explicit-b52336-openshift-default-7dd744d476-nd9gp\" (UID: \"48c61b4e-c170-4cbc-916a-efc2d6964697\") " pod="openshift-ingress/e2e-explicit-b52336-openshift-default-7dd744d476-nd9gp"
Sep 09 02:47:36.884289 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:36.884269    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48c61b4e-c170-4cbc-916a-efc2d6964697-istiod-ca-cert\") pod \"e2e-explicit-b52336-openshift-default-7dd744d476-nd9gp\" (UID: \"48c61b4e-c170-4cbc-916a-efc2d6964697\") " pod="openshift-ingress/e2e-explicit-b52336-openshift-default-7dd744d476-nd9gp"
Sep 09 02:47:36.884358 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:36.884329    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48c61b4e-c170-4cbc-916a-efc2d6964697-workload-socket\") pod \"e2e-explicit-b52336-openshift-default-7dd744d476-nd9gp\" (UID: \"48c61b4e-c170-4cbc-916a-efc2d6964697\") " pod="openshift-ingress/e2e-explicit-b52336-openshift-default-7dd744d476-nd9gp"
Sep 09 02:47:36.884436 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:36.884420    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48c61b4e-c170-4cbc-916a-efc2d6964697-workload-certs\") pod \"e2e-explicit-b52336-openshift-default-7dd744d476-nd9gp\" (UID: \"48c61b4e-c170-4cbc-916a-efc2d6964697\") " pod="openshift-ingress/e2e-explicit-b52336-openshift-default-7dd744d476-nd9gp"
Sep 09 02:47:36.884626 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:36.884605    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c1fc464-6971-4177-b871-6077a2bfef2c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-1816a546-hq6l8\" (UID: \"1c1fc464-6971-4177-b871-6077a2bfef2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1816a546-hq6l8"
Sep 09 02:47:36.886072 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:36.886050    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48c61b4e-c170-4cbc-916a-efc2d6964697-istio-podinfo\") pod \"e2e-explicit-b52336-openshift-default-7dd744d476-nd9gp\" (UID: \"48c61b4e-c170-4cbc-916a-efc2d6964697\") " pod="openshift-ingress/e2e-explicit-b52336-openshift-default-7dd744d476-nd9gp"
Sep 09 02:47:36.886155 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:36.886131    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48c61b4e-c170-4cbc-916a-efc2d6964697-istio-envoy\") pod \"e2e-explicit-b52336-openshift-default-7dd744d476-nd9gp\" (UID: \"48c61b4e-c170-4cbc-916a-efc2d6964697\") " pod="openshift-ingress/e2e-explicit-b52336-openshift-default-7dd744d476-nd9gp"
Sep 09 02:47:36.892551 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:36.892530    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48c61b4e-c170-4cbc-916a-efc2d6964697-istio-token\") pod \"e2e-explicit-b52336-openshift-default-7dd744d476-nd9gp\" (UID: \"48c61b4e-c170-4cbc-916a-efc2d6964697\") " pod="openshift-ingress/e2e-explicit-b52336-openshift-default-7dd744d476-nd9gp"
Sep 09 02:47:36.892652 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:36.892632    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mz6l\" (UniqueName: \"kubernetes.io/projected/48c61b4e-c170-4cbc-916a-efc2d6964697-kube-api-access-7mz6l\") pod \"e2e-explicit-b52336-openshift-default-7dd744d476-nd9gp\" (UID: \"48c61b4e-c170-4cbc-916a-efc2d6964697\") " pod="openshift-ingress/e2e-explicit-b52336-openshift-default-7dd744d476-nd9gp"
Sep 09 02:47:37.000672 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:37.000581    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-b52336-openshift-default-7dd744d476-nd9gp"
Sep 09 02:47:37.044128 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:37.044096    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1816a546-hq6l8"
Sep 09 02:47:37.280057 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:37.280030    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-b52336-openshift-default-7dd744d476-nd9gp"]
Sep 09 02:47:37.281890 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:47:37.281862    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48c61b4e_c170_4cbc_916a_efc2d6964697.slice/crio-599b1c2e45df5c85da61be6df04a5f9866da05bc93ca2904bc346802015dd8af WatchSource:0}: Error finding container 599b1c2e45df5c85da61be6df04a5f9866da05bc93ca2904bc346802015dd8af: Status 404 returned error can't find the container with id 599b1c2e45df5c85da61be6df04a5f9866da05bc93ca2904bc346802015dd8af
Sep 09 02:47:37.284166 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:37.284133    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 02:47:37.284265 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:37.284197    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 02:47:37.284265 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:37.284229    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 02:47:37.328534 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:37.328384    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1816a546-hq6l8"]
Sep 09 02:47:37.331220 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:47:37.331189    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c1fc464_6971_4177_b871_6077a2bfef2c.slice/crio-477eb3a91b2fa69b6bb176463a3aa60ef206ca73d20bec50e132b1af12a02d9c WatchSource:0}: Error finding container 477eb3a91b2fa69b6bb176463a3aa60ef206ca73d20bec50e132b1af12a02d9c: Status 404 returned error can't find the container with id 477eb3a91b2fa69b6bb176463a3aa60ef206ca73d20bec50e132b1af12a02d9c
Sep 09 02:47:37.869424 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:37.869385    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-b52336-openshift-default-7dd744d476-nd9gp" event={"ID":"48c61b4e-c170-4cbc-916a-efc2d6964697","Type":"ContainerStarted","Data":"a69c40d39853a81f1c0ec8e8040dc8cb9fd44afbfc5ee500838dcf31cbdfa61c"}
Sep 09 02:47:37.869888 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:37.869449    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-b52336-openshift-default-7dd744d476-nd9gp" event={"ID":"48c61b4e-c170-4cbc-916a-efc2d6964697","Type":"ContainerStarted","Data":"599b1c2e45df5c85da61be6df04a5f9866da05bc93ca2904bc346802015dd8af"}
Sep 09 02:47:37.871096 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:37.871067    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1816a546-hq6l8" event={"ID":"1c1fc464-6971-4177-b871-6077a2bfef2c","Type":"ContainerStarted","Data":"23d548a4e31e1d14e3681e5faafad0aa3bc08ec53733a6ac5833430c669c3708"}
Sep 09 02:47:37.871199 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:37.871124    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1816a546-hq6l8" event={"ID":"1c1fc464-6971-4177-b871-6077a2bfef2c","Type":"ContainerStarted","Data":"477eb3a91b2fa69b6bb176463a3aa60ef206ca73d20bec50e132b1af12a02d9c"}
Sep 09 02:47:37.894550 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:37.894496    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-b52336-openshift-default-7dd744d476-nd9gp" podStartSLOduration=1.894480884 podStartE2EDuration="1.894480884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:47:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:47:37.893842235 +0000 UTC m=+1040.148145872" watchObservedRunningTime="2026-09-09 02:47:37.894480884 +0000 UTC m=+1040.148784520"
Sep 09 02:47:37.913923 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:37.913866    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1816a546-hq6l8" podStartSLOduration=1.9138519120000002 podStartE2EDuration="1.913851912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:47:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:47:37.91250232 +0000 UTC m=+1040.166805957" watchObservedRunningTime="2026-09-09 02:47:37.913851912 +0000 UTC m=+1040.168155548"
Sep 09 02:47:38.001043 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:38.001010    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-b52336-openshift-default-7dd744d476-nd9gp"
Sep 09 02:47:38.011842 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:38.011806    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-b52336-openshift-default-7dd744d476-nd9gp"
Sep 09 02:47:38.877786 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:38.877751    2570 generic.go:356] "Generic (PLEG): container finished" podID="1c1fc464-6971-4177-b871-6077a2bfef2c" containerID="23d548a4e31e1d14e3681e5faafad0aa3bc08ec53733a6ac5833430c669c3708" exitCode=0
Sep 09 02:47:38.878257 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:38.877837    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1816a546-hq6l8" event={"ID":"1c1fc464-6971-4177-b871-6077a2bfef2c","Type":"ContainerDied","Data":"23d548a4e31e1d14e3681e5faafad0aa3bc08ec53733a6ac5833430c669c3708"}
Sep 09 02:47:38.878257 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:38.878200    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-b52336-openshift-default-7dd744d476-nd9gp"
Sep 09 02:47:38.879172 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:38.879154    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-b52336-openshift-default-7dd744d476-nd9gp"
Sep 09 02:47:39.344356 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:39.344324    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-99qjf"]
Sep 09 02:47:40.019516 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:40.019477    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1816a546-hq6l8"
Sep 09 02:47:40.116796 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:40.116748    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c1fc464-6971-4177-b871-6077a2bfef2c-maas-api-service-ca\") pod \"1c1fc464-6971-4177-b871-6077a2bfef2c\" (UID: \"1c1fc464-6971-4177-b871-6077a2bfef2c\") "
Sep 09 02:47:40.117196 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:40.117170    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c1fc464-6971-4177-b871-6077a2bfef2c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1c1fc464-6971-4177-b871-6077a2bfef2c" (UID: "1c1fc464-6971-4177-b871-6077a2bfef2c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 02:47:40.117293 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:40.117249    2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c1fc464-6971-4177-b871-6077a2bfef2c-maas-api-service-ca\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:47:40.890260 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:40.890225    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1816a546-hq6l8"
Sep 09 02:47:40.890425 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:40.890266    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1816a546-hq6l8" event={"ID":"1c1fc464-6971-4177-b871-6077a2bfef2c","Type":"ContainerDied","Data":"477eb3a91b2fa69b6bb176463a3aa60ef206ca73d20bec50e132b1af12a02d9c"}
Sep 09 02:47:40.890425 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:40.890316    2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="477eb3a91b2fa69b6bb176463a3aa60ef206ca73d20bec50e132b1af12a02d9c"
Sep 09 02:47:43.500794 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:43.500757    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1816a546-hq6l8"]
Sep 09 02:47:43.506476 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:43.506449    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1816a546-hq6l8"]
Sep 09 02:47:44.222876 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:44.222844    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c1fc464-6971-4177-b871-6077a2bfef2c" path="/var/lib/kubelet/pods/1c1fc464-6971-4177-b871-6077a2bfef2c/volumes"
Sep 09 02:47:50.643835 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:50.643788    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-99qjf"]
Sep 09 02:47:53.728876 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:53.728831    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-6b11a2/test-model-6b11a2-kserve-76b7fc9c9f-pk6mx"]
Sep 09 02:47:53.729450 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:53.729428    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c1fc464-6971-4177-b871-6077a2bfef2c" containerName="revoke-keys"
Sep 09 02:47:53.729543 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:53.729453    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c1fc464-6971-4177-b871-6077a2bfef2c" containerName="revoke-keys"
Sep 09 02:47:53.729619 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:53.729604    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c1fc464-6971-4177-b871-6077a2bfef2c" containerName="revoke-keys"
Sep 09 02:47:53.732696 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:53.732674    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-6b11a2/test-model-6b11a2-kserve-76b7fc9c9f-pk6mx"
Sep 09 02:47:53.735477 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:53.735450    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-6b11a2\"/\"openshift-service-ca.crt\""
Sep 09 02:47:53.735600 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:53.735526    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-6b11a2\"/\"default-dockercfg-stgg8\""
Sep 09 02:47:53.735600 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:53.735522    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-6b11a2\"/\"kube-root-ca.crt\""
Sep 09 02:47:53.736670 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:53.736649    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-6b11a2\"/\"test-model-6b11a2-kserve-self-signed-certs\""
Sep 09 02:47:53.742066 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:53.742039    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-6b11a2/test-model-6b11a2-kserve-76b7fc9c9f-pk6mx"]
Sep 09 02:47:53.853808 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:53.853753    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0447baeb-2218-40f2-9a4f-47102c50b8ed-tls-certs\") pod \"test-model-6b11a2-kserve-76b7fc9c9f-pk6mx\" (UID: \"0447baeb-2218-40f2-9a4f-47102c50b8ed\") " pod="ai-tenant-e2e-inf-b-6b11a2/test-model-6b11a2-kserve-76b7fc9c9f-pk6mx"
Sep 09 02:47:53.854013 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:53.853822    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0447baeb-2218-40f2-9a4f-47102c50b8ed-model-cache\") pod \"test-model-6b11a2-kserve-76b7fc9c9f-pk6mx\" (UID: \"0447baeb-2218-40f2-9a4f-47102c50b8ed\") " pod="ai-tenant-e2e-inf-b-6b11a2/test-model-6b11a2-kserve-76b7fc9c9f-pk6mx"
Sep 09 02:47:53.854013 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:53.853914    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0447baeb-2218-40f2-9a4f-47102c50b8ed-home\") pod \"test-model-6b11a2-kserve-76b7fc9c9f-pk6mx\" (UID: \"0447baeb-2218-40f2-9a4f-47102c50b8ed\") " pod="ai-tenant-e2e-inf-b-6b11a2/test-model-6b11a2-kserve-76b7fc9c9f-pk6mx"
Sep 09 02:47:53.854013 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:53.853985    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0447baeb-2218-40f2-9a4f-47102c50b8ed-dshm\") pod \"test-model-6b11a2-kserve-76b7fc9c9f-pk6mx\" (UID: \"0447baeb-2218-40f2-9a4f-47102c50b8ed\") " pod="ai-tenant-e2e-inf-b-6b11a2/test-model-6b11a2-kserve-76b7fc9c9f-pk6mx"
Sep 09 02:47:53.854163 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:53.854031    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0447baeb-2218-40f2-9a4f-47102c50b8ed-tmp-dir\") pod \"test-model-6b11a2-kserve-76b7fc9c9f-pk6mx\" (UID: \"0447baeb-2218-40f2-9a4f-47102c50b8ed\") " pod="ai-tenant-e2e-inf-b-6b11a2/test-model-6b11a2-kserve-76b7fc9c9f-pk6mx"
Sep 09 02:47:53.854163 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:53.854103    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndvrh\" (UniqueName: \"kubernetes.io/projected/0447baeb-2218-40f2-9a4f-47102c50b8ed-kube-api-access-ndvrh\") pod \"test-model-6b11a2-kserve-76b7fc9c9f-pk6mx\" (UID: \"0447baeb-2218-40f2-9a4f-47102c50b8ed\") " pod="ai-tenant-e2e-inf-b-6b11a2/test-model-6b11a2-kserve-76b7fc9c9f-pk6mx"
Sep 09 02:47:53.954733 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:53.954697    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndvrh\" (UniqueName: \"kubernetes.io/projected/0447baeb-2218-40f2-9a4f-47102c50b8ed-kube-api-access-ndvrh\") pod \"test-model-6b11a2-kserve-76b7fc9c9f-pk6mx\" (UID: \"0447baeb-2218-40f2-9a4f-47102c50b8ed\") " pod="ai-tenant-e2e-inf-b-6b11a2/test-model-6b11a2-kserve-76b7fc9c9f-pk6mx"
Sep 09 02:47:53.954918 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:53.954821    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0447baeb-2218-40f2-9a4f-47102c50b8ed-tls-certs\") pod \"test-model-6b11a2-kserve-76b7fc9c9f-pk6mx\" (UID: \"0447baeb-2218-40f2-9a4f-47102c50b8ed\") " pod="ai-tenant-e2e-inf-b-6b11a2/test-model-6b11a2-kserve-76b7fc9c9f-pk6mx"
Sep 09 02:47:53.954918 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:53.954861    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0447baeb-2218-40f2-9a4f-47102c50b8ed-model-cache\") pod \"test-model-6b11a2-kserve-76b7fc9c9f-pk6mx\" (UID: \"0447baeb-2218-40f2-9a4f-47102c50b8ed\") " pod="ai-tenant-e2e-inf-b-6b11a2/test-model-6b11a2-kserve-76b7fc9c9f-pk6mx"
Sep 09 02:47:53.954918 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:53.954900    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0447baeb-2218-40f2-9a4f-47102c50b8ed-home\") pod \"test-model-6b11a2-kserve-76b7fc9c9f-pk6mx\" (UID: \"0447baeb-2218-40f2-9a4f-47102c50b8ed\") " pod="ai-tenant-e2e-inf-b-6b11a2/test-model-6b11a2-kserve-76b7fc9c9f-pk6mx"
Sep 09 02:47:53.955124 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:53.955061    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0447baeb-2218-40f2-9a4f-47102c50b8ed-dshm\") pod \"test-model-6b11a2-kserve-76b7fc9c9f-pk6mx\" (UID: \"0447baeb-2218-40f2-9a4f-47102c50b8ed\") " pod="ai-tenant-e2e-inf-b-6b11a2/test-model-6b11a2-kserve-76b7fc9c9f-pk6mx"
Sep 09 02:47:53.955179 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:53.955150    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0447baeb-2218-40f2-9a4f-47102c50b8ed-tmp-dir\") pod \"test-model-6b11a2-kserve-76b7fc9c9f-pk6mx\" (UID: \"0447baeb-2218-40f2-9a4f-47102c50b8ed\") " pod="ai-tenant-e2e-inf-b-6b11a2/test-model-6b11a2-kserve-76b7fc9c9f-pk6mx"
Sep 09 02:47:53.955376 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:53.955328    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0447baeb-2218-40f2-9a4f-47102c50b8ed-model-cache\") pod \"test-model-6b11a2-kserve-76b7fc9c9f-pk6mx\" (UID: \"0447baeb-2218-40f2-9a4f-47102c50b8ed\") " pod="ai-tenant-e2e-inf-b-6b11a2/test-model-6b11a2-kserve-76b7fc9c9f-pk6mx"
Sep 09 02:47:53.955487 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:53.955372    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0447baeb-2218-40f2-9a4f-47102c50b8ed-home\") pod \"test-model-6b11a2-kserve-76b7fc9c9f-pk6mx\" (UID: \"0447baeb-2218-40f2-9a4f-47102c50b8ed\") " pod="ai-tenant-e2e-inf-b-6b11a2/test-model-6b11a2-kserve-76b7fc9c9f-pk6mx"
Sep 09 02:47:53.955554 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:53.955535    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0447baeb-2218-40f2-9a4f-47102c50b8ed-tmp-dir\") pod \"test-model-6b11a2-kserve-76b7fc9c9f-pk6mx\" (UID: \"0447baeb-2218-40f2-9a4f-47102c50b8ed\") " pod="ai-tenant-e2e-inf-b-6b11a2/test-model-6b11a2-kserve-76b7fc9c9f-pk6mx"
Sep 09 02:47:53.957694 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:53.957667    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0447baeb-2218-40f2-9a4f-47102c50b8ed-tls-certs\") pod \"test-model-6b11a2-kserve-76b7fc9c9f-pk6mx\" (UID: \"0447baeb-2218-40f2-9a4f-47102c50b8ed\") " pod="ai-tenant-e2e-inf-b-6b11a2/test-model-6b11a2-kserve-76b7fc9c9f-pk6mx"
Sep 09 02:47:53.957798 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:53.957761    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0447baeb-2218-40f2-9a4f-47102c50b8ed-dshm\") pod \"test-model-6b11a2-kserve-76b7fc9c9f-pk6mx\" (UID: \"0447baeb-2218-40f2-9a4f-47102c50b8ed\") " pod="ai-tenant-e2e-inf-b-6b11a2/test-model-6b11a2-kserve-76b7fc9c9f-pk6mx"
Sep 09 02:47:53.964757 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:53.964736    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndvrh\" (UniqueName: \"kubernetes.io/projected/0447baeb-2218-40f2-9a4f-47102c50b8ed-kube-api-access-ndvrh\") pod \"test-model-6b11a2-kserve-76b7fc9c9f-pk6mx\" (UID: \"0447baeb-2218-40f2-9a4f-47102c50b8ed\") " pod="ai-tenant-e2e-inf-b-6b11a2/test-model-6b11a2-kserve-76b7fc9c9f-pk6mx"
Sep 09 02:47:54.044324 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:54.044235    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-6b11a2/test-model-6b11a2-kserve-76b7fc9c9f-pk6mx"
Sep 09 02:47:54.319500 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:54.319285    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-6b11a2/test-model-6b11a2-kserve-76b7fc9c9f-pk6mx"]
Sep 09 02:47:54.322236 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:47:54.322208    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0447baeb_2218_40f2_9a4f_47102c50b8ed.slice/crio-6160b3230e5b91d7c7d64b5584eeb54c93511ef07c42c230e016864d5052d36b WatchSource:0}: Error finding container 6160b3230e5b91d7c7d64b5584eeb54c93511ef07c42c230e016864d5052d36b: Status 404 returned error can't find the container with id 6160b3230e5b91d7c7d64b5584eeb54c93511ef07c42c230e016864d5052d36b
Sep 09 02:47:54.920172 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:54.920131    2570 patch_prober.go:28] interesting pod/e2e-explicit-b52336-openshift-default-7dd744d476-nd9gp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 09 02:47:54.920543 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:54.920228    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-explicit-b52336-openshift-default-7dd744d476-nd9gp" podUID="48c61b4e-c170-4cbc-916a-efc2d6964697" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 09 02:47:54.968218 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:54.968180    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-6b11a2/test-model-6b11a2-kserve-76b7fc9c9f-pk6mx" event={"ID":"0447baeb-2218-40f2-9a4f-47102c50b8ed","Type":"ContainerStarted","Data":"dfe485c197a7e9018cf11b9191a37c44862795302b54a6c52a0bf81603058386"}
Sep 09 02:47:54.968371 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:54.968227    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-6b11a2/test-model-6b11a2-kserve-76b7fc9c9f-pk6mx" event={"ID":"0447baeb-2218-40f2-9a4f-47102c50b8ed","Type":"ContainerStarted","Data":"6160b3230e5b91d7c7d64b5584eeb54c93511ef07c42c230e016864d5052d36b"}
Sep 09 02:47:54.995462 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:54.995403    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-6b11a2/test-model-6b11a2-kserve-76b7fc9c9f-pk6mx" podStartSLOduration=1.995383658 podStartE2EDuration="1.995383658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:47:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:47:54.985932739 +0000 UTC m=+1057.240236375" watchObservedRunningTime="2026-09-09 02:47:54.995383658 +0000 UTC m=+1057.249687318"
Sep 09 02:47:55.978390 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:55.978355    2570 generic.go:356] "Generic (PLEG): container finished" podID="48c61b4e-c170-4cbc-916a-efc2d6964697" containerID="a69c40d39853a81f1c0ec8e8040dc8cb9fd44afbfc5ee500838dcf31cbdfa61c" exitCode=0
Sep 09 02:47:55.978931 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:55.978431    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-b52336-openshift-default-7dd744d476-nd9gp" event={"ID":"48c61b4e-c170-4cbc-916a-efc2d6964697","Type":"ContainerDied","Data":"a69c40d39853a81f1c0ec8e8040dc8cb9fd44afbfc5ee500838dcf31cbdfa61c"}
Sep 09 02:47:55.978931 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:55.978909    2570 scope.go:117] "RemoveContainer" containerID="a69c40d39853a81f1c0ec8e8040dc8cb9fd44afbfc5ee500838dcf31cbdfa61c"
Sep 09 02:47:55.979950 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:55.979925    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 02:47:55.980031 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:55.980005    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 02:47:55.980070 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:55.980041    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 02:47:56.985707 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:56.985673    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-b52336-openshift-default-7dd744d476-nd9gp" event={"ID":"48c61b4e-c170-4cbc-916a-efc2d6964697","Type":"ContainerStarted","Data":"884cedd932662a0483965bc7841f6eb9cf5558a05f1fd6ecb0b658f81d82dbea"}
Sep 09 02:47:57.001703 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:57.001671    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-b52336-openshift-default-7dd744d476-nd9gp"
Sep 09 02:47:57.001916 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:57.001890    2570 patch_prober.go:28] interesting pod/e2e-explicit-b52336-openshift-default-7dd744d476-nd9gp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Sep 09 02:47:57.002032 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:57.001950    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-b52336-openshift-default-7dd744d476-nd9gp" podUID="48c61b4e-c170-4cbc-916a-efc2d6964697" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Sep 09 02:47:58.001184 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:58.001151    2570 patch_prober.go:28] interesting pod/e2e-explicit-b52336-openshift-default-7dd744d476-nd9gp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Sep 09 02:47:58.001547 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:58.001208    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-b52336-openshift-default-7dd744d476-nd9gp" podUID="48c61b4e-c170-4cbc-916a-efc2d6964697" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Sep 09 02:47:59.001050 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:59.001016    2570 patch_prober.go:28] interesting pod/e2e-explicit-b52336-openshift-default-7dd744d476-nd9gp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Sep 09 02:47:59.001233 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:47:59.001074    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-b52336-openshift-default-7dd744d476-nd9gp" podUID="48c61b4e-c170-4cbc-916a-efc2d6964697" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Sep 09 02:48:00.001314 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:48:00.001276    2570 patch_prober.go:28] interesting pod/e2e-explicit-b52336-openshift-default-7dd744d476-nd9gp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Sep 09 02:48:00.001700 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:48:00.001327    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-b52336-openshift-default-7dd744d476-nd9gp" podUID="48c61b4e-c170-4cbc-916a-efc2d6964697" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Sep 09 02:48:01.001463 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:48:01.001427    2570 patch_prober.go:28] interesting pod/e2e-explicit-b52336-openshift-default-7dd744d476-nd9gp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Sep 09 02:48:01.001833 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:48:01.001506    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-b52336-openshift-default-7dd744d476-nd9gp" podUID="48c61b4e-c170-4cbc-916a-efc2d6964697" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Sep 09 02:48:02.001356 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:48:02.001314    2570 patch_prober.go:28] interesting pod/e2e-explicit-b52336-openshift-default-7dd744d476-nd9gp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Sep 09 02:48:02.001543 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:48:02.001375    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-b52336-openshift-default-7dd744d476-nd9gp" podUID="48c61b4e-c170-4cbc-916a-efc2d6964697" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Sep 09 02:48:02.375138 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:48:02.375100    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-99qjf"]
Sep 09 02:48:03.184537 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:48:03.184507    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-b52336-openshift-default-7dd744d476-nd9gp"
Sep 09 02:48:03.186017 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:48:03.185994    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-b52336-openshift-default-7dd744d476-nd9gp"
Sep 09 02:48:03.189462 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:48:03.189433    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-b52336-openshift-default-7dd744d476-nd9gp"
Sep 09 02:48:04.044604 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:48:04.044556    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-6b11a2/test-model-6b11a2-kserve-76b7fc9c9f-pk6mx"
Sep 09 02:48:04.044604 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:48:04.044607    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-6b11a2/test-model-6b11a2-kserve-76b7fc9c9f-pk6mx"
Sep 09 02:48:04.056891 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:48:04.056862    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-6b11a2/test-model-6b11a2-kserve-76b7fc9c9f-pk6mx"
Sep 09 02:48:05.038646 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:48:05.038616    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-6b11a2/test-model-6b11a2-kserve-76b7fc9c9f-pk6mx"
Sep 09 02:48:17.345326 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:48:17.345245    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-99qjf"]
Sep 09 02:48:28.854061 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:48:28.854026    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-b52336/explicit-model-b52336-kserve-7989c7cf5f-6gj8z"]
Sep 09 02:48:28.858786 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:48:28.858771    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-b52336/explicit-model-b52336-kserve-7989c7cf5f-6gj8z"
Sep 09 02:48:28.861721 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:48:28.861697    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-b52336\"/\"openshift-service-ca.crt\""
Sep 09 02:48:28.863005 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:48:28.862962    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-b52336\"/\"default-dockercfg-6dm5k\""
Sep 09 02:48:28.863112 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:48:28.863019    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-b52336\"/\"explicit-model-b52336-kserve-self-signed-certs\""
Sep 09 02:48:28.863112 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:48:28.863029    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-b52336\"/\"kube-root-ca.crt\""
Sep 09 02:48:28.868757 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:48:28.868734    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-b52336/explicit-model-b52336-kserve-7989c7cf5f-6gj8z"]
Sep 09 02:48:28.989513 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:48:28.983774    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/569e20eb-756d-4121-a45f-d2ba156664fb-home\") pod \"explicit-model-b52336-kserve-7989c7cf5f-6gj8z\" (UID: \"569e20eb-756d-4121-a45f-d2ba156664fb\") " pod="ai-tenant-e2e-explicit-b52336/explicit-model-b52336-kserve-7989c7cf5f-6gj8z"
Sep 09 02:48:28.989513 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:48:28.983841    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/569e20eb-756d-4121-a45f-d2ba156664fb-tls-certs\") pod \"explicit-model-b52336-kserve-7989c7cf5f-6gj8z\" (UID: \"569e20eb-756d-4121-a45f-d2ba156664fb\") " pod="ai-tenant-e2e-explicit-b52336/explicit-model-b52336-kserve-7989c7cf5f-6gj8z"
Sep 09 02:48:28.989513 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:48:28.983874    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/569e20eb-756d-4121-a45f-d2ba156664fb-dshm\") pod \"explicit-model-b52336-kserve-7989c7cf5f-6gj8z\" (UID: \"569e20eb-756d-4121-a45f-d2ba156664fb\") " pod="ai-tenant-e2e-explicit-b52336/explicit-model-b52336-kserve-7989c7cf5f-6gj8z"
Sep 09 02:48:28.989513 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:48:28.983970    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/569e20eb-756d-4121-a45f-d2ba156664fb-model-cache\") pod \"explicit-model-b52336-kserve-7989c7cf5f-6gj8z\" (UID: \"569e20eb-756d-4121-a45f-d2ba156664fb\") " pod="ai-tenant-e2e-explicit-b52336/explicit-model-b52336-kserve-7989c7cf5f-6gj8z"
Sep 09 02:48:28.989513 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:48:28.984044    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cchzs\" (UniqueName: \"kubernetes.io/projected/569e20eb-756d-4121-a45f-d2ba156664fb-kube-api-access-cchzs\") pod \"explicit-model-b52336-kserve-7989c7cf5f-6gj8z\" (UID: \"569e20eb-756d-4121-a45f-d2ba156664fb\") " pod="ai-tenant-e2e-explicit-b52336/explicit-model-b52336-kserve-7989c7cf5f-6gj8z"
Sep 09 02:48:28.989513 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:48:28.984087    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/569e20eb-756d-4121-a45f-d2ba156664fb-tmp-dir\") pod \"explicit-model-b52336-kserve-7989c7cf5f-6gj8z\" (UID: \"569e20eb-756d-4121-a45f-d2ba156664fb\") " pod="ai-tenant-e2e-explicit-b52336/explicit-model-b52336-kserve-7989c7cf5f-6gj8z"
Sep 09 02:48:29.085126 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:48:29.085093    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/569e20eb-756d-4121-a45f-d2ba156664fb-model-cache\") pod \"explicit-model-b52336-kserve-7989c7cf5f-6gj8z\" (UID: \"569e20eb-756d-4121-a45f-d2ba156664fb\") " pod="ai-tenant-e2e-explicit-b52336/explicit-model-b52336-kserve-7989c7cf5f-6gj8z"
Sep 09 02:48:29.085292 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:48:29.085140    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cchzs\" (UniqueName: \"kubernetes.io/projected/569e20eb-756d-4121-a45f-d2ba156664fb-kube-api-access-cchzs\") pod \"explicit-model-b52336-kserve-7989c7cf5f-6gj8z\" (UID: \"569e20eb-756d-4121-a45f-d2ba156664fb\") " pod="ai-tenant-e2e-explicit-b52336/explicit-model-b52336-kserve-7989c7cf5f-6gj8z"
Sep 09 02:48:29.085292 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:48:29.085164    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/569e20eb-756d-4121-a45f-d2ba156664fb-tmp-dir\") pod \"explicit-model-b52336-kserve-7989c7cf5f-6gj8z\" (UID: \"569e20eb-756d-4121-a45f-d2ba156664fb\") " pod="ai-tenant-e2e-explicit-b52336/explicit-model-b52336-kserve-7989c7cf5f-6gj8z"
Sep 09 02:48:29.085292 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:48:29.085204    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/569e20eb-756d-4121-a45f-d2ba156664fb-home\") pod \"explicit-model-b52336-kserve-7989c7cf5f-6gj8z\" (UID: \"569e20eb-756d-4121-a45f-d2ba156664fb\") " pod="ai-tenant-e2e-explicit-b52336/explicit-model-b52336-kserve-7989c7cf5f-6gj8z"
Sep 09 02:48:29.085292 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:48:29.085225    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/569e20eb-756d-4121-a45f-d2ba156664fb-tls-certs\") pod \"explicit-model-b52336-kserve-7989c7cf5f-6gj8z\" (UID: \"569e20eb-756d-4121-a45f-d2ba156664fb\") " pod="ai-tenant-e2e-explicit-b52336/explicit-model-b52336-kserve-7989c7cf5f-6gj8z"
Sep 09 02:48:29.085292 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:48:29.085239    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/569e20eb-756d-4121-a45f-d2ba156664fb-dshm\") pod \"explicit-model-b52336-kserve-7989c7cf5f-6gj8z\" (UID: \"569e20eb-756d-4121-a45f-d2ba156664fb\") " pod="ai-tenant-e2e-explicit-b52336/explicit-model-b52336-kserve-7989c7cf5f-6gj8z"
Sep 09 02:48:29.085615 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:48:29.085558    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/569e20eb-756d-4121-a45f-d2ba156664fb-model-cache\") pod \"explicit-model-b52336-kserve-7989c7cf5f-6gj8z\" (UID: \"569e20eb-756d-4121-a45f-d2ba156664fb\") " pod="ai-tenant-e2e-explicit-b52336/explicit-model-b52336-kserve-7989c7cf5f-6gj8z"
Sep 09 02:48:29.085615 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:48:29.085592    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/569e20eb-756d-4121-a45f-d2ba156664fb-home\") pod \"explicit-model-b52336-kserve-7989c7cf5f-6gj8z\" (UID: \"569e20eb-756d-4121-a45f-d2ba156664fb\") " pod="ai-tenant-e2e-explicit-b52336/explicit-model-b52336-kserve-7989c7cf5f-6gj8z"
Sep 09 02:48:29.085692 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:48:29.085652    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/569e20eb-756d-4121-a45f-d2ba156664fb-tmp-dir\") pod \"explicit-model-b52336-kserve-7989c7cf5f-6gj8z\" (UID: \"569e20eb-756d-4121-a45f-d2ba156664fb\") " pod="ai-tenant-e2e-explicit-b52336/explicit-model-b52336-kserve-7989c7cf5f-6gj8z"
Sep 09 02:48:29.087430 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:48:29.087410    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/569e20eb-756d-4121-a45f-d2ba156664fb-dshm\") pod \"explicit-model-b52336-kserve-7989c7cf5f-6gj8z\" (UID: \"569e20eb-756d-4121-a45f-d2ba156664fb\") " pod="ai-tenant-e2e-explicit-b52336/explicit-model-b52336-kserve-7989c7cf5f-6gj8z"
Sep 09 02:48:29.087727 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:48:29.087707    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/569e20eb-756d-4121-a45f-d2ba156664fb-tls-certs\") pod \"explicit-model-b52336-kserve-7989c7cf5f-6gj8z\" (UID: \"569e20eb-756d-4121-a45f-d2ba156664fb\") " pod="ai-tenant-e2e-explicit-b52336/explicit-model-b52336-kserve-7989c7cf5f-6gj8z"
Sep 09 02:48:29.093892 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:48:29.093874    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cchzs\" (UniqueName: \"kubernetes.io/projected/569e20eb-756d-4121-a45f-d2ba156664fb-kube-api-access-cchzs\") pod \"explicit-model-b52336-kserve-7989c7cf5f-6gj8z\" (UID: \"569e20eb-756d-4121-a45f-d2ba156664fb\") " pod="ai-tenant-e2e-explicit-b52336/explicit-model-b52336-kserve-7989c7cf5f-6gj8z"
Sep 09 02:48:29.168842 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:48:29.168784    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-b52336/explicit-model-b52336-kserve-7989c7cf5f-6gj8z"
Sep 09 02:48:29.447944 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:48:29.447924    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-b52336/explicit-model-b52336-kserve-7989c7cf5f-6gj8z"]
Sep 09 02:48:29.450389 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:48:29.450347    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod569e20eb_756d_4121_a45f_d2ba156664fb.slice/crio-4f5cd55ec27f26c0e9905ce302034edb5638bf35c84b92a2874934a124e02ab9 WatchSource:0}: Error finding container 4f5cd55ec27f26c0e9905ce302034edb5638bf35c84b92a2874934a124e02ab9: Status 404 returned error can't find the container with id 4f5cd55ec27f26c0e9905ce302034edb5638bf35c84b92a2874934a124e02ab9
Sep 09 02:48:30.159423 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:48:30.159382    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-b52336/explicit-model-b52336-kserve-7989c7cf5f-6gj8z" event={"ID":"569e20eb-756d-4121-a45f-d2ba156664fb","Type":"ContainerStarted","Data":"6052a3e7f43659deef88dbb874cac4590ae78ad5dadd7624348a4676cf302104"}
Sep 09 02:48:30.159423 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:48:30.159434    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-b52336/explicit-model-b52336-kserve-7989c7cf5f-6gj8z" event={"ID":"569e20eb-756d-4121-a45f-d2ba156664fb","Type":"ContainerStarted","Data":"4f5cd55ec27f26c0e9905ce302034edb5638bf35c84b92a2874934a124e02ab9"}
Sep 09 02:48:30.178162 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:48:30.178110    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-b52336/explicit-model-b52336-kserve-7989c7cf5f-6gj8z" podStartSLOduration=2.178093725 podStartE2EDuration="2.178093725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:48:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:48:30.177587361 +0000 UTC m=+1092.431890997" watchObservedRunningTime="2026-09-09 02:48:30.178093725 +0000 UTC m=+1092.432397382"
Sep 09 02:48:30.561559 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:48:30.561476    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-99qjf"]
Sep 09 02:48:38.459322 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:48:38.459284    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-b52336/explicit-model-b52336-kserve-7989c7cf5f-6gj8z"]
Sep 09 02:48:39.169218 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:48:39.169186    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-b52336/explicit-model-b52336-kserve-7989c7cf5f-6gj8z"
Sep 09 02:48:53.480790 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:48:53.480743    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-b52336/explicit-model-b52336-kserve-7989c7cf5f-6gj8z" podUID="569e20eb-756d-4121-a45f-d2ba156664fb" containerName="main" containerID="cri-o://6052a3e7f43659deef88dbb874cac4590ae78ad5dadd7624348a4676cf302104" gracePeriod=45
Sep 09 02:48:53.740285 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:48:53.740228    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-b52336/explicit-model-b52336-kserve-7989c7cf5f-6gj8z"
Sep 09 02:48:53.826241 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:48:53.826211    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/569e20eb-756d-4121-a45f-d2ba156664fb-model-cache\") pod \"569e20eb-756d-4121-a45f-d2ba156664fb\" (UID: \"569e20eb-756d-4121-a45f-d2ba156664fb\") "
Sep 09 02:48:53.826384 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:48:53.826284    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/569e20eb-756d-4121-a45f-d2ba156664fb-tmp-dir\") pod \"569e20eb-756d-4121-a45f-d2ba156664fb\" (UID: \"569e20eb-756d-4121-a45f-d2ba156664fb\") "
Sep 09 02:48:53.826384 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:48:53.826303    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/569e20eb-756d-4121-a45f-d2ba156664fb-dshm\") pod \"569e20eb-756d-4121-a45f-d2ba156664fb\" (UID: \"569e20eb-756d-4121-a45f-d2ba156664fb\") "
Sep 09 02:48:53.826384 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:48:53.826330    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/569e20eb-756d-4121-a45f-d2ba156664fb-home\") pod \"569e20eb-756d-4121-a45f-d2ba156664fb\" (UID: \"569e20eb-756d-4121-a45f-d2ba156664fb\") "
Sep 09 02:48:53.826526 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:48:53.826391    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cchzs\" (UniqueName: \"kubernetes.io/projected/569e20eb-756d-4121-a45f-d2ba156664fb-kube-api-access-cchzs\") pod \"569e20eb-756d-4121-a45f-d2ba156664fb\" (UID: \"569e20eb-756d-4121-a45f-d2ba156664fb\") "
Sep 09 02:48:53.826526 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:48:53.826423    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/569e20eb-756d-4121-a45f-d2ba156664fb-tls-certs\") pod \"569e20eb-756d-4121-a45f-d2ba156664fb\" (UID: \"569e20eb-756d-4121-a45f-d2ba156664fb\") "
Sep 09 02:48:53.826526 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:48:53.826475    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/569e20eb-756d-4121-a45f-d2ba156664fb-model-cache" (OuterVolumeSpecName: "model-cache") pod "569e20eb-756d-4121-a45f-d2ba156664fb" (UID: "569e20eb-756d-4121-a45f-d2ba156664fb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 02:48:53.826692 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:48:53.826577    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/569e20eb-756d-4121-a45f-d2ba156664fb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "569e20eb-756d-4121-a45f-d2ba156664fb" (UID: "569e20eb-756d-4121-a45f-d2ba156664fb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 02:48:53.826692 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:48:53.826649    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/569e20eb-756d-4121-a45f-d2ba156664fb-home" (OuterVolumeSpecName: "home") pod "569e20eb-756d-4121-a45f-d2ba156664fb" (UID: "569e20eb-756d-4121-a45f-d2ba156664fb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 02:48:53.826800 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:48:53.826747    2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/569e20eb-756d-4121-a45f-d2ba156664fb-tmp-dir\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:48:53.826800 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:48:53.826760    2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/569e20eb-756d-4121-a45f-d2ba156664fb-home\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:48:53.826800 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:48:53.826769    2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/569e20eb-756d-4121-a45f-d2ba156664fb-model-cache\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:48:53.828508 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:48:53.828482    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/569e20eb-756d-4121-a45f-d2ba156664fb-dshm" (OuterVolumeSpecName: "dshm") pod "569e20eb-756d-4121-a45f-d2ba156664fb" (UID: "569e20eb-756d-4121-a45f-d2ba156664fb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 02:48:53.828600 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:48:53.828559    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/569e20eb-756d-4121-a45f-d2ba156664fb-kube-api-access-cchzs" (OuterVolumeSpecName: "kube-api-access-cchzs") pod "569e20eb-756d-4121-a45f-d2ba156664fb" (UID: "569e20eb-756d-4121-a45f-d2ba156664fb"). InnerVolumeSpecName "kube-api-access-cchzs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 02:48:53.828654 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:48:53.828635    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/569e20eb-756d-4121-a45f-d2ba156664fb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "569e20eb-756d-4121-a45f-d2ba156664fb" (UID: "569e20eb-756d-4121-a45f-d2ba156664fb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 02:48:53.927943 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:48:53.927914    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cchzs\" (UniqueName: \"kubernetes.io/projected/569e20eb-756d-4121-a45f-d2ba156664fb-kube-api-access-cchzs\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:48:53.927943 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:48:53.927940    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/569e20eb-756d-4121-a45f-d2ba156664fb-tls-certs\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:48:53.928119 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:48:53.927951    2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/569e20eb-756d-4121-a45f-d2ba156664fb-dshm\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:48:54.284108 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:48:54.284073    2570 generic.go:356] "Generic (PLEG): container finished" podID="569e20eb-756d-4121-a45f-d2ba156664fb" containerID="6052a3e7f43659deef88dbb874cac4590ae78ad5dadd7624348a4676cf302104" exitCode=0
Sep 09 02:48:54.284245 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:48:54.284116    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-b52336/explicit-model-b52336-kserve-7989c7cf5f-6gj8z"
Sep 09 02:48:54.284245 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:48:54.284151    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-b52336/explicit-model-b52336-kserve-7989c7cf5f-6gj8z" event={"ID":"569e20eb-756d-4121-a45f-d2ba156664fb","Type":"ContainerDied","Data":"6052a3e7f43659deef88dbb874cac4590ae78ad5dadd7624348a4676cf302104"}
Sep 09 02:48:54.284245 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:48:54.284193    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-b52336/explicit-model-b52336-kserve-7989c7cf5f-6gj8z" event={"ID":"569e20eb-756d-4121-a45f-d2ba156664fb","Type":"ContainerDied","Data":"4f5cd55ec27f26c0e9905ce302034edb5638bf35c84b92a2874934a124e02ab9"}
Sep 09 02:48:54.284245 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:48:54.284210    2570 scope.go:117] "RemoveContainer" containerID="6052a3e7f43659deef88dbb874cac4590ae78ad5dadd7624348a4676cf302104"
Sep 09 02:48:54.292769 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:48:54.292751    2570 scope.go:117] "RemoveContainer" containerID="6052a3e7f43659deef88dbb874cac4590ae78ad5dadd7624348a4676cf302104"
Sep 09 02:48:54.293044 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:48:54.293024    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6052a3e7f43659deef88dbb874cac4590ae78ad5dadd7624348a4676cf302104\": container with ID starting with 6052a3e7f43659deef88dbb874cac4590ae78ad5dadd7624348a4676cf302104 not found: ID does not exist" containerID="6052a3e7f43659deef88dbb874cac4590ae78ad5dadd7624348a4676cf302104"
Sep 09 02:48:54.293125 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:48:54.293054    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6052a3e7f43659deef88dbb874cac4590ae78ad5dadd7624348a4676cf302104"} err="failed to get container status \"6052a3e7f43659deef88dbb874cac4590ae78ad5dadd7624348a4676cf302104\": rpc error: code = NotFound desc = could not find container \"6052a3e7f43659deef88dbb874cac4590ae78ad5dadd7624348a4676cf302104\": container with ID starting with 6052a3e7f43659deef88dbb874cac4590ae78ad5dadd7624348a4676cf302104 not found: ID does not exist"
Sep 09 02:48:54.301656 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:48:54.301635    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-b52336/explicit-model-b52336-kserve-7989c7cf5f-6gj8z"]
Sep 09 02:48:54.305961 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:48:54.305943    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-b52336/explicit-model-b52336-kserve-7989c7cf5f-6gj8z"]
Sep 09 02:48:55.143123 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:48:55.143087    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-99qjf"]
Sep 09 02:48:55.891550 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:48:55.891488    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-b52336-openshift-default-7dd744d476-nd9gp"]
Sep 09 02:48:55.891797 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:48:55.891772    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-b52336-openshift-default-7dd744d476-nd9gp" podUID="48c61b4e-c170-4cbc-916a-efc2d6964697" containerName="istio-proxy" containerID="cri-o://884cedd932662a0483965bc7841f6eb9cf5558a05f1fd6ecb0b658f81d82dbea" gracePeriod=30
Sep 09 02:48:56.223112 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:48:56.223035    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="569e20eb-756d-4121-a45f-d2ba156664fb" path="/var/lib/kubelet/pods/569e20eb-756d-4121-a45f-d2ba156664fb/volumes"
Sep 09 02:49:01.141678 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:01.141655    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-b52336-openshift-default-7dd744d476-nd9gp"
Sep 09 02:49:01.297864 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:01.297773    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48c61b4e-c170-4cbc-916a-efc2d6964697-workload-certs\") pod \"48c61b4e-c170-4cbc-916a-efc2d6964697\" (UID: \"48c61b4e-c170-4cbc-916a-efc2d6964697\") "
Sep 09 02:49:01.297864 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:01.297822    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48c61b4e-c170-4cbc-916a-efc2d6964697-istio-envoy\") pod \"48c61b4e-c170-4cbc-916a-efc2d6964697\" (UID: \"48c61b4e-c170-4cbc-916a-efc2d6964697\") "
Sep 09 02:49:01.297864 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:01.297845    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48c61b4e-c170-4cbc-916a-efc2d6964697-workload-socket\") pod \"48c61b4e-c170-4cbc-916a-efc2d6964697\" (UID: \"48c61b4e-c170-4cbc-916a-efc2d6964697\") "
Sep 09 02:49:01.298191 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:01.297885    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48c61b4e-c170-4cbc-916a-efc2d6964697-istiod-ca-cert\") pod \"48c61b4e-c170-4cbc-916a-efc2d6964697\" (UID: \"48c61b4e-c170-4cbc-916a-efc2d6964697\") "
Sep 09 02:49:01.298191 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:01.297917    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7mz6l\" (UniqueName: \"kubernetes.io/projected/48c61b4e-c170-4cbc-916a-efc2d6964697-kube-api-access-7mz6l\") pod \"48c61b4e-c170-4cbc-916a-efc2d6964697\" (UID: \"48c61b4e-c170-4cbc-916a-efc2d6964697\") "
Sep 09 02:49:01.298191 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:01.297944    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48c61b4e-c170-4cbc-916a-efc2d6964697-credential-socket\") pod \"48c61b4e-c170-4cbc-916a-efc2d6964697\" (UID: \"48c61b4e-c170-4cbc-916a-efc2d6964697\") "
Sep 09 02:49:01.298191 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:01.298077    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48c61b4e-c170-4cbc-916a-efc2d6964697-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "48c61b4e-c170-4cbc-916a-efc2d6964697" (UID: "48c61b4e-c170-4cbc-916a-efc2d6964697"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 02:49:01.298191 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:01.298115    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48c61b4e-c170-4cbc-916a-efc2d6964697-istio-token\") pod \"48c61b4e-c170-4cbc-916a-efc2d6964697\" (UID: \"48c61b4e-c170-4cbc-916a-efc2d6964697\") "
Sep 09 02:49:01.298191 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:01.298167    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48c61b4e-c170-4cbc-916a-efc2d6964697-istio-data\") pod \"48c61b4e-c170-4cbc-916a-efc2d6964697\" (UID: \"48c61b4e-c170-4cbc-916a-efc2d6964697\") "
Sep 09 02:49:01.298498 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:01.298203    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48c61b4e-c170-4cbc-916a-efc2d6964697-istio-podinfo\") pod \"48c61b4e-c170-4cbc-916a-efc2d6964697\" (UID: \"48c61b4e-c170-4cbc-916a-efc2d6964697\") "
Sep 09 02:49:01.298498 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:01.298208    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48c61b4e-c170-4cbc-916a-efc2d6964697-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "48c61b4e-c170-4cbc-916a-efc2d6964697" (UID: "48c61b4e-c170-4cbc-916a-efc2d6964697"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 02:49:01.298498 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:01.298298    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48c61b4e-c170-4cbc-916a-efc2d6964697-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "48c61b4e-c170-4cbc-916a-efc2d6964697" (UID: "48c61b4e-c170-4cbc-916a-efc2d6964697"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 02:49:01.298498 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:01.298443    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48c61b4e-c170-4cbc-916a-efc2d6964697-istio-data" (OuterVolumeSpecName: "istio-data") pod "48c61b4e-c170-4cbc-916a-efc2d6964697" (UID: "48c61b4e-c170-4cbc-916a-efc2d6964697"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 02:49:01.298682 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:01.298653    2570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48c61b4e-c170-4cbc-916a-efc2d6964697-workload-certs\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:49:01.298682 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:01.298672    2570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48c61b4e-c170-4cbc-916a-efc2d6964697-workload-socket\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:49:01.298757 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:01.298684    2570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48c61b4e-c170-4cbc-916a-efc2d6964697-credential-socket\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:49:01.298757 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:01.298697    2570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48c61b4e-c170-4cbc-916a-efc2d6964697-istio-data\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:49:01.298913 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:01.298893    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48c61b4e-c170-4cbc-916a-efc2d6964697-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "48c61b4e-c170-4cbc-916a-efc2d6964697" (UID: "48c61b4e-c170-4cbc-916a-efc2d6964697"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 02:49:01.300205 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:01.300174    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48c61b4e-c170-4cbc-916a-efc2d6964697-istio-token" (OuterVolumeSpecName: "istio-token") pod "48c61b4e-c170-4cbc-916a-efc2d6964697" (UID: "48c61b4e-c170-4cbc-916a-efc2d6964697"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 02:49:01.300333 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:01.300302    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/48c61b4e-c170-4cbc-916a-efc2d6964697-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "48c61b4e-c170-4cbc-916a-efc2d6964697" (UID: "48c61b4e-c170-4cbc-916a-efc2d6964697"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 02:49:01.300555 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:01.300534    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48c61b4e-c170-4cbc-916a-efc2d6964697-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "48c61b4e-c170-4cbc-916a-efc2d6964697" (UID: "48c61b4e-c170-4cbc-916a-efc2d6964697"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 02:49:01.300555 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:01.300543    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48c61b4e-c170-4cbc-916a-efc2d6964697-kube-api-access-7mz6l" (OuterVolumeSpecName: "kube-api-access-7mz6l") pod "48c61b4e-c170-4cbc-916a-efc2d6964697" (UID: "48c61b4e-c170-4cbc-916a-efc2d6964697"). InnerVolumeSpecName "kube-api-access-7mz6l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 02:49:01.323658 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:01.323632    2570 generic.go:356] "Generic (PLEG): container finished" podID="48c61b4e-c170-4cbc-916a-efc2d6964697" containerID="884cedd932662a0483965bc7841f6eb9cf5558a05f1fd6ecb0b658f81d82dbea" exitCode=0
Sep 09 02:49:01.323774 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:01.323685    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-b52336-openshift-default-7dd744d476-nd9gp"
Sep 09 02:49:01.323774 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:01.323717    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-b52336-openshift-default-7dd744d476-nd9gp" event={"ID":"48c61b4e-c170-4cbc-916a-efc2d6964697","Type":"ContainerDied","Data":"884cedd932662a0483965bc7841f6eb9cf5558a05f1fd6ecb0b658f81d82dbea"}
Sep 09 02:49:01.323887 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:01.323775    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-b52336-openshift-default-7dd744d476-nd9gp" event={"ID":"48c61b4e-c170-4cbc-916a-efc2d6964697","Type":"ContainerDied","Data":"599b1c2e45df5c85da61be6df04a5f9866da05bc93ca2904bc346802015dd8af"}
Sep 09 02:49:01.323887 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:01.323800    2570 scope.go:117] "RemoveContainer" containerID="884cedd932662a0483965bc7841f6eb9cf5558a05f1fd6ecb0b658f81d82dbea"
Sep 09 02:49:01.333084 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:01.333069    2570 scope.go:117] "RemoveContainer" containerID="a69c40d39853a81f1c0ec8e8040dc8cb9fd44afbfc5ee500838dcf31cbdfa61c"
Sep 09 02:49:01.340527 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:01.340509    2570 scope.go:117] "RemoveContainer" containerID="884cedd932662a0483965bc7841f6eb9cf5558a05f1fd6ecb0b658f81d82dbea"
Sep 09 02:49:01.340772 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:49:01.340751    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"884cedd932662a0483965bc7841f6eb9cf5558a05f1fd6ecb0b658f81d82dbea\": container with ID starting with 884cedd932662a0483965bc7841f6eb9cf5558a05f1fd6ecb0b658f81d82dbea not found: ID does not exist" containerID="884cedd932662a0483965bc7841f6eb9cf5558a05f1fd6ecb0b658f81d82dbea"
Sep 09 02:49:01.340844 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:01.340782    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"884cedd932662a0483965bc7841f6eb9cf5558a05f1fd6ecb0b658f81d82dbea"} err="failed to get container status \"884cedd932662a0483965bc7841f6eb9cf5558a05f1fd6ecb0b658f81d82dbea\": rpc error: code = NotFound desc = could not find container \"884cedd932662a0483965bc7841f6eb9cf5558a05f1fd6ecb0b658f81d82dbea\": container with ID starting with 884cedd932662a0483965bc7841f6eb9cf5558a05f1fd6ecb0b658f81d82dbea not found: ID does not exist"
Sep 09 02:49:01.340844 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:01.340805    2570 scope.go:117] "RemoveContainer" containerID="a69c40d39853a81f1c0ec8e8040dc8cb9fd44afbfc5ee500838dcf31cbdfa61c"
Sep 09 02:49:01.341066 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:49:01.341048    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a69c40d39853a81f1c0ec8e8040dc8cb9fd44afbfc5ee500838dcf31cbdfa61c\": container with ID starting with a69c40d39853a81f1c0ec8e8040dc8cb9fd44afbfc5ee500838dcf31cbdfa61c not found: ID does not exist" containerID="a69c40d39853a81f1c0ec8e8040dc8cb9fd44afbfc5ee500838dcf31cbdfa61c"
Sep 09 02:49:01.341114 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:01.341072    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a69c40d39853a81f1c0ec8e8040dc8cb9fd44afbfc5ee500838dcf31cbdfa61c"} err="failed to get container status \"a69c40d39853a81f1c0ec8e8040dc8cb9fd44afbfc5ee500838dcf31cbdfa61c\": rpc error: code = NotFound desc = could not find container \"a69c40d39853a81f1c0ec8e8040dc8cb9fd44afbfc5ee500838dcf31cbdfa61c\": container with ID starting with a69c40d39853a81f1c0ec8e8040dc8cb9fd44afbfc5ee500838dcf31cbdfa61c not found: ID does not exist"
Sep 09 02:49:01.348234 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:01.348210    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-b52336-openshift-default-7dd744d476-nd9gp"]
Sep 09 02:49:01.359794 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:01.359766    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-b52336-openshift-default-7dd744d476-nd9gp"]
Sep 09 02:49:01.399374 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:01.399352    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mz6l\" (UniqueName: \"kubernetes.io/projected/48c61b4e-c170-4cbc-916a-efc2d6964697-kube-api-access-7mz6l\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:49:01.399374 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:01.399372    2570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48c61b4e-c170-4cbc-916a-efc2d6964697-istio-token\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:49:01.399484 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:01.399382    2570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48c61b4e-c170-4cbc-916a-efc2d6964697-istio-podinfo\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:49:01.399484 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:01.399390    2570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48c61b4e-c170-4cbc-916a-efc2d6964697-istio-envoy\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:49:01.399484 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:01.399399    2570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48c61b4e-c170-4cbc-916a-efc2d6964697-istiod-ca-cert\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:49:02.223419 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:02.223376    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48c61b4e-c170-4cbc-916a-efc2d6964697" path="/var/lib/kubelet/pods/48c61b4e-c170-4cbc-916a-efc2d6964697/volumes"
Sep 09 02:49:09.550799 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:09.550761    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-ac6878/noten-model-ac6878-kserve-bf6cf6f67-gk428"]
Sep 09 02:49:09.551200 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:09.551185    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48c61b4e-c170-4cbc-916a-efc2d6964697" containerName="istio-proxy"
Sep 09 02:49:09.551255 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:09.551202    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="48c61b4e-c170-4cbc-916a-efc2d6964697" containerName="istio-proxy"
Sep 09 02:49:09.551255 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:09.551228    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="569e20eb-756d-4121-a45f-d2ba156664fb" containerName="main"
Sep 09 02:49:09.551255 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:09.551235    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="569e20eb-756d-4121-a45f-d2ba156664fb" containerName="main"
Sep 09 02:49:09.551403 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:09.551297    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="48c61b4e-c170-4cbc-916a-efc2d6964697" containerName="istio-proxy"
Sep 09 02:49:09.551403 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:09.551311    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="569e20eb-756d-4121-a45f-d2ba156664fb" containerName="main"
Sep 09 02:49:09.555735 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:09.555719    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-ac6878/noten-model-ac6878-kserve-bf6cf6f67-gk428"
Sep 09 02:49:09.560075 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:09.560055    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-ac6878\"/\"openshift-service-ca.crt\""
Sep 09 02:49:09.560075 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:09.560062    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-ac6878\"/\"noten-model-ac6878-kserve-self-signed-certs\""
Sep 09 02:49:09.560256 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:09.560055    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-ac6878\"/\"kube-root-ca.crt\""
Sep 09 02:49:09.560256 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:09.560099    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-ac6878\"/\"default-dockercfg-54bwf\""
Sep 09 02:49:09.565564 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:09.565543    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-ac6878/noten-model-ac6878-kserve-bf6cf6f67-gk428"]
Sep 09 02:49:09.672587 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:09.672551    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftnl7\" (UniqueName: \"kubernetes.io/projected/dcc8f099-7956-4362-af4d-75367a95d07f-kube-api-access-ftnl7\") pod \"noten-model-ac6878-kserve-bf6cf6f67-gk428\" (UID: \"dcc8f099-7956-4362-af4d-75367a95d07f\") " pod="ai-tenant-e2e-noten-ac6878/noten-model-ac6878-kserve-bf6cf6f67-gk428"
Sep 09 02:49:09.672778 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:09.672613    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dcc8f099-7956-4362-af4d-75367a95d07f-dshm\") pod \"noten-model-ac6878-kserve-bf6cf6f67-gk428\" (UID: \"dcc8f099-7956-4362-af4d-75367a95d07f\") " pod="ai-tenant-e2e-noten-ac6878/noten-model-ac6878-kserve-bf6cf6f67-gk428"
Sep 09 02:49:09.672778 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:09.672680    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dcc8f099-7956-4362-af4d-75367a95d07f-tmp-dir\") pod \"noten-model-ac6878-kserve-bf6cf6f67-gk428\" (UID: \"dcc8f099-7956-4362-af4d-75367a95d07f\") " pod="ai-tenant-e2e-noten-ac6878/noten-model-ac6878-kserve-bf6cf6f67-gk428"
Sep 09 02:49:09.672778 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:09.672701    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dcc8f099-7956-4362-af4d-75367a95d07f-home\") pod \"noten-model-ac6878-kserve-bf6cf6f67-gk428\" (UID: \"dcc8f099-7956-4362-af4d-75367a95d07f\") " pod="ai-tenant-e2e-noten-ac6878/noten-model-ac6878-kserve-bf6cf6f67-gk428"
Sep 09 02:49:09.672778 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:09.672726    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dcc8f099-7956-4362-af4d-75367a95d07f-model-cache\") pod \"noten-model-ac6878-kserve-bf6cf6f67-gk428\" (UID: \"dcc8f099-7956-4362-af4d-75367a95d07f\") " pod="ai-tenant-e2e-noten-ac6878/noten-model-ac6878-kserve-bf6cf6f67-gk428"
Sep 09 02:49:09.672778 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:09.672747    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dcc8f099-7956-4362-af4d-75367a95d07f-tls-certs\") pod \"noten-model-ac6878-kserve-bf6cf6f67-gk428\" (UID: \"dcc8f099-7956-4362-af4d-75367a95d07f\") " pod="ai-tenant-e2e-noten-ac6878/noten-model-ac6878-kserve-bf6cf6f67-gk428"
Sep 09 02:49:09.774131 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:09.774083    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dcc8f099-7956-4362-af4d-75367a95d07f-tmp-dir\") pod \"noten-model-ac6878-kserve-bf6cf6f67-gk428\" (UID: \"dcc8f099-7956-4362-af4d-75367a95d07f\") " pod="ai-tenant-e2e-noten-ac6878/noten-model-ac6878-kserve-bf6cf6f67-gk428"
Sep 09 02:49:09.774131 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:09.774130    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dcc8f099-7956-4362-af4d-75367a95d07f-home\") pod \"noten-model-ac6878-kserve-bf6cf6f67-gk428\" (UID: \"dcc8f099-7956-4362-af4d-75367a95d07f\") " pod="ai-tenant-e2e-noten-ac6878/noten-model-ac6878-kserve-bf6cf6f67-gk428"
Sep 09 02:49:09.774406 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:09.774158    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dcc8f099-7956-4362-af4d-75367a95d07f-model-cache\") pod \"noten-model-ac6878-kserve-bf6cf6f67-gk428\" (UID: \"dcc8f099-7956-4362-af4d-75367a95d07f\") " pod="ai-tenant-e2e-noten-ac6878/noten-model-ac6878-kserve-bf6cf6f67-gk428"
Sep 09 02:49:09.774406 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:09.774276    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dcc8f099-7956-4362-af4d-75367a95d07f-tls-certs\") pod \"noten-model-ac6878-kserve-bf6cf6f67-gk428\" (UID: \"dcc8f099-7956-4362-af4d-75367a95d07f\") " pod="ai-tenant-e2e-noten-ac6878/noten-model-ac6878-kserve-bf6cf6f67-gk428"
Sep 09 02:49:09.774406 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:09.774395    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftnl7\" (UniqueName: \"kubernetes.io/projected/dcc8f099-7956-4362-af4d-75367a95d07f-kube-api-access-ftnl7\") pod \"noten-model-ac6878-kserve-bf6cf6f67-gk428\" (UID: \"dcc8f099-7956-4362-af4d-75367a95d07f\") " pod="ai-tenant-e2e-noten-ac6878/noten-model-ac6878-kserve-bf6cf6f67-gk428"
Sep 09 02:49:09.774579 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:09.774457    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dcc8f099-7956-4362-af4d-75367a95d07f-dshm\") pod \"noten-model-ac6878-kserve-bf6cf6f67-gk428\" (UID: \"dcc8f099-7956-4362-af4d-75367a95d07f\") " pod="ai-tenant-e2e-noten-ac6878/noten-model-ac6878-kserve-bf6cf6f67-gk428"
Sep 09 02:49:09.774579 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:09.774563    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dcc8f099-7956-4362-af4d-75367a95d07f-home\") pod \"noten-model-ac6878-kserve-bf6cf6f67-gk428\" (UID: \"dcc8f099-7956-4362-af4d-75367a95d07f\") " pod="ai-tenant-e2e-noten-ac6878/noten-model-ac6878-kserve-bf6cf6f67-gk428"
Sep 09 02:49:09.774662 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:09.774577    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dcc8f099-7956-4362-af4d-75367a95d07f-model-cache\") pod \"noten-model-ac6878-kserve-bf6cf6f67-gk428\" (UID: \"dcc8f099-7956-4362-af4d-75367a95d07f\") " pod="ai-tenant-e2e-noten-ac6878/noten-model-ac6878-kserve-bf6cf6f67-gk428"
Sep 09 02:49:09.774662 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:09.774609    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dcc8f099-7956-4362-af4d-75367a95d07f-tmp-dir\") pod \"noten-model-ac6878-kserve-bf6cf6f67-gk428\" (UID: \"dcc8f099-7956-4362-af4d-75367a95d07f\") " pod="ai-tenant-e2e-noten-ac6878/noten-model-ac6878-kserve-bf6cf6f67-gk428"
Sep 09 02:49:09.776593 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:09.776566    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dcc8f099-7956-4362-af4d-75367a95d07f-dshm\") pod \"noten-model-ac6878-kserve-bf6cf6f67-gk428\" (UID: \"dcc8f099-7956-4362-af4d-75367a95d07f\") " pod="ai-tenant-e2e-noten-ac6878/noten-model-ac6878-kserve-bf6cf6f67-gk428"
Sep 09 02:49:09.776840 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:09.776808    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dcc8f099-7956-4362-af4d-75367a95d07f-tls-certs\") pod \"noten-model-ac6878-kserve-bf6cf6f67-gk428\" (UID: \"dcc8f099-7956-4362-af4d-75367a95d07f\") " pod="ai-tenant-e2e-noten-ac6878/noten-model-ac6878-kserve-bf6cf6f67-gk428"
Sep 09 02:49:09.783491 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:09.783473    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftnl7\" (UniqueName: \"kubernetes.io/projected/dcc8f099-7956-4362-af4d-75367a95d07f-kube-api-access-ftnl7\") pod \"noten-model-ac6878-kserve-bf6cf6f67-gk428\" (UID: \"dcc8f099-7956-4362-af4d-75367a95d07f\") " pod="ai-tenant-e2e-noten-ac6878/noten-model-ac6878-kserve-bf6cf6f67-gk428"
Sep 09 02:49:09.865551 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:09.865525    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-ac6878/noten-model-ac6878-kserve-bf6cf6f67-gk428"
Sep 09 02:49:09.949792 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:09.949753    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-99qjf"]
Sep 09 02:49:10.140228 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:10.140202    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-ac6878/noten-model-ac6878-kserve-bf6cf6f67-gk428"]
Sep 09 02:49:10.142076 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:49:10.142044    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddcc8f099_7956_4362_af4d_75367a95d07f.slice/crio-05762d67d4064800363142f3b51cc30972d224a77c712985eb1803114cc7ae7a WatchSource:0}: Error finding container 05762d67d4064800363142f3b51cc30972d224a77c712985eb1803114cc7ae7a: Status 404 returned error can't find the container with id 05762d67d4064800363142f3b51cc30972d224a77c712985eb1803114cc7ae7a
Sep 09 02:49:10.372253 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:10.372212    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-ac6878/noten-model-ac6878-kserve-bf6cf6f67-gk428" event={"ID":"dcc8f099-7956-4362-af4d-75367a95d07f","Type":"ContainerStarted","Data":"f095a5d7bf2ddd32288139f38b15a5b1beaa5a98718b971214bf7f8abad751aa"}
Sep 09 02:49:10.372416 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:10.372265    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-ac6878/noten-model-ac6878-kserve-bf6cf6f67-gk428" event={"ID":"dcc8f099-7956-4362-af4d-75367a95d07f","Type":"ContainerStarted","Data":"05762d67d4064800363142f3b51cc30972d224a77c712985eb1803114cc7ae7a"}
Sep 09 02:49:10.390452 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:10.390345    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-ac6878/noten-model-ac6878-kserve-bf6cf6f67-gk428" podStartSLOduration=1.390329951 podStartE2EDuration="1.390329951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:49:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:49:10.389100424 +0000 UTC m=+1132.643404060" watchObservedRunningTime="2026-09-09 02:49:10.390329951 +0000 UTC m=+1132.644633586"
Sep 09 02:49:19.866425 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:19.866378    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-ac6878/noten-model-ac6878-kserve-bf6cf6f67-gk428"
Sep 09 02:49:19.866830 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:19.866465    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-ac6878/noten-model-ac6878-kserve-bf6cf6f67-gk428"
Sep 09 02:49:19.879171 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:19.879145    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-ac6878/noten-model-ac6878-kserve-bf6cf6f67-gk428"
Sep 09 02:49:20.433944 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:20.433914    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-ac6878/noten-model-ac6878-kserve-bf6cf6f67-gk428"
Sep 09 02:49:24.308584 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:24.308547    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-99qjf"]
Sep 09 02:49:33.525817 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:33.525774    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-b18109/test-model-b18109-kserve-fd9cff9c9-7pjg2"]
Sep 09 02:49:34.823324 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:34.823291    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-ac6878/noten-model-ac6878-kserve-bf6cf6f67-gk428"]
Sep 09 02:49:36.723903 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:36.723872    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-6b11a2/test-model-6b11a2-kserve-76b7fc9c9f-pk6mx"]
Sep 09 02:49:36.855809 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:36.855775    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b18109-hp4l2"]
Sep 09 02:49:36.856169 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:36.856154    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48c61b4e-c170-4cbc-916a-efc2d6964697" containerName="istio-proxy"
Sep 09 02:49:36.856169 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:36.856171    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="48c61b4e-c170-4cbc-916a-efc2d6964697" containerName="istio-proxy"
Sep 09 02:49:36.856296 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:36.856243    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="48c61b4e-c170-4cbc-916a-efc2d6964697" containerName="istio-proxy"
Sep 09 02:49:36.860665 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:36.860634    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b18109-hp4l2"]
Sep 09 02:49:36.860813 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:36.860737    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b18109-hp4l2"
Sep 09 02:49:36.863916 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:36.863891    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vbr52\""
Sep 09 02:49:36.864063 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:36.863941    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 02:49:36.926782 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:36.926747    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3dc15c48-2cfc-417e-96fc-b0921febea28-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b18109-hp4l2\" (UID: \"3dc15c48-2cfc-417e-96fc-b0921febea28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b18109-hp4l2"
Sep 09 02:49:37.027627 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:37.027551    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3dc15c48-2cfc-417e-96fc-b0921febea28-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b18109-hp4l2\" (UID: \"3dc15c48-2cfc-417e-96fc-b0921febea28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b18109-hp4l2"
Sep 09 02:49:37.028169 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:37.028150    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3dc15c48-2cfc-417e-96fc-b0921febea28-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b18109-hp4l2\" (UID: \"3dc15c48-2cfc-417e-96fc-b0921febea28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b18109-hp4l2"
Sep 09 02:49:37.174453 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:37.174422    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b18109-hp4l2"
Sep 09 02:49:37.444754 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:37.444728    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b18109-hp4l2"]
Sep 09 02:49:37.446867 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:49:37.446839    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dc15c48_2cfc_417e_96fc_b0921febea28.slice/crio-802e9887b2c5990e996237cf4a0fe065277671ab43bf5feff488ac0b2549b260 WatchSource:0}: Error finding container 802e9887b2c5990e996237cf4a0fe065277671ab43bf5feff488ac0b2549b260: Status 404 returned error can't find the container with id 802e9887b2c5990e996237cf4a0fe065277671ab43bf5feff488ac0b2549b260
Sep 09 02:49:37.509267 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:37.509237    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b18109-hp4l2" event={"ID":"3dc15c48-2cfc-417e-96fc-b0921febea28","Type":"ContainerStarted","Data":"802e9887b2c5990e996237cf4a0fe065277671ab43bf5feff488ac0b2549b260"}
Sep 09 02:49:38.348245 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:38.348212    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-99qjf"]
Sep 09 02:49:38.515539 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:38.515463    2570 generic.go:356] "Generic (PLEG): container finished" podID="3dc15c48-2cfc-417e-96fc-b0921febea28" containerID="656cfaa336d42466098cdd1934d64d82b19e0d5d441012643f779c07c9388cdf" exitCode=0
Sep 09 02:49:38.515539 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:38.515493    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b18109-hp4l2" event={"ID":"3dc15c48-2cfc-417e-96fc-b0921febea28","Type":"ContainerDied","Data":"656cfaa336d42466098cdd1934d64d82b19e0d5d441012643f779c07c9388cdf"}
Sep 09 02:49:39.653477 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:39.653452    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b18109-hp4l2"
Sep 09 02:49:39.749632 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:39.749604    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3dc15c48-2cfc-417e-96fc-b0921febea28-maas-api-service-ca\") pod \"3dc15c48-2cfc-417e-96fc-b0921febea28\" (UID: \"3dc15c48-2cfc-417e-96fc-b0921febea28\") "
Sep 09 02:49:39.749939 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:39.749918    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3dc15c48-2cfc-417e-96fc-b0921febea28-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3dc15c48-2cfc-417e-96fc-b0921febea28" (UID: "3dc15c48-2cfc-417e-96fc-b0921febea28"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 02:49:39.850927 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:39.850901    2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3dc15c48-2cfc-417e-96fc-b0921febea28-maas-api-service-ca\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:49:40.526817 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:40.526738    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b18109-hp4l2"
Sep 09 02:49:40.526817 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:40.526783    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b18109-hp4l2" event={"ID":"3dc15c48-2cfc-417e-96fc-b0921febea28","Type":"ContainerDied","Data":"802e9887b2c5990e996237cf4a0fe065277671ab43bf5feff488ac0b2549b260"}
Sep 09 02:49:40.527016 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:40.526850    2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="802e9887b2c5990e996237cf4a0fe065277671ab43bf5feff488ac0b2549b260"
Sep 09 02:49:46.950999 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:46.946107    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-99qjf"]
Sep 09 02:49:47.532174 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:47.532125    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b18109-hp4l2"]
Sep 09 02:49:47.534437 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:47.534406    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b18109-hp4l2"]
Sep 09 02:49:48.223358 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:48.223321    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3dc15c48-2cfc-417e-96fc-b0921febea28" path="/var/lib/kubelet/pods/3dc15c48-2cfc-417e-96fc-b0921febea28/volumes"
Sep 09 02:49:48.545860 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:48.545745    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-b18109/test-model-b18109-kserve-fd9cff9c9-7pjg2" podUID="ec0b7347-6013-4976-8a01-977307fd97bc" containerName="main" containerID="cri-o://79189f8a8fb61d31e57ec7ae6d14d3d56cd91d2e53e5f0b20bc75b719e9af153" gracePeriod=45
Sep 09 02:49:48.800956 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:48.800896    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-b18109/test-model-b18109-kserve-fd9cff9c9-7pjg2"
Sep 09 02:49:48.928889 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:48.928856    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec0b7347-6013-4976-8a01-977307fd97bc-dshm\") pod \"ec0b7347-6013-4976-8a01-977307fd97bc\" (UID: \"ec0b7347-6013-4976-8a01-977307fd97bc\") "
Sep 09 02:49:48.929075 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:48.928903    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec0b7347-6013-4976-8a01-977307fd97bc-home\") pod \"ec0b7347-6013-4976-8a01-977307fd97bc\" (UID: \"ec0b7347-6013-4976-8a01-977307fd97bc\") "
Sep 09 02:49:48.929075 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:48.928934    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t2rqp\" (UniqueName: \"kubernetes.io/projected/ec0b7347-6013-4976-8a01-977307fd97bc-kube-api-access-t2rqp\") pod \"ec0b7347-6013-4976-8a01-977307fd97bc\" (UID: \"ec0b7347-6013-4976-8a01-977307fd97bc\") "
Sep 09 02:49:48.929075 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:48.928962    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec0b7347-6013-4976-8a01-977307fd97bc-model-cache\") pod \"ec0b7347-6013-4976-8a01-977307fd97bc\" (UID: \"ec0b7347-6013-4976-8a01-977307fd97bc\") "
Sep 09 02:49:48.929075 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:48.929014    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec0b7347-6013-4976-8a01-977307fd97bc-tls-certs\") pod \"ec0b7347-6013-4976-8a01-977307fd97bc\" (UID: \"ec0b7347-6013-4976-8a01-977307fd97bc\") "
Sep 09 02:49:48.929075 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:48.929038    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec0b7347-6013-4976-8a01-977307fd97bc-tmp-dir\") pod \"ec0b7347-6013-4976-8a01-977307fd97bc\" (UID: \"ec0b7347-6013-4976-8a01-977307fd97bc\") "
Sep 09 02:49:48.929362 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:48.929192    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec0b7347-6013-4976-8a01-977307fd97bc-home" (OuterVolumeSpecName: "home") pod "ec0b7347-6013-4976-8a01-977307fd97bc" (UID: "ec0b7347-6013-4976-8a01-977307fd97bc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 02:49:48.929362 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:48.929282    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec0b7347-6013-4976-8a01-977307fd97bc-model-cache" (OuterVolumeSpecName: "model-cache") pod "ec0b7347-6013-4976-8a01-977307fd97bc" (UID: "ec0b7347-6013-4976-8a01-977307fd97bc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 02:49:48.929362 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:48.929349    2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec0b7347-6013-4976-8a01-977307fd97bc-home\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:49:48.929474 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:48.929364    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec0b7347-6013-4976-8a01-977307fd97bc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ec0b7347-6013-4976-8a01-977307fd97bc" (UID: "ec0b7347-6013-4976-8a01-977307fd97bc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 02:49:48.931196 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:48.931177    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec0b7347-6013-4976-8a01-977307fd97bc-dshm" (OuterVolumeSpecName: "dshm") pod "ec0b7347-6013-4976-8a01-977307fd97bc" (UID: "ec0b7347-6013-4976-8a01-977307fd97bc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 02:49:48.931615 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:48.931584    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec0b7347-6013-4976-8a01-977307fd97bc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ec0b7347-6013-4976-8a01-977307fd97bc" (UID: "ec0b7347-6013-4976-8a01-977307fd97bc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 02:49:48.931714 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:48.931624    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec0b7347-6013-4976-8a01-977307fd97bc-kube-api-access-t2rqp" (OuterVolumeSpecName: "kube-api-access-t2rqp") pod "ec0b7347-6013-4976-8a01-977307fd97bc" (UID: "ec0b7347-6013-4976-8a01-977307fd97bc"). InnerVolumeSpecName "kube-api-access-t2rqp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 02:49:49.030151 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:49.030123    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2rqp\" (UniqueName: \"kubernetes.io/projected/ec0b7347-6013-4976-8a01-977307fd97bc-kube-api-access-t2rqp\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:49:49.030151 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:49.030145    2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec0b7347-6013-4976-8a01-977307fd97bc-model-cache\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:49:49.030151 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:49.030154    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec0b7347-6013-4976-8a01-977307fd97bc-tls-certs\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:49:49.030333 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:49.030162    2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec0b7347-6013-4976-8a01-977307fd97bc-tmp-dir\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:49:49.030333 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:49.030169    2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec0b7347-6013-4976-8a01-977307fd97bc-dshm\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:49:49.571442 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:49.571404    2570 generic.go:356] "Generic (PLEG): container finished" podID="ec0b7347-6013-4976-8a01-977307fd97bc" containerID="79189f8a8fb61d31e57ec7ae6d14d3d56cd91d2e53e5f0b20bc75b719e9af153" exitCode=0
Sep 09 02:49:49.571840 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:49.571451    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-b18109/test-model-b18109-kserve-fd9cff9c9-7pjg2"
Sep 09 02:49:49.571840 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:49.571486    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-b18109/test-model-b18109-kserve-fd9cff9c9-7pjg2" event={"ID":"ec0b7347-6013-4976-8a01-977307fd97bc","Type":"ContainerDied","Data":"79189f8a8fb61d31e57ec7ae6d14d3d56cd91d2e53e5f0b20bc75b719e9af153"}
Sep 09 02:49:49.571840 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:49.571531    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-b18109/test-model-b18109-kserve-fd9cff9c9-7pjg2" event={"ID":"ec0b7347-6013-4976-8a01-977307fd97bc","Type":"ContainerDied","Data":"27726124a3207ea8def64bdbd2428a43e4ecbbc56be905c9fc8368699903a0cd"}
Sep 09 02:49:49.571840 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:49.571548    2570 scope.go:117] "RemoveContainer" containerID="79189f8a8fb61d31e57ec7ae6d14d3d56cd91d2e53e5f0b20bc75b719e9af153"
Sep 09 02:49:49.585323 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:49.585307    2570 scope.go:117] "RemoveContainer" containerID="79189f8a8fb61d31e57ec7ae6d14d3d56cd91d2e53e5f0b20bc75b719e9af153"
Sep 09 02:49:49.585603 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:49:49.585583    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79189f8a8fb61d31e57ec7ae6d14d3d56cd91d2e53e5f0b20bc75b719e9af153\": container with ID starting with 79189f8a8fb61d31e57ec7ae6d14d3d56cd91d2e53e5f0b20bc75b719e9af153 not found: ID does not exist" containerID="79189f8a8fb61d31e57ec7ae6d14d3d56cd91d2e53e5f0b20bc75b719e9af153"
Sep 09 02:49:49.585655 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:49.585612    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79189f8a8fb61d31e57ec7ae6d14d3d56cd91d2e53e5f0b20bc75b719e9af153"} err="failed to get container status \"79189f8a8fb61d31e57ec7ae6d14d3d56cd91d2e53e5f0b20bc75b719e9af153\": rpc error: code = NotFound desc = could not find container \"79189f8a8fb61d31e57ec7ae6d14d3d56cd91d2e53e5f0b20bc75b719e9af153\": container with ID starting with 79189f8a8fb61d31e57ec7ae6d14d3d56cd91d2e53e5f0b20bc75b719e9af153 not found: ID does not exist"
Sep 09 02:49:49.597111 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:49.597080    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-b18109/test-model-b18109-kserve-fd9cff9c9-7pjg2"]
Sep 09 02:49:49.602473 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:49.602450    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-b18109/test-model-b18109-kserve-fd9cff9c9-7pjg2"]
Sep 09 02:49:49.843246 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:49.843193    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-ac6878/noten-model-ac6878-kserve-bf6cf6f67-gk428" podUID="dcc8f099-7956-4362-af4d-75367a95d07f" containerName="main" containerID="cri-o://f095a5d7bf2ddd32288139f38b15a5b1beaa5a98718b971214bf7f8abad751aa" gracePeriod=45
Sep 09 02:49:50.094764 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:50.094706    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-ac6878/noten-model-ac6878-kserve-bf6cf6f67-gk428"
Sep 09 02:49:50.222991 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:50.222948    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec0b7347-6013-4976-8a01-977307fd97bc" path="/var/lib/kubelet/pods/ec0b7347-6013-4976-8a01-977307fd97bc/volumes"
Sep 09 02:49:50.240009 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:50.239963    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dcc8f099-7956-4362-af4d-75367a95d07f-tls-certs\") pod \"dcc8f099-7956-4362-af4d-75367a95d07f\" (UID: \"dcc8f099-7956-4362-af4d-75367a95d07f\") "
Sep 09 02:49:50.240166 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:50.240037    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ftnl7\" (UniqueName: \"kubernetes.io/projected/dcc8f099-7956-4362-af4d-75367a95d07f-kube-api-access-ftnl7\") pod \"dcc8f099-7956-4362-af4d-75367a95d07f\" (UID: \"dcc8f099-7956-4362-af4d-75367a95d07f\") "
Sep 09 02:49:50.240166 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:50.240077    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dcc8f099-7956-4362-af4d-75367a95d07f-model-cache\") pod \"dcc8f099-7956-4362-af4d-75367a95d07f\" (UID: \"dcc8f099-7956-4362-af4d-75367a95d07f\") "
Sep 09 02:49:50.240166 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:50.240108    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dcc8f099-7956-4362-af4d-75367a95d07f-tmp-dir\") pod \"dcc8f099-7956-4362-af4d-75367a95d07f\" (UID: \"dcc8f099-7956-4362-af4d-75367a95d07f\") "
Sep 09 02:49:50.240166 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:50.240162    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dcc8f099-7956-4362-af4d-75367a95d07f-dshm\") pod \"dcc8f099-7956-4362-af4d-75367a95d07f\" (UID: \"dcc8f099-7956-4362-af4d-75367a95d07f\") "
Sep 09 02:49:50.240386 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:50.240211    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dcc8f099-7956-4362-af4d-75367a95d07f-home\") pod \"dcc8f099-7956-4362-af4d-75367a95d07f\" (UID: \"dcc8f099-7956-4362-af4d-75367a95d07f\") "
Sep 09 02:49:50.240386 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:50.240347    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcc8f099-7956-4362-af4d-75367a95d07f-model-cache" (OuterVolumeSpecName: "model-cache") pod "dcc8f099-7956-4362-af4d-75367a95d07f" (UID: "dcc8f099-7956-4362-af4d-75367a95d07f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 02:49:50.240472 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:50.240393    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcc8f099-7956-4362-af4d-75367a95d07f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "dcc8f099-7956-4362-af4d-75367a95d07f" (UID: "dcc8f099-7956-4362-af4d-75367a95d07f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 02:49:50.240562 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:50.240532    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcc8f099-7956-4362-af4d-75367a95d07f-home" (OuterVolumeSpecName: "home") pod "dcc8f099-7956-4362-af4d-75367a95d07f" (UID: "dcc8f099-7956-4362-af4d-75367a95d07f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 02:49:50.240689 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:50.240633    2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dcc8f099-7956-4362-af4d-75367a95d07f-model-cache\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:49:50.240689 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:50.240653    2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dcc8f099-7956-4362-af4d-75367a95d07f-tmp-dir\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:49:50.240689 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:50.240665    2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dcc8f099-7956-4362-af4d-75367a95d07f-home\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:49:50.242385 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:50.242359    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcc8f099-7956-4362-af4d-75367a95d07f-kube-api-access-ftnl7" (OuterVolumeSpecName: "kube-api-access-ftnl7") pod "dcc8f099-7956-4362-af4d-75367a95d07f" (UID: "dcc8f099-7956-4362-af4d-75367a95d07f"). InnerVolumeSpecName "kube-api-access-ftnl7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 02:49:50.242385 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:50.242374    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcc8f099-7956-4362-af4d-75367a95d07f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dcc8f099-7956-4362-af4d-75367a95d07f" (UID: "dcc8f099-7956-4362-af4d-75367a95d07f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 02:49:50.242515 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:50.242383    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcc8f099-7956-4362-af4d-75367a95d07f-dshm" (OuterVolumeSpecName: "dshm") pod "dcc8f099-7956-4362-af4d-75367a95d07f" (UID: "dcc8f099-7956-4362-af4d-75367a95d07f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 02:49:50.341359 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:50.341317    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dcc8f099-7956-4362-af4d-75367a95d07f-tls-certs\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:49:50.341539 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:50.341476    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftnl7\" (UniqueName: \"kubernetes.io/projected/dcc8f099-7956-4362-af4d-75367a95d07f-kube-api-access-ftnl7\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:49:50.341539 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:50.341509    2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dcc8f099-7956-4362-af4d-75367a95d07f-dshm\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:49:50.580134 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:50.580052    2570 generic.go:356] "Generic (PLEG): container finished" podID="dcc8f099-7956-4362-af4d-75367a95d07f" containerID="f095a5d7bf2ddd32288139f38b15a5b1beaa5a98718b971214bf7f8abad751aa" exitCode=0
Sep 09 02:49:50.580134 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:50.580100    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-ac6878/noten-model-ac6878-kserve-bf6cf6f67-gk428"
Sep 09 02:49:50.580600 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:50.580131    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-ac6878/noten-model-ac6878-kserve-bf6cf6f67-gk428" event={"ID":"dcc8f099-7956-4362-af4d-75367a95d07f","Type":"ContainerDied","Data":"f095a5d7bf2ddd32288139f38b15a5b1beaa5a98718b971214bf7f8abad751aa"}
Sep 09 02:49:50.580600 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:50.580189    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-ac6878/noten-model-ac6878-kserve-bf6cf6f67-gk428" event={"ID":"dcc8f099-7956-4362-af4d-75367a95d07f","Type":"ContainerDied","Data":"05762d67d4064800363142f3b51cc30972d224a77c712985eb1803114cc7ae7a"}
Sep 09 02:49:50.580600 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:50.580213    2570 scope.go:117] "RemoveContainer" containerID="f095a5d7bf2ddd32288139f38b15a5b1beaa5a98718b971214bf7f8abad751aa"
Sep 09 02:49:50.591698 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:50.591672    2570 scope.go:117] "RemoveContainer" containerID="f095a5d7bf2ddd32288139f38b15a5b1beaa5a98718b971214bf7f8abad751aa"
Sep 09 02:49:50.592023 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:49:50.591996    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f095a5d7bf2ddd32288139f38b15a5b1beaa5a98718b971214bf7f8abad751aa\": container with ID starting with f095a5d7bf2ddd32288139f38b15a5b1beaa5a98718b971214bf7f8abad751aa not found: ID does not exist" containerID="f095a5d7bf2ddd32288139f38b15a5b1beaa5a98718b971214bf7f8abad751aa"
Sep 09 02:49:50.592131 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:50.592034    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f095a5d7bf2ddd32288139f38b15a5b1beaa5a98718b971214bf7f8abad751aa"} err="failed to get container status \"f095a5d7bf2ddd32288139f38b15a5b1beaa5a98718b971214bf7f8abad751aa\": rpc error: code = NotFound desc = could not find container \"f095a5d7bf2ddd32288139f38b15a5b1beaa5a98718b971214bf7f8abad751aa\": container with ID starting with f095a5d7bf2ddd32288139f38b15a5b1beaa5a98718b971214bf7f8abad751aa not found: ID does not exist"
Sep 09 02:49:50.610802 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:50.610769    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-ac6878/noten-model-ac6878-kserve-bf6cf6f67-gk428"]
Sep 09 02:49:50.615292 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:50.615263    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-ac6878/noten-model-ac6878-kserve-bf6cf6f67-gk428"]
Sep 09 02:49:51.743770 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:51.743731    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-6b11a2/test-model-6b11a2-kserve-76b7fc9c9f-pk6mx" podUID="0447baeb-2218-40f2-9a4f-47102c50b8ed" containerName="main" containerID="cri-o://dfe485c197a7e9018cf11b9191a37c44862795302b54a6c52a0bf81603058386" gracePeriod=45
Sep 09 02:49:51.995469 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:51.995417    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-6b11a2/test-model-6b11a2-kserve-76b7fc9c9f-pk6mx"
Sep 09 02:49:52.159269 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:52.159236    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0447baeb-2218-40f2-9a4f-47102c50b8ed-model-cache\") pod \"0447baeb-2218-40f2-9a4f-47102c50b8ed\" (UID: \"0447baeb-2218-40f2-9a4f-47102c50b8ed\") "
Sep 09 02:49:52.159438 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:52.159285    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0447baeb-2218-40f2-9a4f-47102c50b8ed-home\") pod \"0447baeb-2218-40f2-9a4f-47102c50b8ed\" (UID: \"0447baeb-2218-40f2-9a4f-47102c50b8ed\") "
Sep 09 02:49:52.159438 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:52.159311    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0447baeb-2218-40f2-9a4f-47102c50b8ed-dshm\") pod \"0447baeb-2218-40f2-9a4f-47102c50b8ed\" (UID: \"0447baeb-2218-40f2-9a4f-47102c50b8ed\") "
Sep 09 02:49:52.159438 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:52.159392    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0447baeb-2218-40f2-9a4f-47102c50b8ed-tmp-dir\") pod \"0447baeb-2218-40f2-9a4f-47102c50b8ed\" (UID: \"0447baeb-2218-40f2-9a4f-47102c50b8ed\") "
Sep 09 02:49:52.159438 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:52.159433    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0447baeb-2218-40f2-9a4f-47102c50b8ed-tls-certs\") pod \"0447baeb-2218-40f2-9a4f-47102c50b8ed\" (UID: \"0447baeb-2218-40f2-9a4f-47102c50b8ed\") "
Sep 09 02:49:52.159613 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:52.159493    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ndvrh\" (UniqueName: \"kubernetes.io/projected/0447baeb-2218-40f2-9a4f-47102c50b8ed-kube-api-access-ndvrh\") pod \"0447baeb-2218-40f2-9a4f-47102c50b8ed\" (UID: \"0447baeb-2218-40f2-9a4f-47102c50b8ed\") "
Sep 09 02:49:52.159613 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:52.159545    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0447baeb-2218-40f2-9a4f-47102c50b8ed-model-cache" (OuterVolumeSpecName: "model-cache") pod "0447baeb-2218-40f2-9a4f-47102c50b8ed" (UID: "0447baeb-2218-40f2-9a4f-47102c50b8ed"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 02:49:52.159613 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:52.159555    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0447baeb-2218-40f2-9a4f-47102c50b8ed-home" (OuterVolumeSpecName: "home") pod "0447baeb-2218-40f2-9a4f-47102c50b8ed" (UID: "0447baeb-2218-40f2-9a4f-47102c50b8ed"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 02:49:52.159775 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:52.159677    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0447baeb-2218-40f2-9a4f-47102c50b8ed-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0447baeb-2218-40f2-9a4f-47102c50b8ed" (UID: "0447baeb-2218-40f2-9a4f-47102c50b8ed"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 02:49:52.159827 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:52.159795    2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0447baeb-2218-40f2-9a4f-47102c50b8ed-tmp-dir\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:49:52.159827 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:52.159816    2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0447baeb-2218-40f2-9a4f-47102c50b8ed-model-cache\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:49:52.159936 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:52.159831    2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0447baeb-2218-40f2-9a4f-47102c50b8ed-home\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:49:52.161608 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:52.161578    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0447baeb-2218-40f2-9a4f-47102c50b8ed-dshm" (OuterVolumeSpecName: "dshm") pod "0447baeb-2218-40f2-9a4f-47102c50b8ed" (UID: "0447baeb-2218-40f2-9a4f-47102c50b8ed"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 02:49:52.161715 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:52.161641    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0447baeb-2218-40f2-9a4f-47102c50b8ed-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0447baeb-2218-40f2-9a4f-47102c50b8ed" (UID: "0447baeb-2218-40f2-9a4f-47102c50b8ed"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 02:49:52.161715 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:52.161705    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0447baeb-2218-40f2-9a4f-47102c50b8ed-kube-api-access-ndvrh" (OuterVolumeSpecName: "kube-api-access-ndvrh") pod "0447baeb-2218-40f2-9a4f-47102c50b8ed" (UID: "0447baeb-2218-40f2-9a4f-47102c50b8ed"). InnerVolumeSpecName "kube-api-access-ndvrh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 02:49:52.222395 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:52.222369    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dcc8f099-7956-4362-af4d-75367a95d07f" path="/var/lib/kubelet/pods/dcc8f099-7956-4362-af4d-75367a95d07f/volumes"
Sep 09 02:49:52.260669 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:52.260602    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndvrh\" (UniqueName: \"kubernetes.io/projected/0447baeb-2218-40f2-9a4f-47102c50b8ed-kube-api-access-ndvrh\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:49:52.260669 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:52.260626    2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0447baeb-2218-40f2-9a4f-47102c50b8ed-dshm\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:49:52.260669 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:52.260636    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0447baeb-2218-40f2-9a4f-47102c50b8ed-tls-certs\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:49:52.593241 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:52.593211    2570 generic.go:356] "Generic (PLEG): container finished" podID="0447baeb-2218-40f2-9a4f-47102c50b8ed" containerID="dfe485c197a7e9018cf11b9191a37c44862795302b54a6c52a0bf81603058386" exitCode=0
Sep 09 02:49:52.593419 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:52.593257    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-6b11a2/test-model-6b11a2-kserve-76b7fc9c9f-pk6mx"
Sep 09 02:49:52.593419 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:52.593298    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-6b11a2/test-model-6b11a2-kserve-76b7fc9c9f-pk6mx" event={"ID":"0447baeb-2218-40f2-9a4f-47102c50b8ed","Type":"ContainerDied","Data":"dfe485c197a7e9018cf11b9191a37c44862795302b54a6c52a0bf81603058386"}
Sep 09 02:49:52.593419 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:52.593334    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-6b11a2/test-model-6b11a2-kserve-76b7fc9c9f-pk6mx" event={"ID":"0447baeb-2218-40f2-9a4f-47102c50b8ed","Type":"ContainerDied","Data":"6160b3230e5b91d7c7d64b5584eeb54c93511ef07c42c230e016864d5052d36b"}
Sep 09 02:49:52.593419 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:52.593353    2570 scope.go:117] "RemoveContainer" containerID="dfe485c197a7e9018cf11b9191a37c44862795302b54a6c52a0bf81603058386"
Sep 09 02:49:52.601920 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:52.601903    2570 scope.go:117] "RemoveContainer" containerID="dfe485c197a7e9018cf11b9191a37c44862795302b54a6c52a0bf81603058386"
Sep 09 02:49:52.602218 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:49:52.602200    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfe485c197a7e9018cf11b9191a37c44862795302b54a6c52a0bf81603058386\": container with ID starting with dfe485c197a7e9018cf11b9191a37c44862795302b54a6c52a0bf81603058386 not found: ID does not exist" containerID="dfe485c197a7e9018cf11b9191a37c44862795302b54a6c52a0bf81603058386"
Sep 09 02:49:52.602273 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:52.602228    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfe485c197a7e9018cf11b9191a37c44862795302b54a6c52a0bf81603058386"} err="failed to get container status \"dfe485c197a7e9018cf11b9191a37c44862795302b54a6c52a0bf81603058386\": rpc error: code = NotFound desc = could not find container \"dfe485c197a7e9018cf11b9191a37c44862795302b54a6c52a0bf81603058386\": container with ID starting with dfe485c197a7e9018cf11b9191a37c44862795302b54a6c52a0bf81603058386 not found: ID does not exist"
Sep 09 02:49:52.610783 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:52.610761    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-6b11a2/test-model-6b11a2-kserve-76b7fc9c9f-pk6mx"]
Sep 09 02:49:52.617505 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:52.617482    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-6b11a2/test-model-6b11a2-kserve-76b7fc9c9f-pk6mx"]
Sep 09 02:49:54.224988 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:54.224943    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0447baeb-2218-40f2-9a4f-47102c50b8ed" path="/var/lib/kubelet/pods/0447baeb-2218-40f2-9a4f-47102c50b8ed/volumes"
Sep 09 02:49:54.591880 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:54.591851    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6b11a2-ffm7q"]
Sep 09 02:49:54.592207 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:54.592193    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0447baeb-2218-40f2-9a4f-47102c50b8ed" containerName="main"
Sep 09 02:49:54.592207 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:54.592208    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="0447baeb-2218-40f2-9a4f-47102c50b8ed" containerName="main"
Sep 09 02:49:54.592287 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:54.592221    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dcc8f099-7956-4362-af4d-75367a95d07f" containerName="main"
Sep 09 02:49:54.592287 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:54.592226    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="dcc8f099-7956-4362-af4d-75367a95d07f" containerName="main"
Sep 09 02:49:54.592287 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:54.592236    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec0b7347-6013-4976-8a01-977307fd97bc" containerName="main"
Sep 09 02:49:54.592287 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:54.592241    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec0b7347-6013-4976-8a01-977307fd97bc" containerName="main"
Sep 09 02:49:54.592287 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:54.592249    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3dc15c48-2cfc-417e-96fc-b0921febea28" containerName="revoke-keys"
Sep 09 02:49:54.592287 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:54.592255    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="3dc15c48-2cfc-417e-96fc-b0921febea28" containerName="revoke-keys"
Sep 09 02:49:54.592468 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:54.592312    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="dcc8f099-7956-4362-af4d-75367a95d07f" containerName="main"
Sep 09 02:49:54.592468 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:54.592323    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec0b7347-6013-4976-8a01-977307fd97bc" containerName="main"
Sep 09 02:49:54.592468 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:54.592330    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="0447baeb-2218-40f2-9a4f-47102c50b8ed" containerName="main"
Sep 09 02:49:54.592468 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:54.592336    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="3dc15c48-2cfc-417e-96fc-b0921febea28" containerName="revoke-keys"
Sep 09 02:49:54.596634 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:54.596618    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6b11a2-ffm7q"
Sep 09 02:49:54.601798 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:54.601774    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 02:49:54.601901 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:54.601797    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vbr52\""
Sep 09 02:49:54.607750 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:54.607727    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6b11a2-ffm7q"]
Sep 09 02:49:54.783265 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:54.783228    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5db50852-b05b-4194-92de-07e973eb7c16-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-6b11a2-ffm7q\" (UID: \"5db50852-b05b-4194-92de-07e973eb7c16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6b11a2-ffm7q"
Sep 09 02:49:54.884735 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:54.884658    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5db50852-b05b-4194-92de-07e973eb7c16-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-6b11a2-ffm7q\" (UID: \"5db50852-b05b-4194-92de-07e973eb7c16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6b11a2-ffm7q"
Sep 09 02:49:54.885393 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:54.885375    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5db50852-b05b-4194-92de-07e973eb7c16-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-6b11a2-ffm7q\" (UID: \"5db50852-b05b-4194-92de-07e973eb7c16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6b11a2-ffm7q"
Sep 09 02:49:54.907690 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:54.907671    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6b11a2-ffm7q"
Sep 09 02:49:55.061260 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:55.061229    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-99qjf"]
Sep 09 02:49:55.176195 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:55.176172    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6b11a2-ffm7q"]
Sep 09 02:49:55.178072 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:49:55.178045    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5db50852_b05b_4194_92de_07e973eb7c16.slice/crio-d2c3552ba560e8b8375cac08011e61468cd1063685fb92ed2504f3efd2526553 WatchSource:0}: Error finding container d2c3552ba560e8b8375cac08011e61468cd1063685fb92ed2504f3efd2526553: Status 404 returned error can't find the container with id d2c3552ba560e8b8375cac08011e61468cd1063685fb92ed2504f3efd2526553
Sep 09 02:49:55.612175 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:55.612134    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6b11a2-ffm7q" event={"ID":"5db50852-b05b-4194-92de-07e973eb7c16","Type":"ContainerStarted","Data":"de3906f816701a277ea78083186b07ac0048ec79c11de1132b391d723d23bbc5"}
Sep 09 02:49:55.612542 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:55.612226    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6b11a2-ffm7q" event={"ID":"5db50852-b05b-4194-92de-07e973eb7c16","Type":"ContainerStarted","Data":"d2c3552ba560e8b8375cac08011e61468cd1063685fb92ed2504f3efd2526553"}
Sep 09 02:49:55.630656 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:55.630592    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6b11a2-ffm7q" podStartSLOduration=1.630567112 podStartE2EDuration="1.630567112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:49:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:49:55.628758793 +0000 UTC m=+1177.883062430" watchObservedRunningTime="2026-09-09 02:49:55.630567112 +0000 UTC m=+1177.884870746"
Sep 09 02:49:56.618831 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:56.618795    2570 generic.go:356] "Generic (PLEG): container finished" podID="5db50852-b05b-4194-92de-07e973eb7c16" containerID="de3906f816701a277ea78083186b07ac0048ec79c11de1132b391d723d23bbc5" exitCode=0
Sep 09 02:49:56.619270 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:56.618879    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6b11a2-ffm7q" event={"ID":"5db50852-b05b-4194-92de-07e973eb7c16","Type":"ContainerDied","Data":"de3906f816701a277ea78083186b07ac0048ec79c11de1132b391d723d23bbc5"}
Sep 09 02:49:57.753566 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:57.753540    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6b11a2-ffm7q"
Sep 09 02:49:57.810925 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:57.810896    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5db50852-b05b-4194-92de-07e973eb7c16-maas-api-service-ca\") pod \"5db50852-b05b-4194-92de-07e973eb7c16\" (UID: \"5db50852-b05b-4194-92de-07e973eb7c16\") "
Sep 09 02:49:57.811226 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:57.811204    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5db50852-b05b-4194-92de-07e973eb7c16-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5db50852-b05b-4194-92de-07e973eb7c16" (UID: "5db50852-b05b-4194-92de-07e973eb7c16"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 02:49:57.911904 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:57.911836    2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5db50852-b05b-4194-92de-07e973eb7c16-maas-api-service-ca\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:49:58.629611 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:58.629582    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6b11a2-ffm7q"
Sep 09 02:49:58.629800 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:58.629624    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6b11a2-ffm7q" event={"ID":"5db50852-b05b-4194-92de-07e973eb7c16","Type":"ContainerDied","Data":"d2c3552ba560e8b8375cac08011e61468cd1063685fb92ed2504f3efd2526553"}
Sep 09 02:49:58.629800 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:49:58.629668    2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d2c3552ba560e8b8375cac08011e61468cd1063685fb92ed2504f3efd2526553"
Sep 09 02:50:05.662836 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:05.662788    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6b11a2-ffm7q"]
Sep 09 02:50:05.665052 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:05.665030    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6b11a2-ffm7q"]
Sep 09 02:50:06.222710 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:06.222672    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5db50852-b05b-4194-92de-07e973eb7c16" path="/var/lib/kubelet/pods/5db50852-b05b-4194-92de-07e973eb7c16/volumes"
Sep 09 02:50:09.351528 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:09.351494    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-99qjf"]
Sep 09 02:50:12.364162 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:12.364125    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-6b11a2-openshift-default-5cfbd4b94-fql7q"]
Sep 09 02:50:12.364735 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:12.364339    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-6b11a2-openshift-default-5cfbd4b94-fql7q" podUID="2a7cadb9-da51-457d-8c72-007236378887" containerName="istio-proxy" containerID="cri-o://71a36d8103aa77732e1409b7531eace16cb2840c4c9676bdbcc7bfc95dcce12d" gracePeriod=30
Sep 09 02:50:17.615119 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:17.615095    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6b11a2-openshift-default-5cfbd4b94-fql7q"
Sep 09 02:50:17.690473 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:17.690396    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5hzlc\" (UniqueName: \"kubernetes.io/projected/2a7cadb9-da51-457d-8c72-007236378887-kube-api-access-5hzlc\") pod \"2a7cadb9-da51-457d-8c72-007236378887\" (UID: \"2a7cadb9-da51-457d-8c72-007236378887\") "
Sep 09 02:50:17.690473 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:17.690435    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a7cadb9-da51-457d-8c72-007236378887-istio-podinfo\") pod \"2a7cadb9-da51-457d-8c72-007236378887\" (UID: \"2a7cadb9-da51-457d-8c72-007236378887\") "
Sep 09 02:50:17.690656 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:17.690476    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a7cadb9-da51-457d-8c72-007236378887-credential-socket\") pod \"2a7cadb9-da51-457d-8c72-007236378887\" (UID: \"2a7cadb9-da51-457d-8c72-007236378887\") "
Sep 09 02:50:17.690656 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:17.690511    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a7cadb9-da51-457d-8c72-007236378887-workload-certs\") pod \"2a7cadb9-da51-457d-8c72-007236378887\" (UID: \"2a7cadb9-da51-457d-8c72-007236378887\") "
Sep 09 02:50:17.690656 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:17.690532    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a7cadb9-da51-457d-8c72-007236378887-istio-token\") pod \"2a7cadb9-da51-457d-8c72-007236378887\" (UID: \"2a7cadb9-da51-457d-8c72-007236378887\") "
Sep 09 02:50:17.690656 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:17.690574    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a7cadb9-da51-457d-8c72-007236378887-workload-socket\") pod \"2a7cadb9-da51-457d-8c72-007236378887\" (UID: \"2a7cadb9-da51-457d-8c72-007236378887\") "
Sep 09 02:50:17.690855 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:17.690659    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a7cadb9-da51-457d-8c72-007236378887-istiod-ca-cert\") pod \"2a7cadb9-da51-457d-8c72-007236378887\" (UID: \"2a7cadb9-da51-457d-8c72-007236378887\") "
Sep 09 02:50:17.690855 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:17.690683    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a7cadb9-da51-457d-8c72-007236378887-istio-data\") pod \"2a7cadb9-da51-457d-8c72-007236378887\" (UID: \"2a7cadb9-da51-457d-8c72-007236378887\") "
Sep 09 02:50:17.690855 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:17.690705    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a7cadb9-da51-457d-8c72-007236378887-istio-envoy\") pod \"2a7cadb9-da51-457d-8c72-007236378887\" (UID: \"2a7cadb9-da51-457d-8c72-007236378887\") "
Sep 09 02:50:17.690855 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:17.690816    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a7cadb9-da51-457d-8c72-007236378887-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2a7cadb9-da51-457d-8c72-007236378887" (UID: "2a7cadb9-da51-457d-8c72-007236378887"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 02:50:17.691093 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:17.690901    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a7cadb9-da51-457d-8c72-007236378887-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2a7cadb9-da51-457d-8c72-007236378887" (UID: "2a7cadb9-da51-457d-8c72-007236378887"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 02:50:17.691151 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:17.691095    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a7cadb9-da51-457d-8c72-007236378887-istio-data" (OuterVolumeSpecName: "istio-data") pod "2a7cadb9-da51-457d-8c72-007236378887" (UID: "2a7cadb9-da51-457d-8c72-007236378887"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 02:50:17.691151 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:17.691103    2570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a7cadb9-da51-457d-8c72-007236378887-credential-socket\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:50:17.691151 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:17.691124    2570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a7cadb9-da51-457d-8c72-007236378887-workload-certs\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:50:17.691308 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:17.691272    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a7cadb9-da51-457d-8c72-007236378887-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2a7cadb9-da51-457d-8c72-007236378887" (UID: "2a7cadb9-da51-457d-8c72-007236378887"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 02:50:17.691374 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:17.691337    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a7cadb9-da51-457d-8c72-007236378887-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2a7cadb9-da51-457d-8c72-007236378887" (UID: "2a7cadb9-da51-457d-8c72-007236378887"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 02:50:17.692732 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:17.692710    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a7cadb9-da51-457d-8c72-007236378887-istio-token" (OuterVolumeSpecName: "istio-token") pod "2a7cadb9-da51-457d-8c72-007236378887" (UID: "2a7cadb9-da51-457d-8c72-007236378887"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 02:50:17.692896 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:17.692881    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a7cadb9-da51-457d-8c72-007236378887-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2a7cadb9-da51-457d-8c72-007236378887" (UID: "2a7cadb9-da51-457d-8c72-007236378887"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 02:50:17.693145 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:17.693116    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a7cadb9-da51-457d-8c72-007236378887-kube-api-access-5hzlc" (OuterVolumeSpecName: "kube-api-access-5hzlc") pod "2a7cadb9-da51-457d-8c72-007236378887" (UID: "2a7cadb9-da51-457d-8c72-007236378887"). InnerVolumeSpecName "kube-api-access-5hzlc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 02:50:17.693145 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:17.693111    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2a7cadb9-da51-457d-8c72-007236378887-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2a7cadb9-da51-457d-8c72-007236378887" (UID: "2a7cadb9-da51-457d-8c72-007236378887"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 02:50:17.732175 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:17.732136    2570 generic.go:356] "Generic (PLEG): container finished" podID="2a7cadb9-da51-457d-8c72-007236378887" containerID="71a36d8103aa77732e1409b7531eace16cb2840c4c9676bdbcc7bfc95dcce12d" exitCode=0
Sep 09 02:50:17.732315 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:17.732185    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6b11a2-openshift-default-5cfbd4b94-fql7q"
Sep 09 02:50:17.732315 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:17.732211    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6b11a2-openshift-default-5cfbd4b94-fql7q" event={"ID":"2a7cadb9-da51-457d-8c72-007236378887","Type":"ContainerDied","Data":"71a36d8103aa77732e1409b7531eace16cb2840c4c9676bdbcc7bfc95dcce12d"}
Sep 09 02:50:17.732315 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:17.732274    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6b11a2-openshift-default-5cfbd4b94-fql7q" event={"ID":"2a7cadb9-da51-457d-8c72-007236378887","Type":"ContainerDied","Data":"6c2b0754b95df1935288f568067eeb9fda0b38ef433569d961eba66e307fc040"}
Sep 09 02:50:17.732315 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:17.732306    2570 scope.go:117] "RemoveContainer" containerID="71a36d8103aa77732e1409b7531eace16cb2840c4c9676bdbcc7bfc95dcce12d"
Sep 09 02:50:17.743455 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:17.743441    2570 scope.go:117] "RemoveContainer" containerID="a87e064bd39e86b069a49c445832c8eccd611de6b23454ca5c136842685b86c4"
Sep 09 02:50:17.751189 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:17.751173    2570 scope.go:117] "RemoveContainer" containerID="71a36d8103aa77732e1409b7531eace16cb2840c4c9676bdbcc7bfc95dcce12d"
Sep 09 02:50:17.751438 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:50:17.751419    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71a36d8103aa77732e1409b7531eace16cb2840c4c9676bdbcc7bfc95dcce12d\": container with ID starting with 71a36d8103aa77732e1409b7531eace16cb2840c4c9676bdbcc7bfc95dcce12d not found: ID does not exist" containerID="71a36d8103aa77732e1409b7531eace16cb2840c4c9676bdbcc7bfc95dcce12d"
Sep 09 02:50:17.751518 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:17.751451    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71a36d8103aa77732e1409b7531eace16cb2840c4c9676bdbcc7bfc95dcce12d"} err="failed to get container status \"71a36d8103aa77732e1409b7531eace16cb2840c4c9676bdbcc7bfc95dcce12d\": rpc error: code = NotFound desc = could not find container \"71a36d8103aa77732e1409b7531eace16cb2840c4c9676bdbcc7bfc95dcce12d\": container with ID starting with 71a36d8103aa77732e1409b7531eace16cb2840c4c9676bdbcc7bfc95dcce12d not found: ID does not exist"
Sep 09 02:50:17.751518 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:17.751473    2570 scope.go:117] "RemoveContainer" containerID="a87e064bd39e86b069a49c445832c8eccd611de6b23454ca5c136842685b86c4"
Sep 09 02:50:17.751719 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:50:17.751699    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a87e064bd39e86b069a49c445832c8eccd611de6b23454ca5c136842685b86c4\": container with ID starting with a87e064bd39e86b069a49c445832c8eccd611de6b23454ca5c136842685b86c4 not found: ID does not exist" containerID="a87e064bd39e86b069a49c445832c8eccd611de6b23454ca5c136842685b86c4"
Sep 09 02:50:17.751763 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:17.751726    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a87e064bd39e86b069a49c445832c8eccd611de6b23454ca5c136842685b86c4"} err="failed to get container status \"a87e064bd39e86b069a49c445832c8eccd611de6b23454ca5c136842685b86c4\": rpc error: code = NotFound desc = could not find container \"a87e064bd39e86b069a49c445832c8eccd611de6b23454ca5c136842685b86c4\": container with ID starting with a87e064bd39e86b069a49c445832c8eccd611de6b23454ca5c136842685b86c4 not found: ID does not exist"
Sep 09 02:50:17.754630 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:17.754611    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-6b11a2-openshift-default-5cfbd4b94-fql7q"]
Sep 09 02:50:17.761442 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:17.761423    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-6b11a2-openshift-default-5cfbd4b94-fql7q"]
Sep 09 02:50:17.792073 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:17.792055    2570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a7cadb9-da51-457d-8c72-007236378887-istiod-ca-cert\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:50:17.792145 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:17.792079    2570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a7cadb9-da51-457d-8c72-007236378887-istio-data\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:50:17.792145 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:17.792087    2570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a7cadb9-da51-457d-8c72-007236378887-istio-envoy\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:50:17.792145 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:17.792095    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hzlc\" (UniqueName: \"kubernetes.io/projected/2a7cadb9-da51-457d-8c72-007236378887-kube-api-access-5hzlc\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:50:17.792145 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:17.792104    2570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a7cadb9-da51-457d-8c72-007236378887-istio-podinfo\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:50:17.792145 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:17.792112    2570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a7cadb9-da51-457d-8c72-007236378887-istio-token\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:50:17.792145 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:17.792120    2570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a7cadb9-da51-457d-8c72-007236378887-workload-socket\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:50:18.223661 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:18.223628    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a7cadb9-da51-457d-8c72-007236378887" path="/var/lib/kubelet/pods/2a7cadb9-da51-457d-8c72-007236378887/volumes"
Sep 09 02:50:20.366400 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:20.366353    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-99qjf"]
Sep 09 02:50:28.783864 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:28.783829    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-d838aa-openshift-default-6989bbb4f6-gj56v"]
Sep 09 02:50:28.784378 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:28.784357    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a7cadb9-da51-457d-8c72-007236378887" containerName="istio-proxy"
Sep 09 02:50:28.784449 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:28.784379    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a7cadb9-da51-457d-8c72-007236378887" containerName="istio-proxy"
Sep 09 02:50:28.784449 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:28.784393    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5db50852-b05b-4194-92de-07e973eb7c16" containerName="revoke-keys"
Sep 09 02:50:28.784449 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:28.784398    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="5db50852-b05b-4194-92de-07e973eb7c16" containerName="revoke-keys"
Sep 09 02:50:28.784449 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:28.784409    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a7cadb9-da51-457d-8c72-007236378887" containerName="istio-proxy"
Sep 09 02:50:28.784449 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:28.784414    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a7cadb9-da51-457d-8c72-007236378887" containerName="istio-proxy"
Sep 09 02:50:28.784716 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:28.784464    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="5db50852-b05b-4194-92de-07e973eb7c16" containerName="revoke-keys"
Sep 09 02:50:28.784716 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:28.784474    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a7cadb9-da51-457d-8c72-007236378887" containerName="istio-proxy"
Sep 09 02:50:28.784716 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:28.784480    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a7cadb9-da51-457d-8c72-007236378887" containerName="istio-proxy"
Sep 09 02:50:28.788992 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:28.788957    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-d838aa-openshift-default-6989bbb4f6-gj56v"
Sep 09 02:50:28.791523 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:28.791501    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-d838aa-openshift-default-dockercfg-qbbwg\""
Sep 09 02:50:28.798729 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:28.798706    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-d838aa-openshift-default-6989bbb4f6-gj56v"]
Sep 09 02:50:28.897658 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:28.896260    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb43df80-abb3-4a89-870a-dc6587abf0fa-istio-token\") pod \"e2e-shared-a-w2-d838aa-openshift-default-6989bbb4f6-gj56v\" (UID: \"cb43df80-abb3-4a89-870a-dc6587abf0fa\") " pod="openshift-ingress/e2e-shared-a-w2-d838aa-openshift-default-6989bbb4f6-gj56v"
Sep 09 02:50:28.897658 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:28.896332    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb43df80-abb3-4a89-870a-dc6587abf0fa-credential-socket\") pod \"e2e-shared-a-w2-d838aa-openshift-default-6989bbb4f6-gj56v\" (UID: \"cb43df80-abb3-4a89-870a-dc6587abf0fa\") " pod="openshift-ingress/e2e-shared-a-w2-d838aa-openshift-default-6989bbb4f6-gj56v"
Sep 09 02:50:28.897658 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:28.896381    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb43df80-abb3-4a89-870a-dc6587abf0fa-istio-envoy\") pod \"e2e-shared-a-w2-d838aa-openshift-default-6989bbb4f6-gj56v\" (UID: \"cb43df80-abb3-4a89-870a-dc6587abf0fa\") " pod="openshift-ingress/e2e-shared-a-w2-d838aa-openshift-default-6989bbb4f6-gj56v"
Sep 09 02:50:28.897658 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:28.896422    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb43df80-abb3-4a89-870a-dc6587abf0fa-istiod-ca-cert\") pod \"e2e-shared-a-w2-d838aa-openshift-default-6989bbb4f6-gj56v\" (UID: \"cb43df80-abb3-4a89-870a-dc6587abf0fa\") " pod="openshift-ingress/e2e-shared-a-w2-d838aa-openshift-default-6989bbb4f6-gj56v"
Sep 09 02:50:28.897658 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:28.896471    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb43df80-abb3-4a89-870a-dc6587abf0fa-workload-certs\") pod \"e2e-shared-a-w2-d838aa-openshift-default-6989bbb4f6-gj56v\" (UID: \"cb43df80-abb3-4a89-870a-dc6587abf0fa\") " pod="openshift-ingress/e2e-shared-a-w2-d838aa-openshift-default-6989bbb4f6-gj56v"
Sep 09 02:50:28.897658 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:28.896524    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb43df80-abb3-4a89-870a-dc6587abf0fa-istio-data\") pod \"e2e-shared-a-w2-d838aa-openshift-default-6989bbb4f6-gj56v\" (UID: \"cb43df80-abb3-4a89-870a-dc6587abf0fa\") " pod="openshift-ingress/e2e-shared-a-w2-d838aa-openshift-default-6989bbb4f6-gj56v"
Sep 09 02:50:28.897658 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:28.896565    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb43df80-abb3-4a89-870a-dc6587abf0fa-workload-socket\") pod \"e2e-shared-a-w2-d838aa-openshift-default-6989bbb4f6-gj56v\" (UID: \"cb43df80-abb3-4a89-870a-dc6587abf0fa\") " pod="openshift-ingress/e2e-shared-a-w2-d838aa-openshift-default-6989bbb4f6-gj56v"
Sep 09 02:50:28.897658 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:28.896603    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb43df80-abb3-4a89-870a-dc6587abf0fa-istio-podinfo\") pod \"e2e-shared-a-w2-d838aa-openshift-default-6989bbb4f6-gj56v\" (UID: \"cb43df80-abb3-4a89-870a-dc6587abf0fa\") " pod="openshift-ingress/e2e-shared-a-w2-d838aa-openshift-default-6989bbb4f6-gj56v"
Sep 09 02:50:28.897658 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:28.896645    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sll9\" (UniqueName: \"kubernetes.io/projected/cb43df80-abb3-4a89-870a-dc6587abf0fa-kube-api-access-4sll9\") pod \"e2e-shared-a-w2-d838aa-openshift-default-6989bbb4f6-gj56v\" (UID: \"cb43df80-abb3-4a89-870a-dc6587abf0fa\") " pod="openshift-ingress/e2e-shared-a-w2-d838aa-openshift-default-6989bbb4f6-gj56v"
Sep 09 02:50:28.997082 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:28.997043    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb43df80-abb3-4a89-870a-dc6587abf0fa-istiod-ca-cert\") pod \"e2e-shared-a-w2-d838aa-openshift-default-6989bbb4f6-gj56v\" (UID: \"cb43df80-abb3-4a89-870a-dc6587abf0fa\") " pod="openshift-ingress/e2e-shared-a-w2-d838aa-openshift-default-6989bbb4f6-gj56v"
Sep 09 02:50:28.997230 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:28.997101    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb43df80-abb3-4a89-870a-dc6587abf0fa-workload-certs\") pod \"e2e-shared-a-w2-d838aa-openshift-default-6989bbb4f6-gj56v\" (UID: \"cb43df80-abb3-4a89-870a-dc6587abf0fa\") " pod="openshift-ingress/e2e-shared-a-w2-d838aa-openshift-default-6989bbb4f6-gj56v"
Sep 09 02:50:28.997230 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:28.997142    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb43df80-abb3-4a89-870a-dc6587abf0fa-istio-data\") pod \"e2e-shared-a-w2-d838aa-openshift-default-6989bbb4f6-gj56v\" (UID: \"cb43df80-abb3-4a89-870a-dc6587abf0fa\") " pod="openshift-ingress/e2e-shared-a-w2-d838aa-openshift-default-6989bbb4f6-gj56v"
Sep 09 02:50:28.997230 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:28.997173    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb43df80-abb3-4a89-870a-dc6587abf0fa-workload-socket\") pod \"e2e-shared-a-w2-d838aa-openshift-default-6989bbb4f6-gj56v\" (UID: \"cb43df80-abb3-4a89-870a-dc6587abf0fa\") " pod="openshift-ingress/e2e-shared-a-w2-d838aa-openshift-default-6989bbb4f6-gj56v"
Sep 09 02:50:28.997230 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:28.997201    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb43df80-abb3-4a89-870a-dc6587abf0fa-istio-podinfo\") pod \"e2e-shared-a-w2-d838aa-openshift-default-6989bbb4f6-gj56v\" (UID: \"cb43df80-abb3-4a89-870a-dc6587abf0fa\") " pod="openshift-ingress/e2e-shared-a-w2-d838aa-openshift-default-6989bbb4f6-gj56v"
Sep 09 02:50:28.997518 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:28.997236    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sll9\" (UniqueName: \"kubernetes.io/projected/cb43df80-abb3-4a89-870a-dc6587abf0fa-kube-api-access-4sll9\") pod \"e2e-shared-a-w2-d838aa-openshift-default-6989bbb4f6-gj56v\" (UID: \"cb43df80-abb3-4a89-870a-dc6587abf0fa\") " pod="openshift-ingress/e2e-shared-a-w2-d838aa-openshift-default-6989bbb4f6-gj56v"
Sep 09 02:50:28.997518 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:28.997333    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb43df80-abb3-4a89-870a-dc6587abf0fa-istio-token\") pod \"e2e-shared-a-w2-d838aa-openshift-default-6989bbb4f6-gj56v\" (UID: \"cb43df80-abb3-4a89-870a-dc6587abf0fa\") " pod="openshift-ingress/e2e-shared-a-w2-d838aa-openshift-default-6989bbb4f6-gj56v"
Sep 09 02:50:28.997518 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:28.997360    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb43df80-abb3-4a89-870a-dc6587abf0fa-credential-socket\") pod \"e2e-shared-a-w2-d838aa-openshift-default-6989bbb4f6-gj56v\" (UID: \"cb43df80-abb3-4a89-870a-dc6587abf0fa\") " pod="openshift-ingress/e2e-shared-a-w2-d838aa-openshift-default-6989bbb4f6-gj56v"
Sep 09 02:50:28.997518 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:28.997390    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb43df80-abb3-4a89-870a-dc6587abf0fa-istio-envoy\") pod \"e2e-shared-a-w2-d838aa-openshift-default-6989bbb4f6-gj56v\" (UID: \"cb43df80-abb3-4a89-870a-dc6587abf0fa\") " pod="openshift-ingress/e2e-shared-a-w2-d838aa-openshift-default-6989bbb4f6-gj56v"
Sep 09 02:50:28.997688 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:28.997619    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb43df80-abb3-4a89-870a-dc6587abf0fa-workload-certs\") pod \"e2e-shared-a-w2-d838aa-openshift-default-6989bbb4f6-gj56v\" (UID: \"cb43df80-abb3-4a89-870a-dc6587abf0fa\") " pod="openshift-ingress/e2e-shared-a-w2-d838aa-openshift-default-6989bbb4f6-gj56v"
Sep 09 02:50:28.997963 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:28.997933    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb43df80-abb3-4a89-870a-dc6587abf0fa-istiod-ca-cert\") pod \"e2e-shared-a-w2-d838aa-openshift-default-6989bbb4f6-gj56v\" (UID: \"cb43df80-abb3-4a89-870a-dc6587abf0fa\") " pod="openshift-ingress/e2e-shared-a-w2-d838aa-openshift-default-6989bbb4f6-gj56v"
Sep 09 02:50:28.998127 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:28.998072    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb43df80-abb3-4a89-870a-dc6587abf0fa-istio-data\") pod \"e2e-shared-a-w2-d838aa-openshift-default-6989bbb4f6-gj56v\" (UID: \"cb43df80-abb3-4a89-870a-dc6587abf0fa\") " pod="openshift-ingress/e2e-shared-a-w2-d838aa-openshift-default-6989bbb4f6-gj56v"
Sep 09 02:50:28.998208 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:28.998154    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb43df80-abb3-4a89-870a-dc6587abf0fa-workload-socket\") pod \"e2e-shared-a-w2-d838aa-openshift-default-6989bbb4f6-gj56v\" (UID: \"cb43df80-abb3-4a89-870a-dc6587abf0fa\") " pod="openshift-ingress/e2e-shared-a-w2-d838aa-openshift-default-6989bbb4f6-gj56v"
Sep 09 02:50:28.998272 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:28.998203    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb43df80-abb3-4a89-870a-dc6587abf0fa-credential-socket\") pod \"e2e-shared-a-w2-d838aa-openshift-default-6989bbb4f6-gj56v\" (UID: \"cb43df80-abb3-4a89-870a-dc6587abf0fa\") " pod="openshift-ingress/e2e-shared-a-w2-d838aa-openshift-default-6989bbb4f6-gj56v"
Sep 09 02:50:28.999607 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:28.999582    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb43df80-abb3-4a89-870a-dc6587abf0fa-istio-envoy\") pod \"e2e-shared-a-w2-d838aa-openshift-default-6989bbb4f6-gj56v\" (UID: \"cb43df80-abb3-4a89-870a-dc6587abf0fa\") " pod="openshift-ingress/e2e-shared-a-w2-d838aa-openshift-default-6989bbb4f6-gj56v"
Sep 09 02:50:28.999795 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:28.999777    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb43df80-abb3-4a89-870a-dc6587abf0fa-istio-podinfo\") pod \"e2e-shared-a-w2-d838aa-openshift-default-6989bbb4f6-gj56v\" (UID: \"cb43df80-abb3-4a89-870a-dc6587abf0fa\") " pod="openshift-ingress/e2e-shared-a-w2-d838aa-openshift-default-6989bbb4f6-gj56v"
Sep 09 02:50:29.005763 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:29.005739    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sll9\" (UniqueName: \"kubernetes.io/projected/cb43df80-abb3-4a89-870a-dc6587abf0fa-kube-api-access-4sll9\") pod \"e2e-shared-a-w2-d838aa-openshift-default-6989bbb4f6-gj56v\" (UID: \"cb43df80-abb3-4a89-870a-dc6587abf0fa\") " pod="openshift-ingress/e2e-shared-a-w2-d838aa-openshift-default-6989bbb4f6-gj56v"
Sep 09 02:50:29.005846 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:29.005764    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb43df80-abb3-4a89-870a-dc6587abf0fa-istio-token\") pod \"e2e-shared-a-w2-d838aa-openshift-default-6989bbb4f6-gj56v\" (UID: \"cb43df80-abb3-4a89-870a-dc6587abf0fa\") " pod="openshift-ingress/e2e-shared-a-w2-d838aa-openshift-default-6989bbb4f6-gj56v"
Sep 09 02:50:29.101144 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:29.101123    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-d838aa-openshift-default-6989bbb4f6-gj56v"
Sep 09 02:50:29.382534 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:29.382507    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-d838aa-openshift-default-6989bbb4f6-gj56v"]
Sep 09 02:50:29.383705 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:50:29.383672    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb43df80_abb3_4a89_870a_dc6587abf0fa.slice/crio-c6066d8369142baa3c571400667a39be1ed1706020417d9c427f3234e7490666 WatchSource:0}: Error finding container c6066d8369142baa3c571400667a39be1ed1706020417d9c427f3234e7490666: Status 404 returned error can't find the container with id c6066d8369142baa3c571400667a39be1ed1706020417d9c427f3234e7490666
Sep 09 02:50:29.385758 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:29.385726    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 02:50:29.385843 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:29.385790    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 02:50:29.385843 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:29.385820    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 02:50:29.793014 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:29.792900    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-d838aa-openshift-default-6989bbb4f6-gj56v" event={"ID":"cb43df80-abb3-4a89-870a-dc6587abf0fa","Type":"ContainerStarted","Data":"53bf0dd407af569b1c526edc5e56a2d679ba6b239bd737d75637550054786114"}
Sep 09 02:50:29.793014 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:29.792956    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-d838aa-openshift-default-6989bbb4f6-gj56v" event={"ID":"cb43df80-abb3-4a89-870a-dc6587abf0fa","Type":"ContainerStarted","Data":"c6066d8369142baa3c571400667a39be1ed1706020417d9c427f3234e7490666"}
Sep 09 02:50:29.832190 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:29.832133    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-d838aa-openshift-default-6989bbb4f6-gj56v" podStartSLOduration=1.8321141349999999 podStartE2EDuration="1.832114135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:50:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:50:29.831066211 +0000 UTC m=+1212.085369846" watchObservedRunningTime="2026-09-09 02:50:29.832114135 +0000 UTC m=+1212.086417782"
Sep 09 02:50:30.101629 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:30.101587    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-d838aa-openshift-default-6989bbb4f6-gj56v"
Sep 09 02:50:30.106463 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:30.106437    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-d838aa-openshift-default-6989bbb4f6-gj56v"
Sep 09 02:50:30.798450 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:30.798418    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-d838aa-openshift-default-6989bbb4f6-gj56v"
Sep 09 02:50:30.799344 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:30.799325    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-d838aa-openshift-default-6989bbb4f6-gj56v"
Sep 09 02:50:37.644008 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:37.643959    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-99qjf"]
Sep 09 02:50:42.070992 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:42.070926    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-c173ad-openshift-default-55b8b5b556-hbcds"]
Sep 09 02:50:42.074707 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:42.074686    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-c173ad-openshift-default-55b8b5b556-hbcds"
Sep 09 02:50:42.079254 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:42.079232    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-c173ad-openshift-default-dockercfg-rglgx\""
Sep 09 02:50:42.088390 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:42.088362    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-c173ad-openshift-default-55b8b5b556-hbcds"]
Sep 09 02:50:42.121054 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:42.121016    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efa812e6-5ca9-45a7-9249-7d1cb2de135a-istio-data\") pod \"e2e-shared-b-w3-c173ad-openshift-default-55b8b5b556-hbcds\" (UID: \"efa812e6-5ca9-45a7-9249-7d1cb2de135a\") " pod="openshift-ingress/e2e-shared-b-w3-c173ad-openshift-default-55b8b5b556-hbcds"
Sep 09 02:50:42.121182 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:42.121087    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efa812e6-5ca9-45a7-9249-7d1cb2de135a-istio-envoy\") pod \"e2e-shared-b-w3-c173ad-openshift-default-55b8b5b556-hbcds\" (UID: \"efa812e6-5ca9-45a7-9249-7d1cb2de135a\") " pod="openshift-ingress/e2e-shared-b-w3-c173ad-openshift-default-55b8b5b556-hbcds"
Sep 09 02:50:42.121182 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:42.121160    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efa812e6-5ca9-45a7-9249-7d1cb2de135a-workload-socket\") pod \"e2e-shared-b-w3-c173ad-openshift-default-55b8b5b556-hbcds\" (UID: \"efa812e6-5ca9-45a7-9249-7d1cb2de135a\") " pod="openshift-ingress/e2e-shared-b-w3-c173ad-openshift-default-55b8b5b556-hbcds"
Sep 09 02:50:42.121302 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:42.121206    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efa812e6-5ca9-45a7-9249-7d1cb2de135a-istio-token\") pod \"e2e-shared-b-w3-c173ad-openshift-default-55b8b5b556-hbcds\" (UID: \"efa812e6-5ca9-45a7-9249-7d1cb2de135a\") " pod="openshift-ingress/e2e-shared-b-w3-c173ad-openshift-default-55b8b5b556-hbcds"
Sep 09 02:50:42.121302 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:42.121242    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efa812e6-5ca9-45a7-9249-7d1cb2de135a-istiod-ca-cert\") pod \"e2e-shared-b-w3-c173ad-openshift-default-55b8b5b556-hbcds\" (UID: \"efa812e6-5ca9-45a7-9249-7d1cb2de135a\") " pod="openshift-ingress/e2e-shared-b-w3-c173ad-openshift-default-55b8b5b556-hbcds"
Sep 09 02:50:42.121302 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:42.121267    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efa812e6-5ca9-45a7-9249-7d1cb2de135a-workload-certs\") pod \"e2e-shared-b-w3-c173ad-openshift-default-55b8b5b556-hbcds\" (UID: \"efa812e6-5ca9-45a7-9249-7d1cb2de135a\") " pod="openshift-ingress/e2e-shared-b-w3-c173ad-openshift-default-55b8b5b556-hbcds"
Sep 09 02:50:42.121444 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:42.121304    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efa812e6-5ca9-45a7-9249-7d1cb2de135a-istio-podinfo\") pod \"e2e-shared-b-w3-c173ad-openshift-default-55b8b5b556-hbcds\" (UID: \"efa812e6-5ca9-45a7-9249-7d1cb2de135a\") " pod="openshift-ingress/e2e-shared-b-w3-c173ad-openshift-default-55b8b5b556-hbcds"
Sep 09 02:50:42.121444 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:42.121346    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jgjr\" (UniqueName: \"kubernetes.io/projected/efa812e6-5ca9-45a7-9249-7d1cb2de135a-kube-api-access-7jgjr\") pod \"e2e-shared-b-w3-c173ad-openshift-default-55b8b5b556-hbcds\" (UID: \"efa812e6-5ca9-45a7-9249-7d1cb2de135a\") " pod="openshift-ingress/e2e-shared-b-w3-c173ad-openshift-default-55b8b5b556-hbcds"
Sep 09 02:50:42.121444 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:42.121416    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efa812e6-5ca9-45a7-9249-7d1cb2de135a-credential-socket\") pod \"e2e-shared-b-w3-c173ad-openshift-default-55b8b5b556-hbcds\" (UID: \"efa812e6-5ca9-45a7-9249-7d1cb2de135a\") " pod="openshift-ingress/e2e-shared-b-w3-c173ad-openshift-default-55b8b5b556-hbcds"
Sep 09 02:50:42.222529 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:42.222497    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efa812e6-5ca9-45a7-9249-7d1cb2de135a-istio-envoy\") pod \"e2e-shared-b-w3-c173ad-openshift-default-55b8b5b556-hbcds\" (UID: \"efa812e6-5ca9-45a7-9249-7d1cb2de135a\") " pod="openshift-ingress/e2e-shared-b-w3-c173ad-openshift-default-55b8b5b556-hbcds"
Sep 09 02:50:42.222720 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:42.222535    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efa812e6-5ca9-45a7-9249-7d1cb2de135a-workload-socket\") pod \"e2e-shared-b-w3-c173ad-openshift-default-55b8b5b556-hbcds\" (UID: \"efa812e6-5ca9-45a7-9249-7d1cb2de135a\") " pod="openshift-ingress/e2e-shared-b-w3-c173ad-openshift-default-55b8b5b556-hbcds"
Sep 09 02:50:42.222720 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:42.222557    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efa812e6-5ca9-45a7-9249-7d1cb2de135a-istio-token\") pod \"e2e-shared-b-w3-c173ad-openshift-default-55b8b5b556-hbcds\" (UID: \"efa812e6-5ca9-45a7-9249-7d1cb2de135a\") " pod="openshift-ingress/e2e-shared-b-w3-c173ad-openshift-default-55b8b5b556-hbcds"
Sep 09 02:50:42.222720 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:42.222580    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efa812e6-5ca9-45a7-9249-7d1cb2de135a-istiod-ca-cert\") pod \"e2e-shared-b-w3-c173ad-openshift-default-55b8b5b556-hbcds\" (UID: \"efa812e6-5ca9-45a7-9249-7d1cb2de135a\") " pod="openshift-ingress/e2e-shared-b-w3-c173ad-openshift-default-55b8b5b556-hbcds"
Sep 09 02:50:42.222720 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:42.222606    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efa812e6-5ca9-45a7-9249-7d1cb2de135a-workload-certs\") pod \"e2e-shared-b-w3-c173ad-openshift-default-55b8b5b556-hbcds\" (UID: \"efa812e6-5ca9-45a7-9249-7d1cb2de135a\") " pod="openshift-ingress/e2e-shared-b-w3-c173ad-openshift-default-55b8b5b556-hbcds"
Sep 09 02:50:42.222720 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:42.222641    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efa812e6-5ca9-45a7-9249-7d1cb2de135a-istio-podinfo\") pod \"e2e-shared-b-w3-c173ad-openshift-default-55b8b5b556-hbcds\" (UID: \"efa812e6-5ca9-45a7-9249-7d1cb2de135a\") " pod="openshift-ingress/e2e-shared-b-w3-c173ad-openshift-default-55b8b5b556-hbcds"
Sep 09 02:50:42.222720 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:42.222682    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jgjr\" (UniqueName: \"kubernetes.io/projected/efa812e6-5ca9-45a7-9249-7d1cb2de135a-kube-api-access-7jgjr\") pod \"e2e-shared-b-w3-c173ad-openshift-default-55b8b5b556-hbcds\" (UID: \"efa812e6-5ca9-45a7-9249-7d1cb2de135a\") " pod="openshift-ingress/e2e-shared-b-w3-c173ad-openshift-default-55b8b5b556-hbcds"
Sep 09 02:50:42.223079 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:42.222739    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efa812e6-5ca9-45a7-9249-7d1cb2de135a-credential-socket\") pod \"e2e-shared-b-w3-c173ad-openshift-default-55b8b5b556-hbcds\" (UID: \"efa812e6-5ca9-45a7-9249-7d1cb2de135a\") " pod="openshift-ingress/e2e-shared-b-w3-c173ad-openshift-default-55b8b5b556-hbcds"
Sep 09 02:50:42.223079 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:42.222777    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efa812e6-5ca9-45a7-9249-7d1cb2de135a-istio-data\") pod \"e2e-shared-b-w3-c173ad-openshift-default-55b8b5b556-hbcds\" (UID: \"efa812e6-5ca9-45a7-9249-7d1cb2de135a\") " pod="openshift-ingress/e2e-shared-b-w3-c173ad-openshift-default-55b8b5b556-hbcds"
Sep 09 02:50:42.223188 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:42.223070    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efa812e6-5ca9-45a7-9249-7d1cb2de135a-workload-certs\") pod \"e2e-shared-b-w3-c173ad-openshift-default-55b8b5b556-hbcds\" (UID: \"efa812e6-5ca9-45a7-9249-7d1cb2de135a\") " pod="openshift-ingress/e2e-shared-b-w3-c173ad-openshift-default-55b8b5b556-hbcds"
Sep 09 02:50:42.223188 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:42.223159    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efa812e6-5ca9-45a7-9249-7d1cb2de135a-istio-data\") pod \"e2e-shared-b-w3-c173ad-openshift-default-55b8b5b556-hbcds\" (UID: \"efa812e6-5ca9-45a7-9249-7d1cb2de135a\") " pod="openshift-ingress/e2e-shared-b-w3-c173ad-openshift-default-55b8b5b556-hbcds"
Sep 09 02:50:42.223272 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:42.223194    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efa812e6-5ca9-45a7-9249-7d1cb2de135a-credential-socket\") pod \"e2e-shared-b-w3-c173ad-openshift-default-55b8b5b556-hbcds\" (UID: \"efa812e6-5ca9-45a7-9249-7d1cb2de135a\") " pod="openshift-ingress/e2e-shared-b-w3-c173ad-openshift-default-55b8b5b556-hbcds"
Sep 09 02:50:42.223311 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:42.223271    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efa812e6-5ca9-45a7-9249-7d1cb2de135a-workload-socket\") pod \"e2e-shared-b-w3-c173ad-openshift-default-55b8b5b556-hbcds\" (UID: \"efa812e6-5ca9-45a7-9249-7d1cb2de135a\") " pod="openshift-ingress/e2e-shared-b-w3-c173ad-openshift-default-55b8b5b556-hbcds"
Sep 09 02:50:42.223481 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:42.223455    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efa812e6-5ca9-45a7-9249-7d1cb2de135a-istiod-ca-cert\") pod \"e2e-shared-b-w3-c173ad-openshift-default-55b8b5b556-hbcds\" (UID: \"efa812e6-5ca9-45a7-9249-7d1cb2de135a\") " pod="openshift-ingress/e2e-shared-b-w3-c173ad-openshift-default-55b8b5b556-hbcds"
Sep 09 02:50:42.225176 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:42.225156    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efa812e6-5ca9-45a7-9249-7d1cb2de135a-istio-envoy\") pod \"e2e-shared-b-w3-c173ad-openshift-default-55b8b5b556-hbcds\" (UID: \"efa812e6-5ca9-45a7-9249-7d1cb2de135a\") " pod="openshift-ingress/e2e-shared-b-w3-c173ad-openshift-default-55b8b5b556-hbcds"
Sep 09 02:50:42.225261 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:42.225235    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efa812e6-5ca9-45a7-9249-7d1cb2de135a-istio-podinfo\") pod \"e2e-shared-b-w3-c173ad-openshift-default-55b8b5b556-hbcds\" (UID: \"efa812e6-5ca9-45a7-9249-7d1cb2de135a\") " pod="openshift-ingress/e2e-shared-b-w3-c173ad-openshift-default-55b8b5b556-hbcds"
Sep 09 02:50:42.231402 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:42.231381    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jgjr\" (UniqueName: \"kubernetes.io/projected/efa812e6-5ca9-45a7-9249-7d1cb2de135a-kube-api-access-7jgjr\") pod \"e2e-shared-b-w3-c173ad-openshift-default-55b8b5b556-hbcds\" (UID: \"efa812e6-5ca9-45a7-9249-7d1cb2de135a\") " pod="openshift-ingress/e2e-shared-b-w3-c173ad-openshift-default-55b8b5b556-hbcds"
Sep 09 02:50:42.234204 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:42.234184    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efa812e6-5ca9-45a7-9249-7d1cb2de135a-istio-token\") pod \"e2e-shared-b-w3-c173ad-openshift-default-55b8b5b556-hbcds\" (UID: \"efa812e6-5ca9-45a7-9249-7d1cb2de135a\") " pod="openshift-ingress/e2e-shared-b-w3-c173ad-openshift-default-55b8b5b556-hbcds"
Sep 09 02:50:42.386222 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:42.386188    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-c173ad-openshift-default-55b8b5b556-hbcds"
Sep 09 02:50:42.674580 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:42.674551    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-c173ad-openshift-default-55b8b5b556-hbcds"]
Sep 09 02:50:42.676430 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:50:42.676403    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefa812e6_5ca9_45a7_9249_7d1cb2de135a.slice/crio-66c2ce14b8c269cee3c9551010d192272a8c61970b4da4c4ac3613936dd2e204 WatchSource:0}: Error finding container 66c2ce14b8c269cee3c9551010d192272a8c61970b4da4c4ac3613936dd2e204: Status 404 returned error can't find the container with id 66c2ce14b8c269cee3c9551010d192272a8c61970b4da4c4ac3613936dd2e204
Sep 09 02:50:42.678732 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:42.678703    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 02:50:42.678818 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:42.678762    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 02:50:42.678818 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:42.678789    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 02:50:42.870093 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:42.870056    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-c173ad-openshift-default-55b8b5b556-hbcds" event={"ID":"efa812e6-5ca9-45a7-9249-7d1cb2de135a","Type":"ContainerStarted","Data":"cbe941bdd98570226bc8dd220760a5541aeb1b6f0bf82071cabdeed4db1ef090"}
Sep 09 02:50:42.870265 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:42.870115    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-c173ad-openshift-default-55b8b5b556-hbcds" event={"ID":"efa812e6-5ca9-45a7-9249-7d1cb2de135a","Type":"ContainerStarted","Data":"66c2ce14b8c269cee3c9551010d192272a8c61970b4da4c4ac3613936dd2e204"}
Sep 09 02:50:42.891754 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:42.891705    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-c173ad-openshift-default-55b8b5b556-hbcds" podStartSLOduration=0.891688829 podStartE2EDuration="891.688829ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:50:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:50:42.890330347 +0000 UTC m=+1225.144633983" watchObservedRunningTime="2026-09-09 02:50:42.891688829 +0000 UTC m=+1225.145992464"
Sep 09 02:50:43.387194 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:43.387154    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-c173ad-openshift-default-55b8b5b556-hbcds"
Sep 09 02:50:43.392043 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:43.392018    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-c173ad-openshift-default-55b8b5b556-hbcds"
Sep 09 02:50:43.874984 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:43.874943    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-c173ad-openshift-default-55b8b5b556-hbcds"
Sep 09 02:50:43.876059 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:43.876034    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-c173ad-openshift-default-55b8b5b556-hbcds"
Sep 09 02:50:46.840111 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:46.840078    2570 patch_prober.go:28] interesting pod/e2e-shared-a-w2-d838aa-openshift-default-6989bbb4f6-gj56v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 09 02:50:46.840486 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:46.840213    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-d838aa-openshift-default-6989bbb4f6-gj56v" podUID="cb43df80-abb3-4a89-870a-dc6587abf0fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 09 02:50:48.903523 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:48.903490    2570 generic.go:356] "Generic (PLEG): container finished" podID="cb43df80-abb3-4a89-870a-dc6587abf0fa" containerID="53bf0dd407af569b1c526edc5e56a2d679ba6b239bd737d75637550054786114" exitCode=0
Sep 09 02:50:48.903936 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:48.903558    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-d838aa-openshift-default-6989bbb4f6-gj56v" event={"ID":"cb43df80-abb3-4a89-870a-dc6587abf0fa","Type":"ContainerDied","Data":"53bf0dd407af569b1c526edc5e56a2d679ba6b239bd737d75637550054786114"}
Sep 09 02:50:48.903936 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:48.903929    2570 scope.go:117] "RemoveContainer" containerID="53bf0dd407af569b1c526edc5e56a2d679ba6b239bd737d75637550054786114"
Sep 09 02:50:48.904912 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:48.904887    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 02:50:48.904969 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:48.904950    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 02:50:48.905031 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:48.904999    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 02:50:49.910419 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:49.910381    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-d838aa-openshift-default-6989bbb4f6-gj56v" event={"ID":"cb43df80-abb3-4a89-870a-dc6587abf0fa","Type":"ContainerStarted","Data":"e825f79926523f2902e7cbbf9b3068fd4841d3ed4ed7aec0f75721deff0d4345"}
Sep 09 02:50:50.101825 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:50.101785    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-d838aa-openshift-default-6989bbb4f6-gj56v"
Sep 09 02:50:50.102081 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:50.102057    2570 patch_prober.go:28] interesting pod/e2e-shared-a-w2-d838aa-openshift-default-6989bbb4f6-gj56v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Sep 09 02:50:50.102198 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:50.102125    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-d838aa-openshift-default-6989bbb4f6-gj56v" podUID="cb43df80-abb3-4a89-870a-dc6587abf0fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Sep 09 02:50:50.165637 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:50.165548    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-99qjf"]
Sep 09 02:50:51.102313 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:51.102275    2570 patch_prober.go:28] interesting pod/e2e-shared-a-w2-d838aa-openshift-default-6989bbb4f6-gj56v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Sep 09 02:50:51.102700 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:51.102340    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-d838aa-openshift-default-6989bbb4f6-gj56v" podUID="cb43df80-abb3-4a89-870a-dc6587abf0fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Sep 09 02:50:52.102518 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:52.102477    2570 patch_prober.go:28] interesting pod/e2e-shared-a-w2-d838aa-openshift-default-6989bbb4f6-gj56v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Sep 09 02:50:52.102901 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:52.102533    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-d838aa-openshift-default-6989bbb4f6-gj56v" podUID="cb43df80-abb3-4a89-870a-dc6587abf0fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Sep 09 02:50:53.102414 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:53.102377    2570 patch_prober.go:28] interesting pod/e2e-shared-a-w2-d838aa-openshift-default-6989bbb4f6-gj56v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Sep 09 02:50:53.102614 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:53.102447    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-d838aa-openshift-default-6989bbb4f6-gj56v" podUID="cb43df80-abb3-4a89-870a-dc6587abf0fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Sep 09 02:50:54.102422 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:54.102378    2570 patch_prober.go:28] interesting pod/e2e-shared-a-w2-d838aa-openshift-default-6989bbb4f6-gj56v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Sep 09 02:50:54.102627 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:54.102443    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-d838aa-openshift-default-6989bbb4f6-gj56v" podUID="cb43df80-abb3-4a89-870a-dc6587abf0fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Sep 09 02:50:55.101832 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:55.101790    2570 patch_prober.go:28] interesting pod/e2e-shared-a-w2-d838aa-openshift-default-6989bbb4f6-gj56v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Sep 09 02:50:55.102041 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:55.101850    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-d838aa-openshift-default-6989bbb4f6-gj56v" podUID="cb43df80-abb3-4a89-870a-dc6587abf0fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Sep 09 02:50:56.105752 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:56.105725    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-d838aa-openshift-default-6989bbb4f6-gj56v"
Sep 09 02:50:56.106211 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:56.106010    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-d838aa-openshift-default-6989bbb4f6-gj56v"
Sep 09 02:50:56.106741 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:56.106725    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-d838aa-openshift-default-6989bbb4f6-gj56v"
Sep 09 02:50:57.341061 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:57.341027    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-f5aebb-openshift-default-7c76c475d-r9lxr"]
Sep 09 02:50:57.345399 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:57.345376    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-f5aebb-openshift-default-7c76c475d-r9lxr"
Sep 09 02:50:57.348358 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:57.348335    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-f5aebb-openshift-default-dockercfg-kfh9b\""
Sep 09 02:50:57.356310 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:57.356288    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-f5aebb-openshift-default-7c76c475d-r9lxr"]
Sep 09 02:50:57.488103 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:57.488052    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7463a0d2-3ecb-479f-b47b-b4b5e8190396-workload-certs\") pod \"e2e-shared-b-w2-f5aebb-openshift-default-7c76c475d-r9lxr\" (UID: \"7463a0d2-3ecb-479f-b47b-b4b5e8190396\") " pod="openshift-ingress/e2e-shared-b-w2-f5aebb-openshift-default-7c76c475d-r9lxr"
Sep 09 02:50:57.488103 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:57.488092    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcvcw\" (UniqueName: \"kubernetes.io/projected/7463a0d2-3ecb-479f-b47b-b4b5e8190396-kube-api-access-wcvcw\") pod \"e2e-shared-b-w2-f5aebb-openshift-default-7c76c475d-r9lxr\" (UID: \"7463a0d2-3ecb-479f-b47b-b4b5e8190396\") " pod="openshift-ingress/e2e-shared-b-w2-f5aebb-openshift-default-7c76c475d-r9lxr"
Sep 09 02:50:57.488394 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:57.488121    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7463a0d2-3ecb-479f-b47b-b4b5e8190396-istio-token\") pod \"e2e-shared-b-w2-f5aebb-openshift-default-7c76c475d-r9lxr\" (UID: \"7463a0d2-3ecb-479f-b47b-b4b5e8190396\") " pod="openshift-ingress/e2e-shared-b-w2-f5aebb-openshift-default-7c76c475d-r9lxr"
Sep 09 02:50:57.488394 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:57.488158    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7463a0d2-3ecb-479f-b47b-b4b5e8190396-workload-socket\") pod \"e2e-shared-b-w2-f5aebb-openshift-default-7c76c475d-r9lxr\" (UID: \"7463a0d2-3ecb-479f-b47b-b4b5e8190396\") " pod="openshift-ingress/e2e-shared-b-w2-f5aebb-openshift-default-7c76c475d-r9lxr"
Sep 09 02:50:57.488394 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:57.488178    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7463a0d2-3ecb-479f-b47b-b4b5e8190396-credential-socket\") pod \"e2e-shared-b-w2-f5aebb-openshift-default-7c76c475d-r9lxr\" (UID: \"7463a0d2-3ecb-479f-b47b-b4b5e8190396\") " pod="openshift-ingress/e2e-shared-b-w2-f5aebb-openshift-default-7c76c475d-r9lxr"
Sep 09 02:50:57.488394 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:57.488281    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7463a0d2-3ecb-479f-b47b-b4b5e8190396-istio-envoy\") pod \"e2e-shared-b-w2-f5aebb-openshift-default-7c76c475d-r9lxr\" (UID: \"7463a0d2-3ecb-479f-b47b-b4b5e8190396\") " pod="openshift-ingress/e2e-shared-b-w2-f5aebb-openshift-default-7c76c475d-r9lxr"
Sep 09 02:50:57.488394 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:57.488333    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7463a0d2-3ecb-479f-b47b-b4b5e8190396-istio-podinfo\") pod \"e2e-shared-b-w2-f5aebb-openshift-default-7c76c475d-r9lxr\" (UID: \"7463a0d2-3ecb-479f-b47b-b4b5e8190396\") " pod="openshift-ingress/e2e-shared-b-w2-f5aebb-openshift-default-7c76c475d-r9lxr"
Sep 09 02:50:57.488394 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:57.488359    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7463a0d2-3ecb-479f-b47b-b4b5e8190396-istiod-ca-cert\") pod \"e2e-shared-b-w2-f5aebb-openshift-default-7c76c475d-r9lxr\" (UID: \"7463a0d2-3ecb-479f-b47b-b4b5e8190396\") " pod="openshift-ingress/e2e-shared-b-w2-f5aebb-openshift-default-7c76c475d-r9lxr"
Sep 09 02:50:57.488718 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:57.488398    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7463a0d2-3ecb-479f-b47b-b4b5e8190396-istio-data\") pod \"e2e-shared-b-w2-f5aebb-openshift-default-7c76c475d-r9lxr\" (UID: \"7463a0d2-3ecb-479f-b47b-b4b5e8190396\") " pod="openshift-ingress/e2e-shared-b-w2-f5aebb-openshift-default-7c76c475d-r9lxr"
Sep 09 02:50:57.589683 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:57.589641    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7463a0d2-3ecb-479f-b47b-b4b5e8190396-credential-socket\") pod \"e2e-shared-b-w2-f5aebb-openshift-default-7c76c475d-r9lxr\" (UID: \"7463a0d2-3ecb-479f-b47b-b4b5e8190396\") " pod="openshift-ingress/e2e-shared-b-w2-f5aebb-openshift-default-7c76c475d-r9lxr"
Sep 09 02:50:57.589904 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:57.589721    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7463a0d2-3ecb-479f-b47b-b4b5e8190396-istio-envoy\") pod \"e2e-shared-b-w2-f5aebb-openshift-default-7c76c475d-r9lxr\" (UID: \"7463a0d2-3ecb-479f-b47b-b4b5e8190396\") " pod="openshift-ingress/e2e-shared-b-w2-f5aebb-openshift-default-7c76c475d-r9lxr"
Sep 09 02:50:57.589904 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:57.589758    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7463a0d2-3ecb-479f-b47b-b4b5e8190396-istio-podinfo\") pod \"e2e-shared-b-w2-f5aebb-openshift-default-7c76c475d-r9lxr\" (UID: \"7463a0d2-3ecb-479f-b47b-b4b5e8190396\") " pod="openshift-ingress/e2e-shared-b-w2-f5aebb-openshift-default-7c76c475d-r9lxr"
Sep 09 02:50:57.589904 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:57.589781    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7463a0d2-3ecb-479f-b47b-b4b5e8190396-istiod-ca-cert\") pod \"e2e-shared-b-w2-f5aebb-openshift-default-7c76c475d-r9lxr\" (UID: \"7463a0d2-3ecb-479f-b47b-b4b5e8190396\") " pod="openshift-ingress/e2e-shared-b-w2-f5aebb-openshift-default-7c76c475d-r9lxr"
Sep 09 02:50:57.589904 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:57.589821    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7463a0d2-3ecb-479f-b47b-b4b5e8190396-istio-data\") pod \"e2e-shared-b-w2-f5aebb-openshift-default-7c76c475d-r9lxr\" (UID: \"7463a0d2-3ecb-479f-b47b-b4b5e8190396\") " pod="openshift-ingress/e2e-shared-b-w2-f5aebb-openshift-default-7c76c475d-r9lxr"
Sep 09 02:50:57.590164 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:57.589913    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7463a0d2-3ecb-479f-b47b-b4b5e8190396-workload-certs\") pod \"e2e-shared-b-w2-f5aebb-openshift-default-7c76c475d-r9lxr\" (UID: \"7463a0d2-3ecb-479f-b47b-b4b5e8190396\") " pod="openshift-ingress/e2e-shared-b-w2-f5aebb-openshift-default-7c76c475d-r9lxr"
Sep 09 02:50:57.590164 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:57.589939    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcvcw\" (UniqueName: \"kubernetes.io/projected/7463a0d2-3ecb-479f-b47b-b4b5e8190396-kube-api-access-wcvcw\") pod \"e2e-shared-b-w2-f5aebb-openshift-default-7c76c475d-r9lxr\" (UID: \"7463a0d2-3ecb-479f-b47b-b4b5e8190396\") " pod="openshift-ingress/e2e-shared-b-w2-f5aebb-openshift-default-7c76c475d-r9lxr"
Sep 09 02:50:57.590320 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:57.590280    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7463a0d2-3ecb-479f-b47b-b4b5e8190396-istio-data\") pod \"e2e-shared-b-w2-f5aebb-openshift-default-7c76c475d-r9lxr\" (UID: \"7463a0d2-3ecb-479f-b47b-b4b5e8190396\") " pod="openshift-ingress/e2e-shared-b-w2-f5aebb-openshift-default-7c76c475d-r9lxr"
Sep 09 02:50:57.590320 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:57.590305    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7463a0d2-3ecb-479f-b47b-b4b5e8190396-workload-certs\") pod \"e2e-shared-b-w2-f5aebb-openshift-default-7c76c475d-r9lxr\" (UID: \"7463a0d2-3ecb-479f-b47b-b4b5e8190396\") " pod="openshift-ingress/e2e-shared-b-w2-f5aebb-openshift-default-7c76c475d-r9lxr"
Sep 09 02:50:57.590510 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:57.590365    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7463a0d2-3ecb-479f-b47b-b4b5e8190396-istio-token\") pod \"e2e-shared-b-w2-f5aebb-openshift-default-7c76c475d-r9lxr\" (UID: \"7463a0d2-3ecb-479f-b47b-b4b5e8190396\") " pod="openshift-ingress/e2e-shared-b-w2-f5aebb-openshift-default-7c76c475d-r9lxr"
Sep 09 02:50:57.590510 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:57.590439    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7463a0d2-3ecb-479f-b47b-b4b5e8190396-workload-socket\") pod \"e2e-shared-b-w2-f5aebb-openshift-default-7c76c475d-r9lxr\" (UID: \"7463a0d2-3ecb-479f-b47b-b4b5e8190396\") " pod="openshift-ingress/e2e-shared-b-w2-f5aebb-openshift-default-7c76c475d-r9lxr"
Sep 09 02:50:57.590699 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:57.590680    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7463a0d2-3ecb-479f-b47b-b4b5e8190396-credential-socket\") pod \"e2e-shared-b-w2-f5aebb-openshift-default-7c76c475d-r9lxr\" (UID: \"7463a0d2-3ecb-479f-b47b-b4b5e8190396\") " pod="openshift-ingress/e2e-shared-b-w2-f5aebb-openshift-default-7c76c475d-r9lxr"
Sep 09 02:50:57.590763 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:57.590699    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7463a0d2-3ecb-479f-b47b-b4b5e8190396-workload-socket\") pod \"e2e-shared-b-w2-f5aebb-openshift-default-7c76c475d-r9lxr\" (UID: \"7463a0d2-3ecb-479f-b47b-b4b5e8190396\") " pod="openshift-ingress/e2e-shared-b-w2-f5aebb-openshift-default-7c76c475d-r9lxr"
Sep 09 02:50:57.590938 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:57.590921    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7463a0d2-3ecb-479f-b47b-b4b5e8190396-istiod-ca-cert\") pod \"e2e-shared-b-w2-f5aebb-openshift-default-7c76c475d-r9lxr\" (UID: \"7463a0d2-3ecb-479f-b47b-b4b5e8190396\") " pod="openshift-ingress/e2e-shared-b-w2-f5aebb-openshift-default-7c76c475d-r9lxr"
Sep 09 02:50:57.592723 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:57.592653    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7463a0d2-3ecb-479f-b47b-b4b5e8190396-istio-envoy\") pod \"e2e-shared-b-w2-f5aebb-openshift-default-7c76c475d-r9lxr\" (UID: \"7463a0d2-3ecb-479f-b47b-b4b5e8190396\") " pod="openshift-ingress/e2e-shared-b-w2-f5aebb-openshift-default-7c76c475d-r9lxr"
Sep 09 02:50:57.592964 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:57.592943    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7463a0d2-3ecb-479f-b47b-b4b5e8190396-istio-podinfo\") pod \"e2e-shared-b-w2-f5aebb-openshift-default-7c76c475d-r9lxr\" (UID: \"7463a0d2-3ecb-479f-b47b-b4b5e8190396\") " pod="openshift-ingress/e2e-shared-b-w2-f5aebb-openshift-default-7c76c475d-r9lxr"
Sep 09 02:50:57.598310 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:57.598287    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcvcw\" (UniqueName: \"kubernetes.io/projected/7463a0d2-3ecb-479f-b47b-b4b5e8190396-kube-api-access-wcvcw\") pod \"e2e-shared-b-w2-f5aebb-openshift-default-7c76c475d-r9lxr\" (UID: \"7463a0d2-3ecb-479f-b47b-b4b5e8190396\") " pod="openshift-ingress/e2e-shared-b-w2-f5aebb-openshift-default-7c76c475d-r9lxr"
Sep 09 02:50:57.602470 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:57.602444    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7463a0d2-3ecb-479f-b47b-b4b5e8190396-istio-token\") pod \"e2e-shared-b-w2-f5aebb-openshift-default-7c76c475d-r9lxr\" (UID: \"7463a0d2-3ecb-479f-b47b-b4b5e8190396\") " pod="openshift-ingress/e2e-shared-b-w2-f5aebb-openshift-default-7c76c475d-r9lxr"
Sep 09 02:50:57.658778 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:57.658743    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-f5aebb-openshift-default-7c76c475d-r9lxr"
Sep 09 02:50:57.938563 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:57.938523    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-f5aebb-openshift-default-7c76c475d-r9lxr"]
Sep 09 02:50:57.940336 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:50:57.940293    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7463a0d2_3ecb_479f_b47b_b4b5e8190396.slice/crio-6634994ba1653b5e02bb3998490ba62aac410d532ef5ab79fcd9f665548f9fa3 WatchSource:0}: Error finding container 6634994ba1653b5e02bb3998490ba62aac410d532ef5ab79fcd9f665548f9fa3: Status 404 returned error can't find the container with id 6634994ba1653b5e02bb3998490ba62aac410d532ef5ab79fcd9f665548f9fa3
Sep 09 02:50:57.943183 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:57.943146    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 02:50:57.943280 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:57.943227    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 02:50:57.943320 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:57.943275    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 02:50:57.952055 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:57.952023    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-f5aebb-openshift-default-7c76c475d-r9lxr" event={"ID":"7463a0d2-3ecb-479f-b47b-b4b5e8190396","Type":"ContainerStarted","Data":"6634994ba1653b5e02bb3998490ba62aac410d532ef5ab79fcd9f665548f9fa3"}
Sep 09 02:50:58.958526 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:58.958489    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-f5aebb-openshift-default-7c76c475d-r9lxr" event={"ID":"7463a0d2-3ecb-479f-b47b-b4b5e8190396","Type":"ContainerStarted","Data":"99723178ceaf97f1e67cf2df749aee119e4fab4187313d3023476ba2c50c5e11"}
Sep 09 02:50:58.980195 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:58.980144    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-f5aebb-openshift-default-7c76c475d-r9lxr" podStartSLOduration=1.980131216 podStartE2EDuration="1.980131216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:50:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:50:58.979604782 +0000 UTC m=+1241.233908444" watchObservedRunningTime="2026-09-09 02:50:58.980131216 +0000 UTC m=+1241.234434851"
Sep 09 02:50:59.659444 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:59.659407    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-f5aebb-openshift-default-7c76c475d-r9lxr"
Sep 09 02:50:59.664720 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:59.664693    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-f5aebb-openshift-default-7c76c475d-r9lxr"
Sep 09 02:50:59.774185 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:59.772878    2570 patch_prober.go:28] interesting pod/e2e-shared-b-w3-c173ad-openshift-default-55b8b5b556-hbcds container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": read tcp 10.134.0.2:40842->10.134.0.57:15021: read: connection reset by peer" start-of-body=
Sep 09 02:50:59.774185 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:59.772943    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-c173ad-openshift-default-55b8b5b556-hbcds" podUID="efa812e6-5ca9-45a7-9249-7d1cb2de135a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": read tcp 10.134.0.2:40842->10.134.0.57:15021: read: connection reset by peer"
Sep 09 02:50:59.964759 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:59.964678    2570 generic.go:356] "Generic (PLEG): container finished" podID="efa812e6-5ca9-45a7-9249-7d1cb2de135a" containerID="cbe941bdd98570226bc8dd220760a5541aeb1b6f0bf82071cabdeed4db1ef090" exitCode=0
Sep 09 02:50:59.965133 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:59.964752    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-c173ad-openshift-default-55b8b5b556-hbcds" event={"ID":"efa812e6-5ca9-45a7-9249-7d1cb2de135a","Type":"ContainerDied","Data":"cbe941bdd98570226bc8dd220760a5541aeb1b6f0bf82071cabdeed4db1ef090"}
Sep 09 02:50:59.965133 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:59.964940    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-f5aebb-openshift-default-7c76c475d-r9lxr"
Sep 09 02:50:59.965243 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:59.965226    2570 scope.go:117] "RemoveContainer" containerID="cbe941bdd98570226bc8dd220760a5541aeb1b6f0bf82071cabdeed4db1ef090"
Sep 09 02:50:59.966070 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:59.965858    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-f5aebb-openshift-default-7c76c475d-r9lxr"
Sep 09 02:50:59.966419 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:59.966391    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 02:50:59.966496 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:59.966480    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 02:50:59.966532 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:50:59.966525    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 02:51:00.972026 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:00.971985    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-c173ad-openshift-default-55b8b5b556-hbcds" event={"ID":"efa812e6-5ca9-45a7-9249-7d1cb2de135a","Type":"ContainerStarted","Data":"cc983b1a0496fcd61e0a89c916ea32bc27970aa2e60c44160150c873f1cee4b8"}
Sep 09 02:51:01.386984 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:01.386943    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-c173ad-openshift-default-55b8b5b556-hbcds"
Sep 09 02:51:01.387182 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:01.387160    2570 patch_prober.go:28] interesting pod/e2e-shared-b-w3-c173ad-openshift-default-55b8b5b556-hbcds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Sep 09 02:51:01.387241 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:01.387221    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-c173ad-openshift-default-55b8b5b556-hbcds" podUID="efa812e6-5ca9-45a7-9249-7d1cb2de135a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Sep 09 02:51:02.387398 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:02.387362    2570 patch_prober.go:28] interesting pod/e2e-shared-b-w3-c173ad-openshift-default-55b8b5b556-hbcds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Sep 09 02:51:02.387800 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:02.387420    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-c173ad-openshift-default-55b8b5b556-hbcds" podUID="efa812e6-5ca9-45a7-9249-7d1cb2de135a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Sep 09 02:51:03.386689 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:03.386644    2570 patch_prober.go:28] interesting pod/e2e-shared-b-w3-c173ad-openshift-default-55b8b5b556-hbcds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Sep 09 02:51:03.386903 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:03.386723    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-c173ad-openshift-default-55b8b5b556-hbcds" podUID="efa812e6-5ca9-45a7-9249-7d1cb2de135a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Sep 09 02:51:04.386898 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:04.386869    2570 patch_prober.go:28] interesting pod/e2e-shared-b-w3-c173ad-openshift-default-55b8b5b556-hbcds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Sep 09 02:51:04.387284 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:04.386917    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-c173ad-openshift-default-55b8b5b556-hbcds" podUID="efa812e6-5ca9-45a7-9249-7d1cb2de135a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Sep 09 02:51:05.387086 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:05.387047    2570 patch_prober.go:28] interesting pod/e2e-shared-b-w3-c173ad-openshift-default-55b8b5b556-hbcds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Sep 09 02:51:05.387486 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:05.387120    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-c173ad-openshift-default-55b8b5b556-hbcds" podUID="efa812e6-5ca9-45a7-9249-7d1cb2de135a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Sep 09 02:51:06.390161 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:06.390131    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-c173ad-openshift-default-55b8b5b556-hbcds"
Sep 09 02:51:06.390613 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:06.390591    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-c173ad-openshift-default-55b8b5b556-hbcds"
Sep 09 02:51:06.391218 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:06.391201    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-c173ad-openshift-default-55b8b5b556-hbcds"
Sep 09 02:51:06.748674 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:06.748571    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-99qjf"]
Sep 09 02:51:15.631811 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:15.631776    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-56dbc0/auth-test-model-56dbc0-kserve-fd985d4cf-2fmsj"]
Sep 09 02:51:15.636476 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:15.636451    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-56dbc0/auth-test-model-56dbc0-kserve-fd985d4cf-2fmsj"
Sep 09 02:51:15.639237 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:15.639216    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-56dbc0\"/\"kube-root-ca.crt\""
Sep 09 02:51:15.640497 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:15.640467    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-56dbc0\"/\"default-dockercfg-rdn6j\""
Sep 09 02:51:15.640607 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:15.640496    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-56dbc0\"/\"auth-test-model-56dbc0-kserve-self-signed-certs\""
Sep 09 02:51:15.640806 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:15.640785    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-56dbc0\"/\"openshift-service-ca.crt\""
Sep 09 02:51:15.646893 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:15.646839    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-56dbc0/auth-test-model-56dbc0-kserve-fd985d4cf-2fmsj"]
Sep 09 02:51:15.707441 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:15.707405    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fed344c4-8783-44fb-af19-c7e6971c5ffb-tls-certs\") pod \"auth-test-model-56dbc0-kserve-fd985d4cf-2fmsj\" (UID: \"fed344c4-8783-44fb-af19-c7e6971c5ffb\") " pod="ai-tenant-e2e-shared-a-w3-56dbc0/auth-test-model-56dbc0-kserve-fd985d4cf-2fmsj"
Sep 09 02:51:15.707598 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:15.707444    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fed344c4-8783-44fb-af19-c7e6971c5ffb-tmp-dir\") pod \"auth-test-model-56dbc0-kserve-fd985d4cf-2fmsj\" (UID: \"fed344c4-8783-44fb-af19-c7e6971c5ffb\") " pod="ai-tenant-e2e-shared-a-w3-56dbc0/auth-test-model-56dbc0-kserve-fd985d4cf-2fmsj"
Sep 09 02:51:15.707598 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:15.707474    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fed344c4-8783-44fb-af19-c7e6971c5ffb-model-cache\") pod \"auth-test-model-56dbc0-kserve-fd985d4cf-2fmsj\" (UID: \"fed344c4-8783-44fb-af19-c7e6971c5ffb\") " pod="ai-tenant-e2e-shared-a-w3-56dbc0/auth-test-model-56dbc0-kserve-fd985d4cf-2fmsj"
Sep 09 02:51:15.707598 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:15.707554    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fed344c4-8783-44fb-af19-c7e6971c5ffb-home\") pod \"auth-test-model-56dbc0-kserve-fd985d4cf-2fmsj\" (UID: \"fed344c4-8783-44fb-af19-c7e6971c5ffb\") " pod="ai-tenant-e2e-shared-a-w3-56dbc0/auth-test-model-56dbc0-kserve-fd985d4cf-2fmsj"
Sep 09 02:51:15.707734 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:15.707608    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fed344c4-8783-44fb-af19-c7e6971c5ffb-dshm\") pod \"auth-test-model-56dbc0-kserve-fd985d4cf-2fmsj\" (UID: \"fed344c4-8783-44fb-af19-c7e6971c5ffb\") " pod="ai-tenant-e2e-shared-a-w3-56dbc0/auth-test-model-56dbc0-kserve-fd985d4cf-2fmsj"
Sep 09 02:51:15.707734 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:15.707630    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgnpf\" (UniqueName: \"kubernetes.io/projected/fed344c4-8783-44fb-af19-c7e6971c5ffb-kube-api-access-kgnpf\") pod \"auth-test-model-56dbc0-kserve-fd985d4cf-2fmsj\" (UID: \"fed344c4-8783-44fb-af19-c7e6971c5ffb\") " pod="ai-tenant-e2e-shared-a-w3-56dbc0/auth-test-model-56dbc0-kserve-fd985d4cf-2fmsj"
Sep 09 02:51:15.809024 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:15.808960    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fed344c4-8783-44fb-af19-c7e6971c5ffb-tls-certs\") pod \"auth-test-model-56dbc0-kserve-fd985d4cf-2fmsj\" (UID: \"fed344c4-8783-44fb-af19-c7e6971c5ffb\") " pod="ai-tenant-e2e-shared-a-w3-56dbc0/auth-test-model-56dbc0-kserve-fd985d4cf-2fmsj"
Sep 09 02:51:15.809181 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:15.809122    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fed344c4-8783-44fb-af19-c7e6971c5ffb-tmp-dir\") pod \"auth-test-model-56dbc0-kserve-fd985d4cf-2fmsj\" (UID: \"fed344c4-8783-44fb-af19-c7e6971c5ffb\") " pod="ai-tenant-e2e-shared-a-w3-56dbc0/auth-test-model-56dbc0-kserve-fd985d4cf-2fmsj"
Sep 09 02:51:15.809181 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:15.809157    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fed344c4-8783-44fb-af19-c7e6971c5ffb-model-cache\") pod \"auth-test-model-56dbc0-kserve-fd985d4cf-2fmsj\" (UID: \"fed344c4-8783-44fb-af19-c7e6971c5ffb\") " pod="ai-tenant-e2e-shared-a-w3-56dbc0/auth-test-model-56dbc0-kserve-fd985d4cf-2fmsj"
Sep 09 02:51:15.809256 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:15.809207    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fed344c4-8783-44fb-af19-c7e6971c5ffb-home\") pod \"auth-test-model-56dbc0-kserve-fd985d4cf-2fmsj\" (UID: \"fed344c4-8783-44fb-af19-c7e6971c5ffb\") " pod="ai-tenant-e2e-shared-a-w3-56dbc0/auth-test-model-56dbc0-kserve-fd985d4cf-2fmsj"
Sep 09 02:51:15.809311 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:15.809251    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fed344c4-8783-44fb-af19-c7e6971c5ffb-dshm\") pod \"auth-test-model-56dbc0-kserve-fd985d4cf-2fmsj\" (UID: \"fed344c4-8783-44fb-af19-c7e6971c5ffb\") " pod="ai-tenant-e2e-shared-a-w3-56dbc0/auth-test-model-56dbc0-kserve-fd985d4cf-2fmsj"
Sep 09 02:51:15.809401 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:15.809373    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgnpf\" (UniqueName: \"kubernetes.io/projected/fed344c4-8783-44fb-af19-c7e6971c5ffb-kube-api-access-kgnpf\") pod \"auth-test-model-56dbc0-kserve-fd985d4cf-2fmsj\" (UID: \"fed344c4-8783-44fb-af19-c7e6971c5ffb\") " pod="ai-tenant-e2e-shared-a-w3-56dbc0/auth-test-model-56dbc0-kserve-fd985d4cf-2fmsj"
Sep 09 02:51:15.809587 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:15.809563    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fed344c4-8783-44fb-af19-c7e6971c5ffb-tmp-dir\") pod \"auth-test-model-56dbc0-kserve-fd985d4cf-2fmsj\" (UID: \"fed344c4-8783-44fb-af19-c7e6971c5ffb\") " pod="ai-tenant-e2e-shared-a-w3-56dbc0/auth-test-model-56dbc0-kserve-fd985d4cf-2fmsj"
Sep 09 02:51:15.809755 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:15.809595    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fed344c4-8783-44fb-af19-c7e6971c5ffb-model-cache\") pod \"auth-test-model-56dbc0-kserve-fd985d4cf-2fmsj\" (UID: \"fed344c4-8783-44fb-af19-c7e6971c5ffb\") " pod="ai-tenant-e2e-shared-a-w3-56dbc0/auth-test-model-56dbc0-kserve-fd985d4cf-2fmsj"
Sep 09 02:51:15.809831 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:15.809636    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fed344c4-8783-44fb-af19-c7e6971c5ffb-home\") pod \"auth-test-model-56dbc0-kserve-fd985d4cf-2fmsj\" (UID: \"fed344c4-8783-44fb-af19-c7e6971c5ffb\") " pod="ai-tenant-e2e-shared-a-w3-56dbc0/auth-test-model-56dbc0-kserve-fd985d4cf-2fmsj"
Sep 09 02:51:15.811420 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:15.811396    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fed344c4-8783-44fb-af19-c7e6971c5ffb-dshm\") pod \"auth-test-model-56dbc0-kserve-fd985d4cf-2fmsj\" (UID: \"fed344c4-8783-44fb-af19-c7e6971c5ffb\") " pod="ai-tenant-e2e-shared-a-w3-56dbc0/auth-test-model-56dbc0-kserve-fd985d4cf-2fmsj"
Sep 09 02:51:15.811523 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:15.811475    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fed344c4-8783-44fb-af19-c7e6971c5ffb-tls-certs\") pod \"auth-test-model-56dbc0-kserve-fd985d4cf-2fmsj\" (UID: \"fed344c4-8783-44fb-af19-c7e6971c5ffb\") " pod="ai-tenant-e2e-shared-a-w3-56dbc0/auth-test-model-56dbc0-kserve-fd985d4cf-2fmsj"
Sep 09 02:51:15.821259 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:15.821224    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgnpf\" (UniqueName: \"kubernetes.io/projected/fed344c4-8783-44fb-af19-c7e6971c5ffb-kube-api-access-kgnpf\") pod \"auth-test-model-56dbc0-kserve-fd985d4cf-2fmsj\" (UID: \"fed344c4-8783-44fb-af19-c7e6971c5ffb\") " pod="ai-tenant-e2e-shared-a-w3-56dbc0/auth-test-model-56dbc0-kserve-fd985d4cf-2fmsj"
Sep 09 02:51:15.949225 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:15.949145    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-56dbc0/auth-test-model-56dbc0-kserve-fd985d4cf-2fmsj"
Sep 09 02:51:16.007349 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:16.007277    2570 patch_prober.go:28] interesting pod/e2e-shared-b-w2-f5aebb-openshift-default-7c76c475d-r9lxr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 09 02:51:16.007491 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:16.007364    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-f5aebb-openshift-default-7c76c475d-r9lxr" podUID="7463a0d2-3ecb-479f-b47b-b4b5e8190396" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 09 02:51:16.228035 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:16.227893    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-56dbc0/auth-test-model-56dbc0-kserve-fd985d4cf-2fmsj"]
Sep 09 02:51:17.059818 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:17.059786    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-56dbc0/auth-test-model-56dbc0-kserve-fd985d4cf-2fmsj" event={"ID":"fed344c4-8783-44fb-af19-c7e6971c5ffb","Type":"ContainerStarted","Data":"0c1e1c4d67301fed5b892f26323e1527b3b4f8ce5bc139198b532496444db9bd"}
Sep 09 02:51:17.060181 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:17.059835    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-56dbc0/auth-test-model-56dbc0-kserve-fd985d4cf-2fmsj" event={"ID":"fed344c4-8783-44fb-af19-c7e6971c5ffb","Type":"ContainerStarted","Data":"484fdade4f3090deecbe20e2a62d30ab2a9d8a06b5445053e8c56b669338a46d"}
Sep 09 02:51:17.080172 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:17.080127    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-56dbc0/auth-test-model-56dbc0-kserve-fd985d4cf-2fmsj" podStartSLOduration=2.080111672 podStartE2EDuration="2.080111672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:51:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:51:17.077548957 +0000 UTC m=+1259.331852604" watchObservedRunningTime="2026-09-09 02:51:17.080111672 +0000 UTC m=+1259.334415308"
Sep 09 02:51:18.066167 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:18.066134    2570 generic.go:356] "Generic (PLEG): container finished" podID="7463a0d2-3ecb-479f-b47b-b4b5e8190396" containerID="99723178ceaf97f1e67cf2df749aee119e4fab4187313d3023476ba2c50c5e11" exitCode=0
Sep 09 02:51:18.066566 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:18.066205    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-f5aebb-openshift-default-7c76c475d-r9lxr" event={"ID":"7463a0d2-3ecb-479f-b47b-b4b5e8190396","Type":"ContainerDied","Data":"99723178ceaf97f1e67cf2df749aee119e4fab4187313d3023476ba2c50c5e11"}
Sep 09 02:51:18.066644 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:18.066629    2570 scope.go:117] "RemoveContainer" containerID="99723178ceaf97f1e67cf2df749aee119e4fab4187313d3023476ba2c50c5e11"
Sep 09 02:51:18.067704 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:18.067678    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 02:51:18.067814 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:18.067738    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 02:51:18.067814 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:18.067768    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 02:51:19.073287 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:19.073248    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-f5aebb-openshift-default-7c76c475d-r9lxr" event={"ID":"7463a0d2-3ecb-479f-b47b-b4b5e8190396","Type":"ContainerStarted","Data":"f8533d367700abbbe51870e99aa0628352be9d5ccd943215c616f30dd05d07fb"}
Sep 09 02:51:19.659747 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:19.659714    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-f5aebb-openshift-default-7c76c475d-r9lxr"
Sep 09 02:51:19.660097 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:19.660004    2570 patch_prober.go:28] interesting pod/e2e-shared-b-w2-f5aebb-openshift-default-7c76c475d-r9lxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Sep 09 02:51:19.660298 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:19.660276    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-f5aebb-openshift-default-7c76c475d-r9lxr" podUID="7463a0d2-3ecb-479f-b47b-b4b5e8190396" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Sep 09 02:51:20.146559 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:20.146506    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-99qjf"]
Sep 09 02:51:20.660230 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:20.660189    2570 patch_prober.go:28] interesting pod/e2e-shared-b-w2-f5aebb-openshift-default-7c76c475d-r9lxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Sep 09 02:51:20.660515 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:20.660485    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-f5aebb-openshift-default-7c76c475d-r9lxr" podUID="7463a0d2-3ecb-479f-b47b-b4b5e8190396" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Sep 09 02:51:21.659778 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:21.659734    2570 patch_prober.go:28] interesting pod/e2e-shared-b-w2-f5aebb-openshift-default-7c76c475d-r9lxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Sep 09 02:51:21.660319 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:21.659809    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-f5aebb-openshift-default-7c76c475d-r9lxr" podUID="7463a0d2-3ecb-479f-b47b-b4b5e8190396" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Sep 09 02:51:22.659886 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:22.659844    2570 patch_prober.go:28] interesting pod/e2e-shared-b-w2-f5aebb-openshift-default-7c76c475d-r9lxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Sep 09 02:51:22.660504 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:22.660473    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-f5aebb-openshift-default-7c76c475d-r9lxr" podUID="7463a0d2-3ecb-479f-b47b-b4b5e8190396" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Sep 09 02:51:23.659638 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:23.659592    2570 patch_prober.go:28] interesting pod/e2e-shared-b-w2-f5aebb-openshift-default-7c76c475d-r9lxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Sep 09 02:51:23.659848 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:23.659663    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-f5aebb-openshift-default-7c76c475d-r9lxr" podUID="7463a0d2-3ecb-479f-b47b-b4b5e8190396" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Sep 09 02:51:24.663266 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:24.663234    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-f5aebb-openshift-default-7c76c475d-r9lxr"
Sep 09 02:51:24.663712 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:24.663495    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-f5aebb-openshift-default-7c76c475d-r9lxr"
Sep 09 02:51:24.664282 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:24.664265    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-f5aebb-openshift-default-7c76c475d-r9lxr"
Sep 09 02:51:25.949676 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:25.949644    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-56dbc0/auth-test-model-56dbc0-kserve-fd985d4cf-2fmsj"
Sep 09 02:51:25.950065 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:25.949691    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-56dbc0/auth-test-model-56dbc0-kserve-fd985d4cf-2fmsj"
Sep 09 02:51:25.962192 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:25.962161    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-56dbc0/auth-test-model-56dbc0-kserve-fd985d4cf-2fmsj"
Sep 09 02:51:26.120108 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:26.120078    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-56dbc0/auth-test-model-56dbc0-kserve-fd985d4cf-2fmsj"
Sep 09 02:51:39.146108 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:39.146076    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-ad1d02af-openshift-default-95c789d96-6lgzq"]
Sep 09 02:51:39.150636 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:39.150612    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ad1d02af-openshift-default-95c789d96-6lgzq"
Sep 09 02:51:39.153384 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:39.153365    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-ad1d02af-openshift-default-dockercfg-ntclf\""
Sep 09 02:51:39.163177 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:39.163155    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-ad1d02af-openshift-default-95c789d96-6lgzq"]
Sep 09 02:51:39.240031 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:39.239962    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff129a48-26ae-4893-9ca3-f4501cf6eb00-istiod-ca-cert\") pod \"e2e-mt-ad1d02af-openshift-default-95c789d96-6lgzq\" (UID: \"ff129a48-26ae-4893-9ca3-f4501cf6eb00\") " pod="openshift-ingress/e2e-mt-ad1d02af-openshift-default-95c789d96-6lgzq"
Sep 09 02:51:39.240212 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:39.240084    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff129a48-26ae-4893-9ca3-f4501cf6eb00-istio-podinfo\") pod \"e2e-mt-ad1d02af-openshift-default-95c789d96-6lgzq\" (UID: \"ff129a48-26ae-4893-9ca3-f4501cf6eb00\") " pod="openshift-ingress/e2e-mt-ad1d02af-openshift-default-95c789d96-6lgzq"
Sep 09 02:51:39.240212 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:39.240138    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff129a48-26ae-4893-9ca3-f4501cf6eb00-credential-socket\") pod \"e2e-mt-ad1d02af-openshift-default-95c789d96-6lgzq\" (UID: \"ff129a48-26ae-4893-9ca3-f4501cf6eb00\") " pod="openshift-ingress/e2e-mt-ad1d02af-openshift-default-95c789d96-6lgzq"
Sep 09 02:51:39.240212 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:39.240169    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff129a48-26ae-4893-9ca3-f4501cf6eb00-istio-data\") pod \"e2e-mt-ad1d02af-openshift-default-95c789d96-6lgzq\" (UID: \"ff129a48-26ae-4893-9ca3-f4501cf6eb00\") " pod="openshift-ingress/e2e-mt-ad1d02af-openshift-default-95c789d96-6lgzq"
Sep 09 02:51:39.240212 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:39.240186    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q545j\" (UniqueName: \"kubernetes.io/projected/ff129a48-26ae-4893-9ca3-f4501cf6eb00-kube-api-access-q545j\") pod \"e2e-mt-ad1d02af-openshift-default-95c789d96-6lgzq\" (UID: \"ff129a48-26ae-4893-9ca3-f4501cf6eb00\") " pod="openshift-ingress/e2e-mt-ad1d02af-openshift-default-95c789d96-6lgzq"
Sep 09 02:51:39.240395 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:39.240257    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff129a48-26ae-4893-9ca3-f4501cf6eb00-workload-certs\") pod \"e2e-mt-ad1d02af-openshift-default-95c789d96-6lgzq\" (UID: \"ff129a48-26ae-4893-9ca3-f4501cf6eb00\") " pod="openshift-ingress/e2e-mt-ad1d02af-openshift-default-95c789d96-6lgzq"
Sep 09 02:51:39.240395 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:39.240295    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff129a48-26ae-4893-9ca3-f4501cf6eb00-workload-socket\") pod \"e2e-mt-ad1d02af-openshift-default-95c789d96-6lgzq\" (UID: \"ff129a48-26ae-4893-9ca3-f4501cf6eb00\") " pod="openshift-ingress/e2e-mt-ad1d02af-openshift-default-95c789d96-6lgzq"
Sep 09 02:51:39.240395 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:39.240312    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff129a48-26ae-4893-9ca3-f4501cf6eb00-istio-token\") pod \"e2e-mt-ad1d02af-openshift-default-95c789d96-6lgzq\" (UID: \"ff129a48-26ae-4893-9ca3-f4501cf6eb00\") " pod="openshift-ingress/e2e-mt-ad1d02af-openshift-default-95c789d96-6lgzq"
Sep 09 02:51:39.240395 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:39.240351    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff129a48-26ae-4893-9ca3-f4501cf6eb00-istio-envoy\") pod \"e2e-mt-ad1d02af-openshift-default-95c789d96-6lgzq\" (UID: \"ff129a48-26ae-4893-9ca3-f4501cf6eb00\") " pod="openshift-ingress/e2e-mt-ad1d02af-openshift-default-95c789d96-6lgzq"
Sep 09 02:51:39.341567 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:39.341530    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff129a48-26ae-4893-9ca3-f4501cf6eb00-istio-podinfo\") pod \"e2e-mt-ad1d02af-openshift-default-95c789d96-6lgzq\" (UID: \"ff129a48-26ae-4893-9ca3-f4501cf6eb00\") " pod="openshift-ingress/e2e-mt-ad1d02af-openshift-default-95c789d96-6lgzq"
Sep 09 02:51:39.341733 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:39.341581    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff129a48-26ae-4893-9ca3-f4501cf6eb00-credential-socket\") pod \"e2e-mt-ad1d02af-openshift-default-95c789d96-6lgzq\" (UID: \"ff129a48-26ae-4893-9ca3-f4501cf6eb00\") " pod="openshift-ingress/e2e-mt-ad1d02af-openshift-default-95c789d96-6lgzq"
Sep 09 02:51:39.341733 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:39.341609    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff129a48-26ae-4893-9ca3-f4501cf6eb00-istio-data\") pod \"e2e-mt-ad1d02af-openshift-default-95c789d96-6lgzq\" (UID: \"ff129a48-26ae-4893-9ca3-f4501cf6eb00\") " pod="openshift-ingress/e2e-mt-ad1d02af-openshift-default-95c789d96-6lgzq"
Sep 09 02:51:39.341733 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:39.341626    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q545j\" (UniqueName: \"kubernetes.io/projected/ff129a48-26ae-4893-9ca3-f4501cf6eb00-kube-api-access-q545j\") pod \"e2e-mt-ad1d02af-openshift-default-95c789d96-6lgzq\" (UID: \"ff129a48-26ae-4893-9ca3-f4501cf6eb00\") " pod="openshift-ingress/e2e-mt-ad1d02af-openshift-default-95c789d96-6lgzq"
Sep 09 02:51:39.341733 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:39.341666    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff129a48-26ae-4893-9ca3-f4501cf6eb00-workload-certs\") pod \"e2e-mt-ad1d02af-openshift-default-95c789d96-6lgzq\" (UID: \"ff129a48-26ae-4893-9ca3-f4501cf6eb00\") " pod="openshift-ingress/e2e-mt-ad1d02af-openshift-default-95c789d96-6lgzq"
Sep 09 02:51:39.341733 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:39.341693    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff129a48-26ae-4893-9ca3-f4501cf6eb00-workload-socket\") pod \"e2e-mt-ad1d02af-openshift-default-95c789d96-6lgzq\" (UID: \"ff129a48-26ae-4893-9ca3-f4501cf6eb00\") " pod="openshift-ingress/e2e-mt-ad1d02af-openshift-default-95c789d96-6lgzq"
Sep 09 02:51:39.341733 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:39.341719    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff129a48-26ae-4893-9ca3-f4501cf6eb00-istio-token\") pod \"e2e-mt-ad1d02af-openshift-default-95c789d96-6lgzq\" (UID: \"ff129a48-26ae-4893-9ca3-f4501cf6eb00\") " pod="openshift-ingress/e2e-mt-ad1d02af-openshift-default-95c789d96-6lgzq"
Sep 09 02:51:39.342081 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:39.341742    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff129a48-26ae-4893-9ca3-f4501cf6eb00-istio-envoy\") pod \"e2e-mt-ad1d02af-openshift-default-95c789d96-6lgzq\" (UID: \"ff129a48-26ae-4893-9ca3-f4501cf6eb00\") " pod="openshift-ingress/e2e-mt-ad1d02af-openshift-default-95c789d96-6lgzq"
Sep 09 02:51:39.342081 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:39.341803    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff129a48-26ae-4893-9ca3-f4501cf6eb00-istiod-ca-cert\") pod \"e2e-mt-ad1d02af-openshift-default-95c789d96-6lgzq\" (UID: \"ff129a48-26ae-4893-9ca3-f4501cf6eb00\") " pod="openshift-ingress/e2e-mt-ad1d02af-openshift-default-95c789d96-6lgzq"
Sep 09 02:51:39.342182 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:39.342128    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff129a48-26ae-4893-9ca3-f4501cf6eb00-credential-socket\") pod \"e2e-mt-ad1d02af-openshift-default-95c789d96-6lgzq\" (UID: \"ff129a48-26ae-4893-9ca3-f4501cf6eb00\") " pod="openshift-ingress/e2e-mt-ad1d02af-openshift-default-95c789d96-6lgzq"
Sep 09 02:51:39.342226 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:39.342186    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff129a48-26ae-4893-9ca3-f4501cf6eb00-workload-socket\") pod \"e2e-mt-ad1d02af-openshift-default-95c789d96-6lgzq\" (UID: \"ff129a48-26ae-4893-9ca3-f4501cf6eb00\") " pod="openshift-ingress/e2e-mt-ad1d02af-openshift-default-95c789d96-6lgzq"
Sep 09 02:51:39.342303 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:39.342278    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff129a48-26ae-4893-9ca3-f4501cf6eb00-workload-certs\") pod \"e2e-mt-ad1d02af-openshift-default-95c789d96-6lgzq\" (UID: \"ff129a48-26ae-4893-9ca3-f4501cf6eb00\") " pod="openshift-ingress/e2e-mt-ad1d02af-openshift-default-95c789d96-6lgzq"
Sep 09 02:51:39.342362 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:39.342311    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff129a48-26ae-4893-9ca3-f4501cf6eb00-istio-data\") pod \"e2e-mt-ad1d02af-openshift-default-95c789d96-6lgzq\" (UID: \"ff129a48-26ae-4893-9ca3-f4501cf6eb00\") " pod="openshift-ingress/e2e-mt-ad1d02af-openshift-default-95c789d96-6lgzq"
Sep 09 02:51:39.342535 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:39.342514    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff129a48-26ae-4893-9ca3-f4501cf6eb00-istiod-ca-cert\") pod \"e2e-mt-ad1d02af-openshift-default-95c789d96-6lgzq\" (UID: \"ff129a48-26ae-4893-9ca3-f4501cf6eb00\") " pod="openshift-ingress/e2e-mt-ad1d02af-openshift-default-95c789d96-6lgzq"
Sep 09 02:51:39.344201 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:39.344177    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff129a48-26ae-4893-9ca3-f4501cf6eb00-istio-envoy\") pod \"e2e-mt-ad1d02af-openshift-default-95c789d96-6lgzq\" (UID: \"ff129a48-26ae-4893-9ca3-f4501cf6eb00\") " pod="openshift-ingress/e2e-mt-ad1d02af-openshift-default-95c789d96-6lgzq"
Sep 09 02:51:39.344398 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:39.344381    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff129a48-26ae-4893-9ca3-f4501cf6eb00-istio-podinfo\") pod \"e2e-mt-ad1d02af-openshift-default-95c789d96-6lgzq\" (UID: \"ff129a48-26ae-4893-9ca3-f4501cf6eb00\") " pod="openshift-ingress/e2e-mt-ad1d02af-openshift-default-95c789d96-6lgzq"
Sep 09 02:51:39.349750 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:39.349727    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff129a48-26ae-4893-9ca3-f4501cf6eb00-istio-token\") pod \"e2e-mt-ad1d02af-openshift-default-95c789d96-6lgzq\" (UID: \"ff129a48-26ae-4893-9ca3-f4501cf6eb00\") " pod="openshift-ingress/e2e-mt-ad1d02af-openshift-default-95c789d96-6lgzq"
Sep 09 02:51:39.350643 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:39.350625    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q545j\" (UniqueName: \"kubernetes.io/projected/ff129a48-26ae-4893-9ca3-f4501cf6eb00-kube-api-access-q545j\") pod \"e2e-mt-ad1d02af-openshift-default-95c789d96-6lgzq\" (UID: \"ff129a48-26ae-4893-9ca3-f4501cf6eb00\") " pod="openshift-ingress/e2e-mt-ad1d02af-openshift-default-95c789d96-6lgzq"
Sep 09 02:51:39.461677 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:39.461603    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ad1d02af-openshift-default-95c789d96-6lgzq"
Sep 09 02:51:39.741330 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:39.741302    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-ad1d02af-openshift-default-95c789d96-6lgzq"]
Sep 09 02:51:39.743608 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:51:39.743545    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff129a48_26ae_4893_9ca3_f4501cf6eb00.slice/crio-a4a82ecb2561e8a110496bbc4e979ed7bb070a1ac905ce9bc479025e72f42167 WatchSource:0}: Error finding container a4a82ecb2561e8a110496bbc4e979ed7bb070a1ac905ce9bc479025e72f42167: Status 404 returned error can't find the container with id a4a82ecb2561e8a110496bbc4e979ed7bb070a1ac905ce9bc479025e72f42167
Sep 09 02:51:39.745940 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:39.745899    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 02:51:39.746042 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:39.746016    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 02:51:39.746106 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:39.746090    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 02:51:40.191762 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:40.191726    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ad1d02af-openshift-default-95c789d96-6lgzq" event={"ID":"ff129a48-26ae-4893-9ca3-f4501cf6eb00","Type":"ContainerStarted","Data":"d48939f2ab9962c0ed88faf8169cdb57c7e73ef99ef2e9fde411a41996fda1b1"}
Sep 09 02:51:40.192163 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:40.191784    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ad1d02af-openshift-default-95c789d96-6lgzq" event={"ID":"ff129a48-26ae-4893-9ca3-f4501cf6eb00","Type":"ContainerStarted","Data":"a4a82ecb2561e8a110496bbc4e979ed7bb070a1ac905ce9bc479025e72f42167"}
Sep 09 02:51:40.214420 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:40.214364    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-ad1d02af-openshift-default-95c789d96-6lgzq" podStartSLOduration=1.214348256 podStartE2EDuration="1.214348256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:51:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:51:40.211755329 +0000 UTC m=+1282.466058999" watchObservedRunningTime="2026-09-09 02:51:40.214348256 +0000 UTC m=+1282.468651892"
Sep 09 02:51:40.461823 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:40.461733    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-ad1d02af-openshift-default-95c789d96-6lgzq"
Sep 09 02:51:40.462896 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:40.462865    2570 patch_prober.go:28] interesting pod/e2e-mt-ad1d02af-openshift-default-95c789d96-6lgzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Sep 09 02:51:40.463010 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:40.462927    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ad1d02af-openshift-default-95c789d96-6lgzq" podUID="ff129a48-26ae-4893-9ca3-f4501cf6eb00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Sep 09 02:51:41.467108 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:41.467077    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-ad1d02af-openshift-default-95c789d96-6lgzq"
Sep 09 02:51:42.202244 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:42.202211    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-ad1d02af-openshift-default-95c789d96-6lgzq"
Sep 09 02:51:42.203245 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:42.203220    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-ad1d02af-openshift-default-95c789d96-6lgzq"
Sep 09 02:51:51.949590 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:51.949549    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-99qjf"]
Sep 09 02:51:55.643013 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:55.640868    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-c173ad/auth-test-model-c173ad-kserve-7b5d66d859-x9gp8"]
Sep 09 02:51:55.657232 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:55.657179    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-c173ad/auth-test-model-c173ad-kserve-7b5d66d859-x9gp8"
Sep 09 02:51:55.661065 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:55.661036    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-c173ad\"/\"kube-root-ca.crt\""
Sep 09 02:51:55.661466 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:55.661447    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-c173ad\"/\"auth-test-model-c173ad-kserve-self-signed-certs\""
Sep 09 02:51:55.661766 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:55.661751    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-c173ad\"/\"default-dockercfg-99zkq\""
Sep 09 02:51:55.662126 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:55.662107    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-c173ad\"/\"openshift-service-ca.crt\""
Sep 09 02:51:55.672833 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:55.672776    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-c173ad/auth-test-model-c173ad-kserve-7b5d66d859-x9gp8"]
Sep 09 02:51:55.806755 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:55.806726    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efb8e94e-5f79-4b8d-a2a3-23164dd1e70a-tmp-dir\") pod \"auth-test-model-c173ad-kserve-7b5d66d859-x9gp8\" (UID: \"efb8e94e-5f79-4b8d-a2a3-23164dd1e70a\") " pod="ai-tenant-e2e-shared-b-w3-c173ad/auth-test-model-c173ad-kserve-7b5d66d859-x9gp8"
Sep 09 02:51:55.806918 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:55.806787    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/efb8e94e-5f79-4b8d-a2a3-23164dd1e70a-dshm\") pod \"auth-test-model-c173ad-kserve-7b5d66d859-x9gp8\" (UID: \"efb8e94e-5f79-4b8d-a2a3-23164dd1e70a\") " pod="ai-tenant-e2e-shared-b-w3-c173ad/auth-test-model-c173ad-kserve-7b5d66d859-x9gp8"
Sep 09 02:51:55.806918 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:55.806820    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjwqk\" (UniqueName: \"kubernetes.io/projected/efb8e94e-5f79-4b8d-a2a3-23164dd1e70a-kube-api-access-bjwqk\") pod \"auth-test-model-c173ad-kserve-7b5d66d859-x9gp8\" (UID: \"efb8e94e-5f79-4b8d-a2a3-23164dd1e70a\") " pod="ai-tenant-e2e-shared-b-w3-c173ad/auth-test-model-c173ad-kserve-7b5d66d859-x9gp8"
Sep 09 02:51:55.806918 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:55.806861    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/efb8e94e-5f79-4b8d-a2a3-23164dd1e70a-model-cache\") pod \"auth-test-model-c173ad-kserve-7b5d66d859-x9gp8\" (UID: \"efb8e94e-5f79-4b8d-a2a3-23164dd1e70a\") " pod="ai-tenant-e2e-shared-b-w3-c173ad/auth-test-model-c173ad-kserve-7b5d66d859-x9gp8"
Sep 09 02:51:55.807056 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:55.806935    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efb8e94e-5f79-4b8d-a2a3-23164dd1e70a-tls-certs\") pod \"auth-test-model-c173ad-kserve-7b5d66d859-x9gp8\" (UID: \"efb8e94e-5f79-4b8d-a2a3-23164dd1e70a\") " pod="ai-tenant-e2e-shared-b-w3-c173ad/auth-test-model-c173ad-kserve-7b5d66d859-x9gp8"
Sep 09 02:51:55.807056 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:55.807015    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/efb8e94e-5f79-4b8d-a2a3-23164dd1e70a-home\") pod \"auth-test-model-c173ad-kserve-7b5d66d859-x9gp8\" (UID: \"efb8e94e-5f79-4b8d-a2a3-23164dd1e70a\") " pod="ai-tenant-e2e-shared-b-w3-c173ad/auth-test-model-c173ad-kserve-7b5d66d859-x9gp8"
Sep 09 02:51:55.908273 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:55.908176    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/efb8e94e-5f79-4b8d-a2a3-23164dd1e70a-dshm\") pod \"auth-test-model-c173ad-kserve-7b5d66d859-x9gp8\" (UID: \"efb8e94e-5f79-4b8d-a2a3-23164dd1e70a\") " pod="ai-tenant-e2e-shared-b-w3-c173ad/auth-test-model-c173ad-kserve-7b5d66d859-x9gp8"
Sep 09 02:51:55.908273 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:55.908225    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjwqk\" (UniqueName: \"kubernetes.io/projected/efb8e94e-5f79-4b8d-a2a3-23164dd1e70a-kube-api-access-bjwqk\") pod \"auth-test-model-c173ad-kserve-7b5d66d859-x9gp8\" (UID: \"efb8e94e-5f79-4b8d-a2a3-23164dd1e70a\") " pod="ai-tenant-e2e-shared-b-w3-c173ad/auth-test-model-c173ad-kserve-7b5d66d859-x9gp8"
Sep 09 02:51:55.908470 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:55.908295    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/efb8e94e-5f79-4b8d-a2a3-23164dd1e70a-model-cache\") pod \"auth-test-model-c173ad-kserve-7b5d66d859-x9gp8\" (UID: \"efb8e94e-5f79-4b8d-a2a3-23164dd1e70a\") " pod="ai-tenant-e2e-shared-b-w3-c173ad/auth-test-model-c173ad-kserve-7b5d66d859-x9gp8"
Sep 09 02:51:55.908470 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:55.908436    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efb8e94e-5f79-4b8d-a2a3-23164dd1e70a-tls-certs\") pod \"auth-test-model-c173ad-kserve-7b5d66d859-x9gp8\" (UID: \"efb8e94e-5f79-4b8d-a2a3-23164dd1e70a\") " pod="ai-tenant-e2e-shared-b-w3-c173ad/auth-test-model-c173ad-kserve-7b5d66d859-x9gp8"
Sep 09 02:51:55.908557 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:55.908499    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/efb8e94e-5f79-4b8d-a2a3-23164dd1e70a-home\") pod \"auth-test-model-c173ad-kserve-7b5d66d859-x9gp8\" (UID: \"efb8e94e-5f79-4b8d-a2a3-23164dd1e70a\") " pod="ai-tenant-e2e-shared-b-w3-c173ad/auth-test-model-c173ad-kserve-7b5d66d859-x9gp8"
Sep 09 02:51:55.908557 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:55.908549    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efb8e94e-5f79-4b8d-a2a3-23164dd1e70a-tmp-dir\") pod \"auth-test-model-c173ad-kserve-7b5d66d859-x9gp8\" (UID: \"efb8e94e-5f79-4b8d-a2a3-23164dd1e70a\") " pod="ai-tenant-e2e-shared-b-w3-c173ad/auth-test-model-c173ad-kserve-7b5d66d859-x9gp8"
Sep 09 02:51:55.908724 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:55.908702    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/efb8e94e-5f79-4b8d-a2a3-23164dd1e70a-model-cache\") pod \"auth-test-model-c173ad-kserve-7b5d66d859-x9gp8\" (UID: \"efb8e94e-5f79-4b8d-a2a3-23164dd1e70a\") " pod="ai-tenant-e2e-shared-b-w3-c173ad/auth-test-model-c173ad-kserve-7b5d66d859-x9gp8"
Sep 09 02:51:55.908911 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:55.908886    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/efb8e94e-5f79-4b8d-a2a3-23164dd1e70a-home\") pod \"auth-test-model-c173ad-kserve-7b5d66d859-x9gp8\" (UID: \"efb8e94e-5f79-4b8d-a2a3-23164dd1e70a\") " pod="ai-tenant-e2e-shared-b-w3-c173ad/auth-test-model-c173ad-kserve-7b5d66d859-x9gp8"
Sep 09 02:51:55.909068 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:55.908920    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efb8e94e-5f79-4b8d-a2a3-23164dd1e70a-tmp-dir\") pod \"auth-test-model-c173ad-kserve-7b5d66d859-x9gp8\" (UID: \"efb8e94e-5f79-4b8d-a2a3-23164dd1e70a\") " pod="ai-tenant-e2e-shared-b-w3-c173ad/auth-test-model-c173ad-kserve-7b5d66d859-x9gp8"
Sep 09 02:51:55.910540 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:55.910512    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/efb8e94e-5f79-4b8d-a2a3-23164dd1e70a-dshm\") pod \"auth-test-model-c173ad-kserve-7b5d66d859-x9gp8\" (UID: \"efb8e94e-5f79-4b8d-a2a3-23164dd1e70a\") " pod="ai-tenant-e2e-shared-b-w3-c173ad/auth-test-model-c173ad-kserve-7b5d66d859-x9gp8"
Sep 09 02:51:55.910875 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:55.910857    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efb8e94e-5f79-4b8d-a2a3-23164dd1e70a-tls-certs\") pod \"auth-test-model-c173ad-kserve-7b5d66d859-x9gp8\" (UID: \"efb8e94e-5f79-4b8d-a2a3-23164dd1e70a\") " pod="ai-tenant-e2e-shared-b-w3-c173ad/auth-test-model-c173ad-kserve-7b5d66d859-x9gp8"
Sep 09 02:51:55.916240 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:55.916220    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjwqk\" (UniqueName: \"kubernetes.io/projected/efb8e94e-5f79-4b8d-a2a3-23164dd1e70a-kube-api-access-bjwqk\") pod \"auth-test-model-c173ad-kserve-7b5d66d859-x9gp8\" (UID: \"efb8e94e-5f79-4b8d-a2a3-23164dd1e70a\") " pod="ai-tenant-e2e-shared-b-w3-c173ad/auth-test-model-c173ad-kserve-7b5d66d859-x9gp8"
Sep 09 02:51:55.983103 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:55.983077    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-c173ad/auth-test-model-c173ad-kserve-7b5d66d859-x9gp8"
Sep 09 02:51:56.261471 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:56.260844    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-c173ad/auth-test-model-c173ad-kserve-7b5d66d859-x9gp8"]
Sep 09 02:51:56.279431 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:56.279404    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-c173ad/auth-test-model-c173ad-kserve-7b5d66d859-x9gp8" event={"ID":"efb8e94e-5f79-4b8d-a2a3-23164dd1e70a","Type":"ContainerStarted","Data":"5be63250767a7f76e8c7c43ecab861a386c48393735b91c61adf6f932001b511"}
Sep 09 02:51:57.286929 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:57.286887    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-c173ad/auth-test-model-c173ad-kserve-7b5d66d859-x9gp8" event={"ID":"efb8e94e-5f79-4b8d-a2a3-23164dd1e70a","Type":"ContainerStarted","Data":"c881a6818cf1f2fec8cf6a7771a44edfe40d57948a99f79ad76a96d43fa2dad9"}
Sep 09 02:51:57.306703 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:51:57.306653    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-c173ad/auth-test-model-c173ad-kserve-7b5d66d859-x9gp8" podStartSLOduration=2.306638444 podStartE2EDuration="2.306638444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:51:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:51:57.306072117 +0000 UTC m=+1299.560375753" watchObservedRunningTime="2026-09-09 02:51:57.306638444 +0000 UTC m=+1299.560942080"
Sep 09 02:52:02.315261 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:52:02.315179    2570 generic.go:356] "Generic (PLEG): container finished" podID="ff129a48-26ae-4893-9ca3-f4501cf6eb00" containerID="d48939f2ab9962c0ed88faf8169cdb57c7e73ef99ef2e9fde411a41996fda1b1" exitCode=0
Sep 09 02:52:02.315261 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:52:02.315251    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ad1d02af-openshift-default-95c789d96-6lgzq" event={"ID":"ff129a48-26ae-4893-9ca3-f4501cf6eb00","Type":"ContainerDied","Data":"d48939f2ab9962c0ed88faf8169cdb57c7e73ef99ef2e9fde411a41996fda1b1"}
Sep 09 02:52:02.315725 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:52:02.315710    2570 scope.go:117] "RemoveContainer" containerID="d48939f2ab9962c0ed88faf8169cdb57c7e73ef99ef2e9fde411a41996fda1b1"
Sep 09 02:52:02.316792 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:52:02.316766    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 02:52:02.316847 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:52:02.316833    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 02:52:02.316884 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:52:02.316868    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 02:52:03.324508 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:52:03.324458    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ad1d02af-openshift-default-95c789d96-6lgzq" event={"ID":"ff129a48-26ae-4893-9ca3-f4501cf6eb00","Type":"ContainerStarted","Data":"a6b13fa7fac1069e9764925897ae68fcc253846c6dc70bb8e48ed48cf0c14333"}
Sep 09 02:52:03.462015 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:52:03.461955    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-ad1d02af-openshift-default-95c789d96-6lgzq"
Sep 09 02:52:03.462319 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:52:03.462197    2570 patch_prober.go:28] interesting pod/e2e-mt-ad1d02af-openshift-default-95c789d96-6lgzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Sep 09 02:52:03.462569 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:52:03.462539    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ad1d02af-openshift-default-95c789d96-6lgzq" podUID="ff129a48-26ae-4893-9ca3-f4501cf6eb00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Sep 09 02:52:04.462707 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:52:04.462666    2570 patch_prober.go:28] interesting pod/e2e-mt-ad1d02af-openshift-default-95c789d96-6lgzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Sep 09 02:52:04.463500 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:52:04.463454    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ad1d02af-openshift-default-95c789d96-6lgzq" podUID="ff129a48-26ae-4893-9ca3-f4501cf6eb00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Sep 09 02:52:05.463416 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:52:05.462922    2570 patch_prober.go:28] interesting pod/e2e-mt-ad1d02af-openshift-default-95c789d96-6lgzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Sep 09 02:52:05.463416 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:52:05.463018    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ad1d02af-openshift-default-95c789d96-6lgzq" podUID="ff129a48-26ae-4893-9ca3-f4501cf6eb00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Sep 09 02:52:05.983277 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:52:05.983236    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-c173ad/auth-test-model-c173ad-kserve-7b5d66d859-x9gp8"
Sep 09 02:52:05.983457 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:52:05.983288    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-c173ad/auth-test-model-c173ad-kserve-7b5d66d859-x9gp8"
Sep 09 02:52:05.995287 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:52:05.995262    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-c173ad/auth-test-model-c173ad-kserve-7b5d66d859-x9gp8"
Sep 09 02:52:06.372733 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:52:06.372704    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-c173ad/auth-test-model-c173ad-kserve-7b5d66d859-x9gp8"
Sep 09 02:52:06.462356 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:52:06.462317    2570 patch_prober.go:28] interesting pod/e2e-mt-ad1d02af-openshift-default-95c789d96-6lgzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Sep 09 02:52:06.462627 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:52:06.462597    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ad1d02af-openshift-default-95c789d96-6lgzq" podUID="ff129a48-26ae-4893-9ca3-f4501cf6eb00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Sep 09 02:52:07.462511 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:52:07.462467    2570 patch_prober.go:28] interesting pod/e2e-mt-ad1d02af-openshift-default-95c789d96-6lgzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Sep 09 02:52:07.463019 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:52:07.462534    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ad1d02af-openshift-default-95c789d96-6lgzq" podUID="ff129a48-26ae-4893-9ca3-f4501cf6eb00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Sep 09 02:52:08.462164 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:52:08.462118    2570 patch_prober.go:28] interesting pod/e2e-mt-ad1d02af-openshift-default-95c789d96-6lgzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Sep 09 02:52:08.462384 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:52:08.462188    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ad1d02af-openshift-default-95c789d96-6lgzq" podUID="ff129a48-26ae-4893-9ca3-f4501cf6eb00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Sep 09 02:52:09.465942 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:52:09.465911    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-ad1d02af-openshift-default-95c789d96-6lgzq"
Sep 09 02:52:09.466322 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:52:09.466199    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-ad1d02af-openshift-default-95c789d96-6lgzq"
Sep 09 02:52:09.466912 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:52:09.466897    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-ad1d02af-openshift-default-95c789d96-6lgzq"
Sep 09 02:52:09.746412 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:52:09.746320    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-99qjf"]
Sep 09 02:52:18.366736 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:52:18.366706    2570 scope.go:117] "RemoveContainer" containerID="1cc27679f17e7002a33fee9802c7533bd40b3e96a4f83900b646b8d4b9232149"
Sep 09 02:52:20.352584 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:52:20.352544    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-99qjf"]
Sep 09 02:52:30.327809 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:52:30.327769    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-ad1d02af/e2e-lifecycle-model-ad1d02af-kserve-648d77ddc9-wrmhl"]
Sep 09 02:52:30.333285 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:52:30.333261    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-ad1d02af/e2e-lifecycle-model-ad1d02af-kserve-648d77ddc9-wrmhl"
Sep 09 02:52:30.336290 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:52:30.336266    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-ad1d02af\"/\"e2e-lifecycle-model-ad1d02af-kserve-self-signed-certs\""
Sep 09 02:52:30.336405 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:52:30.336367    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-ad1d02af\"/\"openshift-service-ca.crt\""
Sep 09 02:52:30.337732 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:52:30.337711    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-ad1d02af\"/\"default-dockercfg-snk49\""
Sep 09 02:52:30.337835 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:52:30.337738    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-ad1d02af\"/\"kube-root-ca.crt\""
Sep 09 02:52:30.342428 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:52:30.342403    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-ad1d02af/e2e-lifecycle-model-ad1d02af-kserve-648d77ddc9-wrmhl"]
Sep 09 02:52:30.439998 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:52:30.439934    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9fb20aea-e806-4045-8389-d574d0bd8537-home\") pod \"e2e-lifecycle-model-ad1d02af-kserve-648d77ddc9-wrmhl\" (UID: \"9fb20aea-e806-4045-8389-d574d0bd8537\") " pod="ai-tenant-e2e-mt-ad1d02af/e2e-lifecycle-model-ad1d02af-kserve-648d77ddc9-wrmhl"
Sep 09 02:52:30.440174 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:52:30.440044    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gqwj\" (UniqueName: \"kubernetes.io/projected/9fb20aea-e806-4045-8389-d574d0bd8537-kube-api-access-5gqwj\") pod \"e2e-lifecycle-model-ad1d02af-kserve-648d77ddc9-wrmhl\" (UID: \"9fb20aea-e806-4045-8389-d574d0bd8537\") " pod="ai-tenant-e2e-mt-ad1d02af/e2e-lifecycle-model-ad1d02af-kserve-648d77ddc9-wrmhl"
Sep 09 02:52:30.440174 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:52:30.440083    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9fb20aea-e806-4045-8389-d574d0bd8537-model-cache\") pod \"e2e-lifecycle-model-ad1d02af-kserve-648d77ddc9-wrmhl\" (UID: \"9fb20aea-e806-4045-8389-d574d0bd8537\") " pod="ai-tenant-e2e-mt-ad1d02af/e2e-lifecycle-model-ad1d02af-kserve-648d77ddc9-wrmhl"
Sep 09 02:52:30.440174 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:52:30.440125    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9fb20aea-e806-4045-8389-d574d0bd8537-tls-certs\") pod \"e2e-lifecycle-model-ad1d02af-kserve-648d77ddc9-wrmhl\" (UID: \"9fb20aea-e806-4045-8389-d574d0bd8537\") " pod="ai-tenant-e2e-mt-ad1d02af/e2e-lifecycle-model-ad1d02af-kserve-648d77ddc9-wrmhl"
Sep 09 02:52:30.440174 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:52:30.440149    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9fb20aea-e806-4045-8389-d574d0bd8537-dshm\") pod \"e2e-lifecycle-model-ad1d02af-kserve-648d77ddc9-wrmhl\" (UID: \"9fb20aea-e806-4045-8389-d574d0bd8537\") " pod="ai-tenant-e2e-mt-ad1d02af/e2e-lifecycle-model-ad1d02af-kserve-648d77ddc9-wrmhl"
Sep 09 02:52:30.440318 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:52:30.440244    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9fb20aea-e806-4045-8389-d574d0bd8537-tmp-dir\") pod \"e2e-lifecycle-model-ad1d02af-kserve-648d77ddc9-wrmhl\" (UID: \"9fb20aea-e806-4045-8389-d574d0bd8537\") " pod="ai-tenant-e2e-mt-ad1d02af/e2e-lifecycle-model-ad1d02af-kserve-648d77ddc9-wrmhl"
Sep 09 02:52:30.541592 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:52:30.541557    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9fb20aea-e806-4045-8389-d574d0bd8537-dshm\") pod \"e2e-lifecycle-model-ad1d02af-kserve-648d77ddc9-wrmhl\" (UID: \"9fb20aea-e806-4045-8389-d574d0bd8537\") " pod="ai-tenant-e2e-mt-ad1d02af/e2e-lifecycle-model-ad1d02af-kserve-648d77ddc9-wrmhl"
Sep 09 02:52:30.541746 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:52:30.541640    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9fb20aea-e806-4045-8389-d574d0bd8537-tmp-dir\") pod \"e2e-lifecycle-model-ad1d02af-kserve-648d77ddc9-wrmhl\" (UID: \"9fb20aea-e806-4045-8389-d574d0bd8537\") " pod="ai-tenant-e2e-mt-ad1d02af/e2e-lifecycle-model-ad1d02af-kserve-648d77ddc9-wrmhl"
Sep 09 02:52:30.541746 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:52:30.541675    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9fb20aea-e806-4045-8389-d574d0bd8537-home\") pod \"e2e-lifecycle-model-ad1d02af-kserve-648d77ddc9-wrmhl\" (UID: \"9fb20aea-e806-4045-8389-d574d0bd8537\") " pod="ai-tenant-e2e-mt-ad1d02af/e2e-lifecycle-model-ad1d02af-kserve-648d77ddc9-wrmhl"
Sep 09 02:52:30.541746 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:52:30.541731    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gqwj\" (UniqueName: \"kubernetes.io/projected/9fb20aea-e806-4045-8389-d574d0bd8537-kube-api-access-5gqwj\") pod \"e2e-lifecycle-model-ad1d02af-kserve-648d77ddc9-wrmhl\" (UID: \"9fb20aea-e806-4045-8389-d574d0bd8537\") " pod="ai-tenant-e2e-mt-ad1d02af/e2e-lifecycle-model-ad1d02af-kserve-648d77ddc9-wrmhl"
Sep 09 02:52:30.541933 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:52:30.541779    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9fb20aea-e806-4045-8389-d574d0bd8537-model-cache\") pod \"e2e-lifecycle-model-ad1d02af-kserve-648d77ddc9-wrmhl\" (UID: \"9fb20aea-e806-4045-8389-d574d0bd8537\") " pod="ai-tenant-e2e-mt-ad1d02af/e2e-lifecycle-model-ad1d02af-kserve-648d77ddc9-wrmhl"
Sep 09 02:52:30.541933 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:52:30.541819    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9fb20aea-e806-4045-8389-d574d0bd8537-tls-certs\") pod \"e2e-lifecycle-model-ad1d02af-kserve-648d77ddc9-wrmhl\" (UID: \"9fb20aea-e806-4045-8389-d574d0bd8537\") " pod="ai-tenant-e2e-mt-ad1d02af/e2e-lifecycle-model-ad1d02af-kserve-648d77ddc9-wrmhl"
Sep 09 02:52:30.542079 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:52:30.542057    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9fb20aea-e806-4045-8389-d574d0bd8537-tmp-dir\") pod \"e2e-lifecycle-model-ad1d02af-kserve-648d77ddc9-wrmhl\" (UID: \"9fb20aea-e806-4045-8389-d574d0bd8537\") " pod="ai-tenant-e2e-mt-ad1d02af/e2e-lifecycle-model-ad1d02af-kserve-648d77ddc9-wrmhl"
Sep 09 02:52:30.542159 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:52:30.542134    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9fb20aea-e806-4045-8389-d574d0bd8537-home\") pod \"e2e-lifecycle-model-ad1d02af-kserve-648d77ddc9-wrmhl\" (UID: \"9fb20aea-e806-4045-8389-d574d0bd8537\") " pod="ai-tenant-e2e-mt-ad1d02af/e2e-lifecycle-model-ad1d02af-kserve-648d77ddc9-wrmhl"
Sep 09 02:52:30.542215 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:52:30.542161    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9fb20aea-e806-4045-8389-d574d0bd8537-model-cache\") pod \"e2e-lifecycle-model-ad1d02af-kserve-648d77ddc9-wrmhl\" (UID: \"9fb20aea-e806-4045-8389-d574d0bd8537\") " pod="ai-tenant-e2e-mt-ad1d02af/e2e-lifecycle-model-ad1d02af-kserve-648d77ddc9-wrmhl"
Sep 09 02:52:30.544055 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:52:30.544027    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9fb20aea-e806-4045-8389-d574d0bd8537-dshm\") pod \"e2e-lifecycle-model-ad1d02af-kserve-648d77ddc9-wrmhl\" (UID: \"9fb20aea-e806-4045-8389-d574d0bd8537\") " pod="ai-tenant-e2e-mt-ad1d02af/e2e-lifecycle-model-ad1d02af-kserve-648d77ddc9-wrmhl"
Sep 09 02:52:30.544257 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:52:30.544242    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9fb20aea-e806-4045-8389-d574d0bd8537-tls-certs\") pod \"e2e-lifecycle-model-ad1d02af-kserve-648d77ddc9-wrmhl\" (UID: \"9fb20aea-e806-4045-8389-d574d0bd8537\") " pod="ai-tenant-e2e-mt-ad1d02af/e2e-lifecycle-model-ad1d02af-kserve-648d77ddc9-wrmhl"
Sep 09 02:52:30.550626 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:52:30.550606    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gqwj\" (UniqueName: \"kubernetes.io/projected/9fb20aea-e806-4045-8389-d574d0bd8537-kube-api-access-5gqwj\") pod \"e2e-lifecycle-model-ad1d02af-kserve-648d77ddc9-wrmhl\" (UID: \"9fb20aea-e806-4045-8389-d574d0bd8537\") " pod="ai-tenant-e2e-mt-ad1d02af/e2e-lifecycle-model-ad1d02af-kserve-648d77ddc9-wrmhl"
Sep 09 02:52:30.645171 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:52:30.645146    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-ad1d02af/e2e-lifecycle-model-ad1d02af-kserve-648d77ddc9-wrmhl"
Sep 09 02:52:30.917444 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:52:30.917416    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-ad1d02af/e2e-lifecycle-model-ad1d02af-kserve-648d77ddc9-wrmhl"]
Sep 09 02:52:30.919641 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:52:30.919611    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9fb20aea_e806_4045_8389_d574d0bd8537.slice/crio-15d28fbc04b6f09ddd66fa86f46680511bb91415662ef11800f8b3ee2a044f7e WatchSource:0}: Error finding container 15d28fbc04b6f09ddd66fa86f46680511bb91415662ef11800f8b3ee2a044f7e: Status 404 returned error can't find the container with id 15d28fbc04b6f09ddd66fa86f46680511bb91415662ef11800f8b3ee2a044f7e
Sep 09 02:52:31.484276 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:52:31.484246    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-ad1d02af/e2e-lifecycle-model-ad1d02af-kserve-648d77ddc9-wrmhl" event={"ID":"9fb20aea-e806-4045-8389-d574d0bd8537","Type":"ContainerStarted","Data":"6252e547691cfe3f5511b70645567716e9f4a9b918b06a73fdd47b47a98b53ef"}
Sep 09 02:52:31.484276 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:52:31.484283    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-ad1d02af/e2e-lifecycle-model-ad1d02af-kserve-648d77ddc9-wrmhl" event={"ID":"9fb20aea-e806-4045-8389-d574d0bd8537","Type":"ContainerStarted","Data":"15d28fbc04b6f09ddd66fa86f46680511bb91415662ef11800f8b3ee2a044f7e"}
Sep 09 02:52:31.503522 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:52:31.503474    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-ad1d02af/e2e-lifecycle-model-ad1d02af-kserve-648d77ddc9-wrmhl" podStartSLOduration=1.503459157 podStartE2EDuration="1.503459157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:52:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:52:31.502286137 +0000 UTC m=+1333.756589774" watchObservedRunningTime="2026-09-09 02:52:31.503459157 +0000 UTC m=+1333.757762823"
Sep 09 02:52:33.949154 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:52:33.949107    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-99qjf"]
Sep 09 02:52:40.645716 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:52:40.645629    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-ad1d02af/e2e-lifecycle-model-ad1d02af-kserve-648d77ddc9-wrmhl"
Sep 09 02:52:40.645716 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:52:40.645680    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-ad1d02af/e2e-lifecycle-model-ad1d02af-kserve-648d77ddc9-wrmhl"
Sep 09 02:52:40.657633 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:52:40.657611    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-ad1d02af/e2e-lifecycle-model-ad1d02af-kserve-648d77ddc9-wrmhl"
Sep 09 02:52:41.549334 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:52:41.549301    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-ad1d02af/e2e-lifecycle-model-ad1d02af-kserve-648d77ddc9-wrmhl"
Sep 09 02:52:53.441814 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:52:53.441775    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-99qjf"]
Sep 09 02:52:56.801436 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:52:56.801399    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ad1d02af-rr4x4"]
Sep 09 02:52:56.804947 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:52:56.804931    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ad1d02af-rr4x4"
Sep 09 02:52:56.808811 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:52:56.808781    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 02:52:56.808936 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:52:56.808845    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vbr52\""
Sep 09 02:52:56.813083 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:52:56.813061    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ad1d02af-rr4x4"]
Sep 09 02:52:56.896026 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:52:56.895969    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/349ce7e7-a858-449f-a454-cf03dac0b87e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-ad1d02af-rr4x4\" (UID: \"349ce7e7-a858-449f-a454-cf03dac0b87e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ad1d02af-rr4x4"
Sep 09 02:52:56.996708 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:52:56.996667    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/349ce7e7-a858-449f-a454-cf03dac0b87e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-ad1d02af-rr4x4\" (UID: \"349ce7e7-a858-449f-a454-cf03dac0b87e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ad1d02af-rr4x4"
Sep 09 02:52:56.997352 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:52:56.997332    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/349ce7e7-a858-449f-a454-cf03dac0b87e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-ad1d02af-rr4x4\" (UID: \"349ce7e7-a858-449f-a454-cf03dac0b87e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ad1d02af-rr4x4"
Sep 09 02:52:57.115981 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:52:57.115939    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ad1d02af-rr4x4"
Sep 09 02:52:57.387190 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:52:57.387158    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ad1d02af-rr4x4"]
Sep 09 02:52:57.388603 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:52:57.388575    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod349ce7e7_a858_449f_a454_cf03dac0b87e.slice/crio-eed4e25359f1293743cc850fb437496f7e9e0566f8902ea4194640261d8a2943 WatchSource:0}: Error finding container eed4e25359f1293743cc850fb437496f7e9e0566f8902ea4194640261d8a2943: Status 404 returned error can't find the container with id eed4e25359f1293743cc850fb437496f7e9e0566f8902ea4194640261d8a2943
Sep 09 02:52:57.624544 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:52:57.624507    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ad1d02af-rr4x4" event={"ID":"349ce7e7-a858-449f-a454-cf03dac0b87e","Type":"ContainerStarted","Data":"eed4e25359f1293743cc850fb437496f7e9e0566f8902ea4194640261d8a2943"}
Sep 09 02:52:58.632023 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:52:58.631970    2570 generic.go:356] "Generic (PLEG): container finished" podID="349ce7e7-a858-449f-a454-cf03dac0b87e" containerID="1a90a50edd3e3ff4f78a08fc636783af85e4789fdcbd622ff8a8a77fa9b160a0" exitCode=0
Sep 09 02:52:58.632403 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:52:58.632026    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ad1d02af-rr4x4" event={"ID":"349ce7e7-a858-449f-a454-cf03dac0b87e","Type":"ContainerDied","Data":"1a90a50edd3e3ff4f78a08fc636783af85e4789fdcbd622ff8a8a77fa9b160a0"}
Sep 09 02:52:59.786724 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:52:59.786700    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ad1d02af-rr4x4"
Sep 09 02:52:59.823362 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:52:59.823339    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/349ce7e7-a858-449f-a454-cf03dac0b87e-maas-api-service-ca\") pod \"349ce7e7-a858-449f-a454-cf03dac0b87e\" (UID: \"349ce7e7-a858-449f-a454-cf03dac0b87e\") "
Sep 09 02:52:59.823662 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:52:59.823636    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/349ce7e7-a858-449f-a454-cf03dac0b87e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "349ce7e7-a858-449f-a454-cf03dac0b87e" (UID: "349ce7e7-a858-449f-a454-cf03dac0b87e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 02:52:59.924754 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:52:59.924690    2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/349ce7e7-a858-449f-a454-cf03dac0b87e-maas-api-service-ca\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:53:00.644621 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:53:00.644587    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ad1d02af-rr4x4" event={"ID":"349ce7e7-a858-449f-a454-cf03dac0b87e","Type":"ContainerDied","Data":"eed4e25359f1293743cc850fb437496f7e9e0566f8902ea4194640261d8a2943"}
Sep 09 02:53:00.644621 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:53:00.644615    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ad1d02af-rr4x4"
Sep 09 02:53:00.644853 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:53:00.644635    2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eed4e25359f1293743cc850fb437496f7e9e0566f8902ea4194640261d8a2943"
Sep 09 02:53:11.739865 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:53:11.739830    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-99qjf"]
Sep 09 02:53:12.532769 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:53:12.532738    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ad1d02af-rr4x4"]
Sep 09 02:53:12.535325 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:53:12.535300    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ad1d02af-rr4x4"]
Sep 09 02:53:14.223097 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:53:14.223064    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="349ce7e7-a858-449f-a454-cf03dac0b87e" path="/var/lib/kubelet/pods/349ce7e7-a858-449f-a454-cf03dac0b87e/volumes"
Sep 09 02:53:18.386412 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:53:18.386378    2570 scope.go:117] "RemoveContainer" containerID="15e3d6983dcf0946f0613690a09acc259da489e3af8711c93d87cf738259f0da"
Sep 09 02:53:18.395102 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:53:18.395069    2570 scope.go:117] "RemoveContainer" containerID="6a68316c8a574353b77e90cc8c8181986a6ae2b8ef18d9af507a19b305a7afa8"
Sep 09 02:53:20.696210 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:53:20.696174    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-ad1d02af/e2e-lifecycle-model-ad1d02af-kserve-648d77ddc9-wrmhl"]
Sep 09 02:53:29.639910 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:53:29.639869    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-99qjf"]
Sep 09 02:53:35.716069 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:53:35.716008    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-ad1d02af/e2e-lifecycle-model-ad1d02af-kserve-648d77ddc9-wrmhl" podUID="9fb20aea-e806-4045-8389-d574d0bd8537" containerName="main" containerID="cri-o://6252e547691cfe3f5511b70645567716e9f4a9b918b06a73fdd47b47a98b53ef" gracePeriod=45
Sep 09 02:53:35.836652 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:53:35.836624    2570 generic.go:356] "Generic (PLEG): container finished" podID="9fb20aea-e806-4045-8389-d574d0bd8537" containerID="6252e547691cfe3f5511b70645567716e9f4a9b918b06a73fdd47b47a98b53ef" exitCode=0
Sep 09 02:53:35.836778 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:53:35.836693    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-ad1d02af/e2e-lifecycle-model-ad1d02af-kserve-648d77ddc9-wrmhl" event={"ID":"9fb20aea-e806-4045-8389-d574d0bd8537","Type":"ContainerDied","Data":"6252e547691cfe3f5511b70645567716e9f4a9b918b06a73fdd47b47a98b53ef"}
Sep 09 02:53:35.975639 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:53:35.975586    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-ad1d02af/e2e-lifecycle-model-ad1d02af-kserve-648d77ddc9-wrmhl"
Sep 09 02:53:36.070300 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:53:36.070267    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9fb20aea-e806-4045-8389-d574d0bd8537-tmp-dir\") pod \"9fb20aea-e806-4045-8389-d574d0bd8537\" (UID: \"9fb20aea-e806-4045-8389-d574d0bd8537\") "
Sep 09 02:53:36.070475 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:53:36.070310    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5gqwj\" (UniqueName: \"kubernetes.io/projected/9fb20aea-e806-4045-8389-d574d0bd8537-kube-api-access-5gqwj\") pod \"9fb20aea-e806-4045-8389-d574d0bd8537\" (UID: \"9fb20aea-e806-4045-8389-d574d0bd8537\") "
Sep 09 02:53:36.070475 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:53:36.070336    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9fb20aea-e806-4045-8389-d574d0bd8537-model-cache\") pod \"9fb20aea-e806-4045-8389-d574d0bd8537\" (UID: \"9fb20aea-e806-4045-8389-d574d0bd8537\") "
Sep 09 02:53:36.070475 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:53:36.070448    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9fb20aea-e806-4045-8389-d574d0bd8537-dshm\") pod \"9fb20aea-e806-4045-8389-d574d0bd8537\" (UID: \"9fb20aea-e806-4045-8389-d574d0bd8537\") "
Sep 09 02:53:36.070475 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:53:36.070473    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9fb20aea-e806-4045-8389-d574d0bd8537-home\") pod \"9fb20aea-e806-4045-8389-d574d0bd8537\" (UID: \"9fb20aea-e806-4045-8389-d574d0bd8537\") "
Sep 09 02:53:36.070697 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:53:36.070506    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9fb20aea-e806-4045-8389-d574d0bd8537-tls-certs\") pod \"9fb20aea-e806-4045-8389-d574d0bd8537\" (UID: \"9fb20aea-e806-4045-8389-d574d0bd8537\") "
Sep 09 02:53:36.070697 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:53:36.070553    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fb20aea-e806-4045-8389-d574d0bd8537-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9fb20aea-e806-4045-8389-d574d0bd8537" (UID: "9fb20aea-e806-4045-8389-d574d0bd8537"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 02:53:36.070697 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:53:36.070664    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fb20aea-e806-4045-8389-d574d0bd8537-model-cache" (OuterVolumeSpecName: "model-cache") pod "9fb20aea-e806-4045-8389-d574d0bd8537" (UID: "9fb20aea-e806-4045-8389-d574d0bd8537"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 02:53:36.070853 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:53:36.070787    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fb20aea-e806-4045-8389-d574d0bd8537-home" (OuterVolumeSpecName: "home") pod "9fb20aea-e806-4045-8389-d574d0bd8537" (UID: "9fb20aea-e806-4045-8389-d574d0bd8537"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 02:53:36.070954 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:53:36.070936    2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9fb20aea-e806-4045-8389-d574d0bd8537-tmp-dir\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:53:36.071021 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:53:36.070957    2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9fb20aea-e806-4045-8389-d574d0bd8537-model-cache\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:53:36.071021 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:53:36.070965    2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9fb20aea-e806-4045-8389-d574d0bd8537-home\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:53:36.072546 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:53:36.072524    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fb20aea-e806-4045-8389-d574d0bd8537-dshm" (OuterVolumeSpecName: "dshm") pod "9fb20aea-e806-4045-8389-d574d0bd8537" (UID: "9fb20aea-e806-4045-8389-d574d0bd8537"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 02:53:36.072856 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:53:36.072834    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fb20aea-e806-4045-8389-d574d0bd8537-kube-api-access-5gqwj" (OuterVolumeSpecName: "kube-api-access-5gqwj") pod "9fb20aea-e806-4045-8389-d574d0bd8537" (UID: "9fb20aea-e806-4045-8389-d574d0bd8537"). InnerVolumeSpecName "kube-api-access-5gqwj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 02:53:36.072856 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:53:36.072835    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fb20aea-e806-4045-8389-d574d0bd8537-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9fb20aea-e806-4045-8389-d574d0bd8537" (UID: "9fb20aea-e806-4045-8389-d574d0bd8537"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 02:53:36.172091 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:53:36.172057    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gqwj\" (UniqueName: \"kubernetes.io/projected/9fb20aea-e806-4045-8389-d574d0bd8537-kube-api-access-5gqwj\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:53:36.172091 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:53:36.172084    2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9fb20aea-e806-4045-8389-d574d0bd8537-dshm\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:53:36.172286 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:53:36.172099    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9fb20aea-e806-4045-8389-d574d0bd8537-tls-certs\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:53:36.843417 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:53:36.843386    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-ad1d02af/e2e-lifecycle-model-ad1d02af-kserve-648d77ddc9-wrmhl"
Sep 09 02:53:36.843905 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:53:36.843431    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-ad1d02af/e2e-lifecycle-model-ad1d02af-kserve-648d77ddc9-wrmhl" event={"ID":"9fb20aea-e806-4045-8389-d574d0bd8537","Type":"ContainerDied","Data":"15d28fbc04b6f09ddd66fa86f46680511bb91415662ef11800f8b3ee2a044f7e"}
Sep 09 02:53:36.843905 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:53:36.843493    2570 scope.go:117] "RemoveContainer" containerID="6252e547691cfe3f5511b70645567716e9f4a9b918b06a73fdd47b47a98b53ef"
Sep 09 02:53:36.862521 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:53:36.862498    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-ad1d02af/e2e-lifecycle-model-ad1d02af-kserve-648d77ddc9-wrmhl"]
Sep 09 02:53:36.867484 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:53:36.867462    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-ad1d02af/e2e-lifecycle-model-ad1d02af-kserve-648d77ddc9-wrmhl"]
Sep 09 02:53:38.223820 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:53:38.223788    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9fb20aea-e806-4045-8389-d574d0bd8537" path="/var/lib/kubelet/pods/9fb20aea-e806-4045-8389-d574d0bd8537/volumes"
Sep 09 02:53:45.539059 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:53:45.539023    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-99qjf"]
Sep 09 02:54:01.646944 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:01.646904    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-99qjf"]
Sep 09 02:54:07.648812 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:07.648780    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-56dbc0/auth-test-model-56dbc0-kserve-fd985d4cf-2fmsj"]
Sep 09 02:54:08.733627 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:08.733588    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-c173ad/auth-test-model-c173ad-kserve-7b5d66d859-x9gp8"]
Sep 09 02:54:09.369795 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:09.369759    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-ad1d02af-openshift-default-95c789d96-6lgzq"]
Sep 09 02:54:09.370025 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:09.369999    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-ad1d02af-openshift-default-95c789d96-6lgzq" podUID="ff129a48-26ae-4893-9ca3-f4501cf6eb00" containerName="istio-proxy" containerID="cri-o://a6b13fa7fac1069e9764925897ae68fcc253846c6dc70bb8e48ed48cf0c14333" gracePeriod=30
Sep 09 02:54:09.468827 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:09.468789    2570 patch_prober.go:28] interesting pod/e2e-mt-ad1d02af-openshift-default-95c789d96-6lgzq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 09 02:54:09.469155 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:09.469110    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-ad1d02af-openshift-default-95c789d96-6lgzq" podUID="ff129a48-26ae-4893-9ca3-f4501cf6eb00" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 02:54:11.251189 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:11.251111    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-56dbc0/sub-test-model-56dbc0-kserve-65648d856c-68sn6"]
Sep 09 02:54:11.251625 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:11.251569    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fb20aea-e806-4045-8389-d574d0bd8537" containerName="main"
Sep 09 02:54:11.251625 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:11.251587    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fb20aea-e806-4045-8389-d574d0bd8537" containerName="main"
Sep 09 02:54:11.251625 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:11.251614    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="349ce7e7-a858-449f-a454-cf03dac0b87e" containerName="revoke-keys"
Sep 09 02:54:11.251625 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:11.251623    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="349ce7e7-a858-449f-a454-cf03dac0b87e" containerName="revoke-keys"
Sep 09 02:54:11.251849 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:11.251710    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="349ce7e7-a858-449f-a454-cf03dac0b87e" containerName="revoke-keys"
Sep 09 02:54:11.251849 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:11.251720    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="9fb20aea-e806-4045-8389-d574d0bd8537" containerName="main"
Sep 09 02:54:11.255011 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:11.254988    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-56dbc0/sub-test-model-56dbc0-kserve-65648d856c-68sn6"
Sep 09 02:54:11.257846 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:11.257820    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-56dbc0\"/\"sub-test-model-56dbc0-kserve-self-signed-certs\""
Sep 09 02:54:11.264284 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:11.264260    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-56dbc0/sub-test-model-56dbc0-kserve-65648d856c-68sn6"]
Sep 09 02:54:11.319427 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:11.319399    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4488ea6-6df5-4e06-be04-1fc8f8cee5bf-tmp-dir\") pod \"sub-test-model-56dbc0-kserve-65648d856c-68sn6\" (UID: \"d4488ea6-6df5-4e06-be04-1fc8f8cee5bf\") " pod="ai-tenant-e2e-shared-a-w3-56dbc0/sub-test-model-56dbc0-kserve-65648d856c-68sn6"
Sep 09 02:54:11.319584 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:11.319445    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pk9lg\" (UniqueName: \"kubernetes.io/projected/d4488ea6-6df5-4e06-be04-1fc8f8cee5bf-kube-api-access-pk9lg\") pod \"sub-test-model-56dbc0-kserve-65648d856c-68sn6\" (UID: \"d4488ea6-6df5-4e06-be04-1fc8f8cee5bf\") " pod="ai-tenant-e2e-shared-a-w3-56dbc0/sub-test-model-56dbc0-kserve-65648d856c-68sn6"
Sep 09 02:54:11.319584 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:11.319524    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4488ea6-6df5-4e06-be04-1fc8f8cee5bf-home\") pod \"sub-test-model-56dbc0-kserve-65648d856c-68sn6\" (UID: \"d4488ea6-6df5-4e06-be04-1fc8f8cee5bf\") " pod="ai-tenant-e2e-shared-a-w3-56dbc0/sub-test-model-56dbc0-kserve-65648d856c-68sn6"
Sep 09 02:54:11.319584 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:11.319549    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4488ea6-6df5-4e06-be04-1fc8f8cee5bf-model-cache\") pod \"sub-test-model-56dbc0-kserve-65648d856c-68sn6\" (UID: \"d4488ea6-6df5-4e06-be04-1fc8f8cee5bf\") " pod="ai-tenant-e2e-shared-a-w3-56dbc0/sub-test-model-56dbc0-kserve-65648d856c-68sn6"
Sep 09 02:54:11.319720 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:11.319598    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4488ea6-6df5-4e06-be04-1fc8f8cee5bf-tls-certs\") pod \"sub-test-model-56dbc0-kserve-65648d856c-68sn6\" (UID: \"d4488ea6-6df5-4e06-be04-1fc8f8cee5bf\") " pod="ai-tenant-e2e-shared-a-w3-56dbc0/sub-test-model-56dbc0-kserve-65648d856c-68sn6"
Sep 09 02:54:11.319720 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:11.319637    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4488ea6-6df5-4e06-be04-1fc8f8cee5bf-dshm\") pod \"sub-test-model-56dbc0-kserve-65648d856c-68sn6\" (UID: \"d4488ea6-6df5-4e06-be04-1fc8f8cee5bf\") " pod="ai-tenant-e2e-shared-a-w3-56dbc0/sub-test-model-56dbc0-kserve-65648d856c-68sn6"
Sep 09 02:54:11.420305 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:11.420274    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4488ea6-6df5-4e06-be04-1fc8f8cee5bf-tls-certs\") pod \"sub-test-model-56dbc0-kserve-65648d856c-68sn6\" (UID: \"d4488ea6-6df5-4e06-be04-1fc8f8cee5bf\") " pod="ai-tenant-e2e-shared-a-w3-56dbc0/sub-test-model-56dbc0-kserve-65648d856c-68sn6"
Sep 09 02:54:11.420305 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:11.420308    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4488ea6-6df5-4e06-be04-1fc8f8cee5bf-dshm\") pod \"sub-test-model-56dbc0-kserve-65648d856c-68sn6\" (UID: \"d4488ea6-6df5-4e06-be04-1fc8f8cee5bf\") " pod="ai-tenant-e2e-shared-a-w3-56dbc0/sub-test-model-56dbc0-kserve-65648d856c-68sn6"
Sep 09 02:54:11.420504 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:11.420352    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4488ea6-6df5-4e06-be04-1fc8f8cee5bf-tmp-dir\") pod \"sub-test-model-56dbc0-kserve-65648d856c-68sn6\" (UID: \"d4488ea6-6df5-4e06-be04-1fc8f8cee5bf\") " pod="ai-tenant-e2e-shared-a-w3-56dbc0/sub-test-model-56dbc0-kserve-65648d856c-68sn6"
Sep 09 02:54:11.420504 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:11.420385    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pk9lg\" (UniqueName: \"kubernetes.io/projected/d4488ea6-6df5-4e06-be04-1fc8f8cee5bf-kube-api-access-pk9lg\") pod \"sub-test-model-56dbc0-kserve-65648d856c-68sn6\" (UID: \"d4488ea6-6df5-4e06-be04-1fc8f8cee5bf\") " pod="ai-tenant-e2e-shared-a-w3-56dbc0/sub-test-model-56dbc0-kserve-65648d856c-68sn6"
Sep 09 02:54:11.420504 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:11.420450    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4488ea6-6df5-4e06-be04-1fc8f8cee5bf-home\") pod \"sub-test-model-56dbc0-kserve-65648d856c-68sn6\" (UID: \"d4488ea6-6df5-4e06-be04-1fc8f8cee5bf\") " pod="ai-tenant-e2e-shared-a-w3-56dbc0/sub-test-model-56dbc0-kserve-65648d856c-68sn6"
Sep 09 02:54:11.420504 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:11.420465    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4488ea6-6df5-4e06-be04-1fc8f8cee5bf-model-cache\") pod \"sub-test-model-56dbc0-kserve-65648d856c-68sn6\" (UID: \"d4488ea6-6df5-4e06-be04-1fc8f8cee5bf\") " pod="ai-tenant-e2e-shared-a-w3-56dbc0/sub-test-model-56dbc0-kserve-65648d856c-68sn6"
Sep 09 02:54:11.420860 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:11.420831    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4488ea6-6df5-4e06-be04-1fc8f8cee5bf-model-cache\") pod \"sub-test-model-56dbc0-kserve-65648d856c-68sn6\" (UID: \"d4488ea6-6df5-4e06-be04-1fc8f8cee5bf\") " pod="ai-tenant-e2e-shared-a-w3-56dbc0/sub-test-model-56dbc0-kserve-65648d856c-68sn6"
Sep 09 02:54:11.421016 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:11.420865    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4488ea6-6df5-4e06-be04-1fc8f8cee5bf-home\") pod \"sub-test-model-56dbc0-kserve-65648d856c-68sn6\" (UID: \"d4488ea6-6df5-4e06-be04-1fc8f8cee5bf\") " pod="ai-tenant-e2e-shared-a-w3-56dbc0/sub-test-model-56dbc0-kserve-65648d856c-68sn6"
Sep 09 02:54:11.421016 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:11.420915    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4488ea6-6df5-4e06-be04-1fc8f8cee5bf-tmp-dir\") pod \"sub-test-model-56dbc0-kserve-65648d856c-68sn6\" (UID: \"d4488ea6-6df5-4e06-be04-1fc8f8cee5bf\") " pod="ai-tenant-e2e-shared-a-w3-56dbc0/sub-test-model-56dbc0-kserve-65648d856c-68sn6"
Sep 09 02:54:11.422719 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:11.422697    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4488ea6-6df5-4e06-be04-1fc8f8cee5bf-dshm\") pod \"sub-test-model-56dbc0-kserve-65648d856c-68sn6\" (UID: \"d4488ea6-6df5-4e06-be04-1fc8f8cee5bf\") " pod="ai-tenant-e2e-shared-a-w3-56dbc0/sub-test-model-56dbc0-kserve-65648d856c-68sn6"
Sep 09 02:54:11.422850 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:11.422808    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4488ea6-6df5-4e06-be04-1fc8f8cee5bf-tls-certs\") pod \"sub-test-model-56dbc0-kserve-65648d856c-68sn6\" (UID: \"d4488ea6-6df5-4e06-be04-1fc8f8cee5bf\") " pod="ai-tenant-e2e-shared-a-w3-56dbc0/sub-test-model-56dbc0-kserve-65648d856c-68sn6"
Sep 09 02:54:11.434719 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:11.434701    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pk9lg\" (UniqueName: \"kubernetes.io/projected/d4488ea6-6df5-4e06-be04-1fc8f8cee5bf-kube-api-access-pk9lg\") pod \"sub-test-model-56dbc0-kserve-65648d856c-68sn6\" (UID: \"d4488ea6-6df5-4e06-be04-1fc8f8cee5bf\") " pod="ai-tenant-e2e-shared-a-w3-56dbc0/sub-test-model-56dbc0-kserve-65648d856c-68sn6"
Sep 09 02:54:11.565015 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:11.564898    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-56dbc0/sub-test-model-56dbc0-kserve-65648d856c-68sn6"
Sep 09 02:54:11.862358 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:11.862022    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-56dbc0/sub-test-model-56dbc0-kserve-65648d856c-68sn6"]
Sep 09 02:54:11.865202 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:54:11.865144    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd4488ea6_6df5_4e06_be04_1fc8f8cee5bf.slice/crio-a5c6bb67399ad1b8129cdcbdc5749a1037373935453645acc1b19c721ef62fe8 WatchSource:0}: Error finding container a5c6bb67399ad1b8129cdcbdc5749a1037373935453645acc1b19c721ef62fe8: Status 404 returned error can't find the container with id a5c6bb67399ad1b8129cdcbdc5749a1037373935453645acc1b19c721ef62fe8
Sep 09 02:54:12.036464 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:12.036427    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-56dbc0/sub-test-model-56dbc0-kserve-65648d856c-68sn6" event={"ID":"d4488ea6-6df5-4e06-be04-1fc8f8cee5bf","Type":"ContainerStarted","Data":"5579821312c8c6a1b1fe851e14019d82128a96545e3b98d861fb74d5bb3db6e4"}
Sep 09 02:54:12.036629 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:12.036475    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-56dbc0/sub-test-model-56dbc0-kserve-65648d856c-68sn6" event={"ID":"d4488ea6-6df5-4e06-be04-1fc8f8cee5bf","Type":"ContainerStarted","Data":"a5c6bb67399ad1b8129cdcbdc5749a1037373935453645acc1b19c721ef62fe8"}
Sep 09 02:54:12.054850 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:12.054786    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-56dbc0/sub-test-model-56dbc0-kserve-65648d856c-68sn6" podStartSLOduration=1.054763629 podStartE2EDuration="1.054763629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:54:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:54:12.053777163 +0000 UTC m=+1434.308080794" watchObservedRunningTime="2026-09-09 02:54:12.054763629 +0000 UTC m=+1434.309067268"
Sep 09 02:54:14.636816 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:14.636755    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ad1d02af-openshift-default-95c789d96-6lgzq"
Sep 09 02:54:14.645790 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:14.645767    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff129a48-26ae-4893-9ca3-f4501cf6eb00-workload-certs\") pod \"ff129a48-26ae-4893-9ca3-f4501cf6eb00\" (UID: \"ff129a48-26ae-4893-9ca3-f4501cf6eb00\") "
Sep 09 02:54:14.645886 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:14.645803    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff129a48-26ae-4893-9ca3-f4501cf6eb00-istio-podinfo\") pod \"ff129a48-26ae-4893-9ca3-f4501cf6eb00\" (UID: \"ff129a48-26ae-4893-9ca3-f4501cf6eb00\") "
Sep 09 02:54:14.645886 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:14.645869    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q545j\" (UniqueName: \"kubernetes.io/projected/ff129a48-26ae-4893-9ca3-f4501cf6eb00-kube-api-access-q545j\") pod \"ff129a48-26ae-4893-9ca3-f4501cf6eb00\" (UID: \"ff129a48-26ae-4893-9ca3-f4501cf6eb00\") "
Sep 09 02:54:14.645984 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:14.645899    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff129a48-26ae-4893-9ca3-f4501cf6eb00-istio-envoy\") pod \"ff129a48-26ae-4893-9ca3-f4501cf6eb00\" (UID: \"ff129a48-26ae-4893-9ca3-f4501cf6eb00\") "
Sep 09 02:54:14.646057 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:14.646035    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff129a48-26ae-4893-9ca3-f4501cf6eb00-workload-socket\") pod \"ff129a48-26ae-4893-9ca3-f4501cf6eb00\" (UID: \"ff129a48-26ae-4893-9ca3-f4501cf6eb00\") "
Sep 09 02:54:14.646110 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:14.646050    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff129a48-26ae-4893-9ca3-f4501cf6eb00-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ff129a48-26ae-4893-9ca3-f4501cf6eb00" (UID: "ff129a48-26ae-4893-9ca3-f4501cf6eb00"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 02:54:14.646168 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:14.646112    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff129a48-26ae-4893-9ca3-f4501cf6eb00-istiod-ca-cert\") pod \"ff129a48-26ae-4893-9ca3-f4501cf6eb00\" (UID: \"ff129a48-26ae-4893-9ca3-f4501cf6eb00\") "
Sep 09 02:54:14.646168 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:14.646144    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff129a48-26ae-4893-9ca3-f4501cf6eb00-istio-data\") pod \"ff129a48-26ae-4893-9ca3-f4501cf6eb00\" (UID: \"ff129a48-26ae-4893-9ca3-f4501cf6eb00\") "
Sep 09 02:54:14.646274 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:14.646205    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff129a48-26ae-4893-9ca3-f4501cf6eb00-credential-socket\") pod \"ff129a48-26ae-4893-9ca3-f4501cf6eb00\" (UID: \"ff129a48-26ae-4893-9ca3-f4501cf6eb00\") "
Sep 09 02:54:14.646274 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:14.646270    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff129a48-26ae-4893-9ca3-f4501cf6eb00-istio-token\") pod \"ff129a48-26ae-4893-9ca3-f4501cf6eb00\" (UID: \"ff129a48-26ae-4893-9ca3-f4501cf6eb00\") "
Sep 09 02:54:14.646393 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:14.646299    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff129a48-26ae-4893-9ca3-f4501cf6eb00-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ff129a48-26ae-4893-9ca3-f4501cf6eb00" (UID: "ff129a48-26ae-4893-9ca3-f4501cf6eb00"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 02:54:14.646533 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:14.646511    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff129a48-26ae-4893-9ca3-f4501cf6eb00-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ff129a48-26ae-4893-9ca3-f4501cf6eb00" (UID: "ff129a48-26ae-4893-9ca3-f4501cf6eb00"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 02:54:14.646618 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:14.646529    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff129a48-26ae-4893-9ca3-f4501cf6eb00-istio-data" (OuterVolumeSpecName: "istio-data") pod "ff129a48-26ae-4893-9ca3-f4501cf6eb00" (UID: "ff129a48-26ae-4893-9ca3-f4501cf6eb00"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 02:54:14.646764 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:14.646740    2570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff129a48-26ae-4893-9ca3-f4501cf6eb00-istio-data\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:54:14.646885 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:14.646768    2570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff129a48-26ae-4893-9ca3-f4501cf6eb00-credential-socket\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:54:14.646885 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:14.646783    2570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff129a48-26ae-4893-9ca3-f4501cf6eb00-workload-certs\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:54:14.646885 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:14.646797    2570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff129a48-26ae-4893-9ca3-f4501cf6eb00-workload-socket\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:54:14.647098 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:14.646953    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff129a48-26ae-4893-9ca3-f4501cf6eb00-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ff129a48-26ae-4893-9ca3-f4501cf6eb00" (UID: "ff129a48-26ae-4893-9ca3-f4501cf6eb00"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 02:54:14.648211 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:14.648184    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff129a48-26ae-4893-9ca3-f4501cf6eb00-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ff129a48-26ae-4893-9ca3-f4501cf6eb00" (UID: "ff129a48-26ae-4893-9ca3-f4501cf6eb00"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 02:54:14.648313 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:14.648260    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff129a48-26ae-4893-9ca3-f4501cf6eb00-kube-api-access-q545j" (OuterVolumeSpecName: "kube-api-access-q545j") pod "ff129a48-26ae-4893-9ca3-f4501cf6eb00" (UID: "ff129a48-26ae-4893-9ca3-f4501cf6eb00"). InnerVolumeSpecName "kube-api-access-q545j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 02:54:14.648402 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:14.648385    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ff129a48-26ae-4893-9ca3-f4501cf6eb00-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ff129a48-26ae-4893-9ca3-f4501cf6eb00" (UID: "ff129a48-26ae-4893-9ca3-f4501cf6eb00"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 02:54:14.648462 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:14.648423    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff129a48-26ae-4893-9ca3-f4501cf6eb00-istio-token" (OuterVolumeSpecName: "istio-token") pod "ff129a48-26ae-4893-9ca3-f4501cf6eb00" (UID: "ff129a48-26ae-4893-9ca3-f4501cf6eb00"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 02:54:14.747809 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:14.747781    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q545j\" (UniqueName: \"kubernetes.io/projected/ff129a48-26ae-4893-9ca3-f4501cf6eb00-kube-api-access-q545j\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:54:14.747809 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:14.747806    2570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff129a48-26ae-4893-9ca3-f4501cf6eb00-istio-envoy\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:54:14.747967 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:14.747816    2570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff129a48-26ae-4893-9ca3-f4501cf6eb00-istiod-ca-cert\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:54:14.747967 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:14.747824    2570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff129a48-26ae-4893-9ca3-f4501cf6eb00-istio-token\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:54:14.747967 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:14.747831    2570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff129a48-26ae-4893-9ca3-f4501cf6eb00-istio-podinfo\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:54:15.059239 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:15.059165    2570 generic.go:356] "Generic (PLEG): container finished" podID="ff129a48-26ae-4893-9ca3-f4501cf6eb00" containerID="a6b13fa7fac1069e9764925897ae68fcc253846c6dc70bb8e48ed48cf0c14333" exitCode=0
Sep 09 02:54:15.059239 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:15.059218    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ad1d02af-openshift-default-95c789d96-6lgzq"
Sep 09 02:54:15.059412 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:15.059257    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ad1d02af-openshift-default-95c789d96-6lgzq" event={"ID":"ff129a48-26ae-4893-9ca3-f4501cf6eb00","Type":"ContainerDied","Data":"a6b13fa7fac1069e9764925897ae68fcc253846c6dc70bb8e48ed48cf0c14333"}
Sep 09 02:54:15.059412 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:15.059347    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ad1d02af-openshift-default-95c789d96-6lgzq" event={"ID":"ff129a48-26ae-4893-9ca3-f4501cf6eb00","Type":"ContainerDied","Data":"a4a82ecb2561e8a110496bbc4e979ed7bb070a1ac905ce9bc479025e72f42167"}
Sep 09 02:54:15.059412 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:15.059375    2570 scope.go:117] "RemoveContainer" containerID="a6b13fa7fac1069e9764925897ae68fcc253846c6dc70bb8e48ed48cf0c14333"
Sep 09 02:54:15.069361 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:15.069338    2570 scope.go:117] "RemoveContainer" containerID="d48939f2ab9962c0ed88faf8169cdb57c7e73ef99ef2e9fde411a41996fda1b1"
Sep 09 02:54:15.077587 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:15.077571    2570 scope.go:117] "RemoveContainer" containerID="a6b13fa7fac1069e9764925897ae68fcc253846c6dc70bb8e48ed48cf0c14333"
Sep 09 02:54:15.077834 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:54:15.077815    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6b13fa7fac1069e9764925897ae68fcc253846c6dc70bb8e48ed48cf0c14333\": container with ID starting with a6b13fa7fac1069e9764925897ae68fcc253846c6dc70bb8e48ed48cf0c14333 not found: ID does not exist" containerID="a6b13fa7fac1069e9764925897ae68fcc253846c6dc70bb8e48ed48cf0c14333"
Sep 09 02:54:15.077873 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:15.077844    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6b13fa7fac1069e9764925897ae68fcc253846c6dc70bb8e48ed48cf0c14333"} err="failed to get container status \"a6b13fa7fac1069e9764925897ae68fcc253846c6dc70bb8e48ed48cf0c14333\": rpc error: code = NotFound desc = could not find container \"a6b13fa7fac1069e9764925897ae68fcc253846c6dc70bb8e48ed48cf0c14333\": container with ID starting with a6b13fa7fac1069e9764925897ae68fcc253846c6dc70bb8e48ed48cf0c14333 not found: ID does not exist"
Sep 09 02:54:15.077873 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:15.077859    2570 scope.go:117] "RemoveContainer" containerID="d48939f2ab9962c0ed88faf8169cdb57c7e73ef99ef2e9fde411a41996fda1b1"
Sep 09 02:54:15.078127 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:54:15.078109    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d48939f2ab9962c0ed88faf8169cdb57c7e73ef99ef2e9fde411a41996fda1b1\": container with ID starting with d48939f2ab9962c0ed88faf8169cdb57c7e73ef99ef2e9fde411a41996fda1b1 not found: ID does not exist" containerID="d48939f2ab9962c0ed88faf8169cdb57c7e73ef99ef2e9fde411a41996fda1b1"
Sep 09 02:54:15.078169 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:15.078136    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d48939f2ab9962c0ed88faf8169cdb57c7e73ef99ef2e9fde411a41996fda1b1"} err="failed to get container status \"d48939f2ab9962c0ed88faf8169cdb57c7e73ef99ef2e9fde411a41996fda1b1\": rpc error: code = NotFound desc = could not find container \"d48939f2ab9962c0ed88faf8169cdb57c7e73ef99ef2e9fde411a41996fda1b1\": container with ID starting with d48939f2ab9962c0ed88faf8169cdb57c7e73ef99ef2e9fde411a41996fda1b1 not found: ID does not exist"
Sep 09 02:54:15.083054 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:15.083033    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-ad1d02af-openshift-default-95c789d96-6lgzq"]
Sep 09 02:54:15.087260 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:15.087241    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-ad1d02af-openshift-default-95c789d96-6lgzq"]
Sep 09 02:54:16.223523 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:16.223482    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff129a48-26ae-4893-9ca3-f4501cf6eb00" path="/var/lib/kubelet/pods/ff129a48-26ae-4893-9ca3-f4501cf6eb00/volumes"
Sep 09 02:54:18.418890 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:18.418860    2570 scope.go:117] "RemoveContainer" containerID="23d548a4e31e1d14e3681e5faafad0aa3bc08ec53733a6ac5833430c669c3708"
Sep 09 02:54:18.971843 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:18.971805    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-99qjf"]
Sep 09 02:54:21.565857 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:21.565809    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-56dbc0/sub-test-model-56dbc0-kserve-65648d856c-68sn6"
Sep 09 02:54:21.566375 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:21.565959    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-56dbc0/sub-test-model-56dbc0-kserve-65648d856c-68sn6"
Sep 09 02:54:21.578592 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:21.578570    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-56dbc0/sub-test-model-56dbc0-kserve-65648d856c-68sn6"
Sep 09 02:54:22.111551 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:22.111523    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-56dbc0/sub-test-model-56dbc0-kserve-65648d856c-68sn6"
Sep 09 02:54:22.668560 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:22.668519    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-56dbc0/auth-test-model-56dbc0-kserve-fd985d4cf-2fmsj" podUID="fed344c4-8783-44fb-af19-c7e6971c5ffb" containerName="main" containerID="cri-o://0c1e1c4d67301fed5b892f26323e1527b3b4f8ce5bc139198b532496444db9bd" gracePeriod=45
Sep 09 02:54:22.912558 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:22.912536    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-56dbc0/auth-test-model-56dbc0-kserve-fd985d4cf-2fmsj"
Sep 09 02:54:23.027621 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:23.027530    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fed344c4-8783-44fb-af19-c7e6971c5ffb-model-cache\") pod \"fed344c4-8783-44fb-af19-c7e6971c5ffb\" (UID: \"fed344c4-8783-44fb-af19-c7e6971c5ffb\") "
Sep 09 02:54:23.027780 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:23.027639    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fed344c4-8783-44fb-af19-c7e6971c5ffb-dshm\") pod \"fed344c4-8783-44fb-af19-c7e6971c5ffb\" (UID: \"fed344c4-8783-44fb-af19-c7e6971c5ffb\") "
Sep 09 02:54:23.027780 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:23.027661    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fed344c4-8783-44fb-af19-c7e6971c5ffb-tls-certs\") pod \"fed344c4-8783-44fb-af19-c7e6971c5ffb\" (UID: \"fed344c4-8783-44fb-af19-c7e6971c5ffb\") "
Sep 09 02:54:23.027780 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:23.027680    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fed344c4-8783-44fb-af19-c7e6971c5ffb-tmp-dir\") pod \"fed344c4-8783-44fb-af19-c7e6971c5ffb\" (UID: \"fed344c4-8783-44fb-af19-c7e6971c5ffb\") "
Sep 09 02:54:23.027780 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:23.027707    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kgnpf\" (UniqueName: \"kubernetes.io/projected/fed344c4-8783-44fb-af19-c7e6971c5ffb-kube-api-access-kgnpf\") pod \"fed344c4-8783-44fb-af19-c7e6971c5ffb\" (UID: \"fed344c4-8783-44fb-af19-c7e6971c5ffb\") "
Sep 09 02:54:23.027780 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:23.027748    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fed344c4-8783-44fb-af19-c7e6971c5ffb-home\") pod \"fed344c4-8783-44fb-af19-c7e6971c5ffb\" (UID: \"fed344c4-8783-44fb-af19-c7e6971c5ffb\") "
Sep 09 02:54:23.028090 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:23.027809    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fed344c4-8783-44fb-af19-c7e6971c5ffb-model-cache" (OuterVolumeSpecName: "model-cache") pod "fed344c4-8783-44fb-af19-c7e6971c5ffb" (UID: "fed344c4-8783-44fb-af19-c7e6971c5ffb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 02:54:23.028090 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:23.028001    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fed344c4-8783-44fb-af19-c7e6971c5ffb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fed344c4-8783-44fb-af19-c7e6971c5ffb" (UID: "fed344c4-8783-44fb-af19-c7e6971c5ffb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 02:54:23.028090 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:23.028035    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fed344c4-8783-44fb-af19-c7e6971c5ffb-home" (OuterVolumeSpecName: "home") pod "fed344c4-8783-44fb-af19-c7e6971c5ffb" (UID: "fed344c4-8783-44fb-af19-c7e6971c5ffb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 02:54:23.028238 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:23.028193    2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fed344c4-8783-44fb-af19-c7e6971c5ffb-tmp-dir\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:54:23.028238 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:23.028215    2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fed344c4-8783-44fb-af19-c7e6971c5ffb-home\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:54:23.028238 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:23.028229    2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fed344c4-8783-44fb-af19-c7e6971c5ffb-model-cache\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:54:23.030132 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:23.030108    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fed344c4-8783-44fb-af19-c7e6971c5ffb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fed344c4-8783-44fb-af19-c7e6971c5ffb" (UID: "fed344c4-8783-44fb-af19-c7e6971c5ffb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 02:54:23.030132 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:23.030127    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fed344c4-8783-44fb-af19-c7e6971c5ffb-dshm" (OuterVolumeSpecName: "dshm") pod "fed344c4-8783-44fb-af19-c7e6971c5ffb" (UID: "fed344c4-8783-44fb-af19-c7e6971c5ffb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 02:54:23.030252 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:23.030167    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fed344c4-8783-44fb-af19-c7e6971c5ffb-kube-api-access-kgnpf" (OuterVolumeSpecName: "kube-api-access-kgnpf") pod "fed344c4-8783-44fb-af19-c7e6971c5ffb" (UID: "fed344c4-8783-44fb-af19-c7e6971c5ffb"). InnerVolumeSpecName "kube-api-access-kgnpf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 02:54:23.106382 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:23.106344    2570 generic.go:356] "Generic (PLEG): container finished" podID="fed344c4-8783-44fb-af19-c7e6971c5ffb" containerID="0c1e1c4d67301fed5b892f26323e1527b3b4f8ce5bc139198b532496444db9bd" exitCode=0
Sep 09 02:54:23.106469 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:23.106396    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-56dbc0/auth-test-model-56dbc0-kserve-fd985d4cf-2fmsj"
Sep 09 02:54:23.106469 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:23.106424    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-56dbc0/auth-test-model-56dbc0-kserve-fd985d4cf-2fmsj" event={"ID":"fed344c4-8783-44fb-af19-c7e6971c5ffb","Type":"ContainerDied","Data":"0c1e1c4d67301fed5b892f26323e1527b3b4f8ce5bc139198b532496444db9bd"}
Sep 09 02:54:23.106469 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:23.106465    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-56dbc0/auth-test-model-56dbc0-kserve-fd985d4cf-2fmsj" event={"ID":"fed344c4-8783-44fb-af19-c7e6971c5ffb","Type":"ContainerDied","Data":"484fdade4f3090deecbe20e2a62d30ab2a9d8a06b5445053e8c56b669338a46d"}
Sep 09 02:54:23.106586 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:23.106484    2570 scope.go:117] "RemoveContainer" containerID="0c1e1c4d67301fed5b892f26323e1527b3b4f8ce5bc139198b532496444db9bd"
Sep 09 02:54:23.115472 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:23.115456    2570 scope.go:117] "RemoveContainer" containerID="0c1e1c4d67301fed5b892f26323e1527b3b4f8ce5bc139198b532496444db9bd"
Sep 09 02:54:23.115735 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:54:23.115715    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c1e1c4d67301fed5b892f26323e1527b3b4f8ce5bc139198b532496444db9bd\": container with ID starting with 0c1e1c4d67301fed5b892f26323e1527b3b4f8ce5bc139198b532496444db9bd not found: ID does not exist" containerID="0c1e1c4d67301fed5b892f26323e1527b3b4f8ce5bc139198b532496444db9bd"
Sep 09 02:54:23.115792 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:23.115742    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c1e1c4d67301fed5b892f26323e1527b3b4f8ce5bc139198b532496444db9bd"} err="failed to get container status \"0c1e1c4d67301fed5b892f26323e1527b3b4f8ce5bc139198b532496444db9bd\": rpc error: code = NotFound desc = could not find container \"0c1e1c4d67301fed5b892f26323e1527b3b4f8ce5bc139198b532496444db9bd\": container with ID starting with 0c1e1c4d67301fed5b892f26323e1527b3b4f8ce5bc139198b532496444db9bd not found: ID does not exist"
Sep 09 02:54:23.128893 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:23.128875    2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fed344c4-8783-44fb-af19-c7e6971c5ffb-dshm\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:54:23.128893 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:23.128893    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fed344c4-8783-44fb-af19-c7e6971c5ffb-tls-certs\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:54:23.129047 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:23.128902    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgnpf\" (UniqueName: \"kubernetes.io/projected/fed344c4-8783-44fb-af19-c7e6971c5ffb-kube-api-access-kgnpf\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:54:23.130839 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:23.130822    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-56dbc0/auth-test-model-56dbc0-kserve-fd985d4cf-2fmsj"]
Sep 09 02:54:23.134539 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:23.134521    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-56dbc0/auth-test-model-56dbc0-kserve-fd985d4cf-2fmsj"]
Sep 09 02:54:23.753409 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:23.753369    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-c173ad/auth-test-model-c173ad-kserve-7b5d66d859-x9gp8" podUID="efb8e94e-5f79-4b8d-a2a3-23164dd1e70a" containerName="main" containerID="cri-o://c881a6818cf1f2fec8cf6a7771a44edfe40d57948a99f79ad76a96d43fa2dad9" gracePeriod=45
Sep 09 02:54:24.003680 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:24.003630    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-c173ad/auth-test-model-c173ad-kserve-7b5d66d859-x9gp8"
Sep 09 02:54:24.035377 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:24.035353    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/efb8e94e-5f79-4b8d-a2a3-23164dd1e70a-dshm\") pod \"efb8e94e-5f79-4b8d-a2a3-23164dd1e70a\" (UID: \"efb8e94e-5f79-4b8d-a2a3-23164dd1e70a\") "
Sep 09 02:54:24.035487 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:24.035397    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efb8e94e-5f79-4b8d-a2a3-23164dd1e70a-tls-certs\") pod \"efb8e94e-5f79-4b8d-a2a3-23164dd1e70a\" (UID: \"efb8e94e-5f79-4b8d-a2a3-23164dd1e70a\") "
Sep 09 02:54:24.035487 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:24.035425    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/efb8e94e-5f79-4b8d-a2a3-23164dd1e70a-model-cache\") pod \"efb8e94e-5f79-4b8d-a2a3-23164dd1e70a\" (UID: \"efb8e94e-5f79-4b8d-a2a3-23164dd1e70a\") "
Sep 09 02:54:24.035487 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:24.035474    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efb8e94e-5f79-4b8d-a2a3-23164dd1e70a-tmp-dir\") pod \"efb8e94e-5f79-4b8d-a2a3-23164dd1e70a\" (UID: \"efb8e94e-5f79-4b8d-a2a3-23164dd1e70a\") "
Sep 09 02:54:24.035650 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:24.035510    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bjwqk\" (UniqueName: \"kubernetes.io/projected/efb8e94e-5f79-4b8d-a2a3-23164dd1e70a-kube-api-access-bjwqk\") pod \"efb8e94e-5f79-4b8d-a2a3-23164dd1e70a\" (UID: \"efb8e94e-5f79-4b8d-a2a3-23164dd1e70a\") "
Sep 09 02:54:24.035650 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:24.035537    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/efb8e94e-5f79-4b8d-a2a3-23164dd1e70a-home\") pod \"efb8e94e-5f79-4b8d-a2a3-23164dd1e70a\" (UID: \"efb8e94e-5f79-4b8d-a2a3-23164dd1e70a\") "
Sep 09 02:54:24.035782 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:24.035753    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efb8e94e-5f79-4b8d-a2a3-23164dd1e70a-model-cache" (OuterVolumeSpecName: "model-cache") pod "efb8e94e-5f79-4b8d-a2a3-23164dd1e70a" (UID: "efb8e94e-5f79-4b8d-a2a3-23164dd1e70a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 02:54:24.035839 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:24.035773    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efb8e94e-5f79-4b8d-a2a3-23164dd1e70a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "efb8e94e-5f79-4b8d-a2a3-23164dd1e70a" (UID: "efb8e94e-5f79-4b8d-a2a3-23164dd1e70a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 02:54:24.035924 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:24.035893    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efb8e94e-5f79-4b8d-a2a3-23164dd1e70a-home" (OuterVolumeSpecName: "home") pod "efb8e94e-5f79-4b8d-a2a3-23164dd1e70a" (UID: "efb8e94e-5f79-4b8d-a2a3-23164dd1e70a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 02:54:24.036092 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:24.035988    2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/efb8e94e-5f79-4b8d-a2a3-23164dd1e70a-model-cache\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:54:24.036092 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:24.036009    2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efb8e94e-5f79-4b8d-a2a3-23164dd1e70a-tmp-dir\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:54:24.038261 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:24.038234    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efb8e94e-5f79-4b8d-a2a3-23164dd1e70a-kube-api-access-bjwqk" (OuterVolumeSpecName: "kube-api-access-bjwqk") pod "efb8e94e-5f79-4b8d-a2a3-23164dd1e70a" (UID: "efb8e94e-5f79-4b8d-a2a3-23164dd1e70a"). InnerVolumeSpecName "kube-api-access-bjwqk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 02:54:24.038373 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:24.038243    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efb8e94e-5f79-4b8d-a2a3-23164dd1e70a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "efb8e94e-5f79-4b8d-a2a3-23164dd1e70a" (UID: "efb8e94e-5f79-4b8d-a2a3-23164dd1e70a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 02:54:24.038373 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:24.038273    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efb8e94e-5f79-4b8d-a2a3-23164dd1e70a-dshm" (OuterVolumeSpecName: "dshm") pod "efb8e94e-5f79-4b8d-a2a3-23164dd1e70a" (UID: "efb8e94e-5f79-4b8d-a2a3-23164dd1e70a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 02:54:24.113219 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:24.113196    2570 generic.go:356] "Generic (PLEG): container finished" podID="efb8e94e-5f79-4b8d-a2a3-23164dd1e70a" containerID="c881a6818cf1f2fec8cf6a7771a44edfe40d57948a99f79ad76a96d43fa2dad9" exitCode=0
Sep 09 02:54:24.113321 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:24.113247    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-c173ad/auth-test-model-c173ad-kserve-7b5d66d859-x9gp8"
Sep 09 02:54:24.113321 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:24.113282    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-c173ad/auth-test-model-c173ad-kserve-7b5d66d859-x9gp8" event={"ID":"efb8e94e-5f79-4b8d-a2a3-23164dd1e70a","Type":"ContainerDied","Data":"c881a6818cf1f2fec8cf6a7771a44edfe40d57948a99f79ad76a96d43fa2dad9"}
Sep 09 02:54:24.113321 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:24.113319    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-c173ad/auth-test-model-c173ad-kserve-7b5d66d859-x9gp8" event={"ID":"efb8e94e-5f79-4b8d-a2a3-23164dd1e70a","Type":"ContainerDied","Data":"5be63250767a7f76e8c7c43ecab861a386c48393735b91c61adf6f932001b511"}
Sep 09 02:54:24.113434 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:24.113334    2570 scope.go:117] "RemoveContainer" containerID="c881a6818cf1f2fec8cf6a7771a44edfe40d57948a99f79ad76a96d43fa2dad9"
Sep 09 02:54:24.122591 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:24.122574    2570 scope.go:117] "RemoveContainer" containerID="c881a6818cf1f2fec8cf6a7771a44edfe40d57948a99f79ad76a96d43fa2dad9"
Sep 09 02:54:24.122842 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:54:24.122826    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c881a6818cf1f2fec8cf6a7771a44edfe40d57948a99f79ad76a96d43fa2dad9\": container with ID starting with c881a6818cf1f2fec8cf6a7771a44edfe40d57948a99f79ad76a96d43fa2dad9 not found: ID does not exist" containerID="c881a6818cf1f2fec8cf6a7771a44edfe40d57948a99f79ad76a96d43fa2dad9"
Sep 09 02:54:24.122908 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:24.122848    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c881a6818cf1f2fec8cf6a7771a44edfe40d57948a99f79ad76a96d43fa2dad9"} err="failed to get container status \"c881a6818cf1f2fec8cf6a7771a44edfe40d57948a99f79ad76a96d43fa2dad9\": rpc error: code = NotFound desc = could not find container \"c881a6818cf1f2fec8cf6a7771a44edfe40d57948a99f79ad76a96d43fa2dad9\": container with ID starting with c881a6818cf1f2fec8cf6a7771a44edfe40d57948a99f79ad76a96d43fa2dad9 not found: ID does not exist"
Sep 09 02:54:24.135707 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:24.135678    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-c173ad/auth-test-model-c173ad-kserve-7b5d66d859-x9gp8"]
Sep 09 02:54:24.136672 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:24.136652    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjwqk\" (UniqueName: \"kubernetes.io/projected/efb8e94e-5f79-4b8d-a2a3-23164dd1e70a-kube-api-access-bjwqk\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:54:24.136672 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:24.136672    2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/efb8e94e-5f79-4b8d-a2a3-23164dd1e70a-home\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:54:24.136814 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:24.136681    2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/efb8e94e-5f79-4b8d-a2a3-23164dd1e70a-dshm\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:54:24.136814 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:24.136690    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efb8e94e-5f79-4b8d-a2a3-23164dd1e70a-tls-certs\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:54:24.137642 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:24.137622    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-c173ad/auth-test-model-c173ad-kserve-7b5d66d859-x9gp8"]
Sep 09 02:54:24.229303 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:24.229273    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="efb8e94e-5f79-4b8d-a2a3-23164dd1e70a" path="/var/lib/kubelet/pods/efb8e94e-5f79-4b8d-a2a3-23164dd1e70a/volumes"
Sep 09 02:54:24.229661 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:24.229648    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fed344c4-8783-44fb-af19-c7e6971c5ffb" path="/var/lib/kubelet/pods/fed344c4-8783-44fb-af19-c7e6971c5ffb/volumes"
Sep 09 02:54:29.940997 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:29.940939    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-99qjf"]
Sep 09 02:54:36.346420 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:36.346388    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-c173ad/sub-test-model-c173ad-kserve-757777b6f-2ph56"]
Sep 09 02:54:36.346786 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:36.346744    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff129a48-26ae-4893-9ca3-f4501cf6eb00" containerName="istio-proxy"
Sep 09 02:54:36.346786 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:36.346755    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff129a48-26ae-4893-9ca3-f4501cf6eb00" containerName="istio-proxy"
Sep 09 02:54:36.346786 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:36.346765    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fed344c4-8783-44fb-af19-c7e6971c5ffb" containerName="main"
Sep 09 02:54:36.346786 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:36.346771    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="fed344c4-8783-44fb-af19-c7e6971c5ffb" containerName="main"
Sep 09 02:54:36.346786 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:36.346780    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efb8e94e-5f79-4b8d-a2a3-23164dd1e70a" containerName="main"
Sep 09 02:54:36.346786 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:36.346786    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="efb8e94e-5f79-4b8d-a2a3-23164dd1e70a" containerName="main"
Sep 09 02:54:36.347009 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:36.346848    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff129a48-26ae-4893-9ca3-f4501cf6eb00" containerName="istio-proxy"
Sep 09 02:54:36.347009 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:36.346857    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff129a48-26ae-4893-9ca3-f4501cf6eb00" containerName="istio-proxy"
Sep 09 02:54:36.347009 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:36.346866    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="fed344c4-8783-44fb-af19-c7e6971c5ffb" containerName="main"
Sep 09 02:54:36.347009 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:36.346873    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="efb8e94e-5f79-4b8d-a2a3-23164dd1e70a" containerName="main"
Sep 09 02:54:36.353056 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:36.352704    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-c173ad/sub-test-model-c173ad-kserve-757777b6f-2ph56"
Sep 09 02:54:36.356057 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:36.356030    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-c173ad\"/\"default-dockercfg-99zkq\""
Sep 09 02:54:36.357599 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:36.357459    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-c173ad\"/\"kube-root-ca.crt\""
Sep 09 02:54:36.357599 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:36.357471    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-c173ad\"/\"openshift-service-ca.crt\""
Sep 09 02:54:36.357771 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:36.357696    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-c173ad\"/\"sub-test-model-c173ad-kserve-self-signed-certs\""
Sep 09 02:54:36.360329 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:36.360309    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-c173ad/sub-test-model-c173ad-kserve-757777b6f-2ph56"]
Sep 09 02:54:36.457096 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:36.457057    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2a4c6b8-11bf-42f4-81dd-5bd0eae58783-tmp-dir\") pod \"sub-test-model-c173ad-kserve-757777b6f-2ph56\" (UID: \"a2a4c6b8-11bf-42f4-81dd-5bd0eae58783\") " pod="ai-tenant-e2e-shared-b-w3-c173ad/sub-test-model-c173ad-kserve-757777b6f-2ph56"
Sep 09 02:54:36.457280 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:36.457103    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2a4c6b8-11bf-42f4-81dd-5bd0eae58783-tls-certs\") pod \"sub-test-model-c173ad-kserve-757777b6f-2ph56\" (UID: \"a2a4c6b8-11bf-42f4-81dd-5bd0eae58783\") " pod="ai-tenant-e2e-shared-b-w3-c173ad/sub-test-model-c173ad-kserve-757777b6f-2ph56"
Sep 09 02:54:36.457280 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:36.457213    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2a4c6b8-11bf-42f4-81dd-5bd0eae58783-home\") pod \"sub-test-model-c173ad-kserve-757777b6f-2ph56\" (UID: \"a2a4c6b8-11bf-42f4-81dd-5bd0eae58783\") " pod="ai-tenant-e2e-shared-b-w3-c173ad/sub-test-model-c173ad-kserve-757777b6f-2ph56"
Sep 09 02:54:36.457280 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:36.457254    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2a4c6b8-11bf-42f4-81dd-5bd0eae58783-model-cache\") pod \"sub-test-model-c173ad-kserve-757777b6f-2ph56\" (UID: \"a2a4c6b8-11bf-42f4-81dd-5bd0eae58783\") " pod="ai-tenant-e2e-shared-b-w3-c173ad/sub-test-model-c173ad-kserve-757777b6f-2ph56"
Sep 09 02:54:36.457457 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:36.457292    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvsqf\" (UniqueName: \"kubernetes.io/projected/a2a4c6b8-11bf-42f4-81dd-5bd0eae58783-kube-api-access-kvsqf\") pod \"sub-test-model-c173ad-kserve-757777b6f-2ph56\" (UID: \"a2a4c6b8-11bf-42f4-81dd-5bd0eae58783\") " pod="ai-tenant-e2e-shared-b-w3-c173ad/sub-test-model-c173ad-kserve-757777b6f-2ph56"
Sep 09 02:54:36.457457 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:36.457352    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2a4c6b8-11bf-42f4-81dd-5bd0eae58783-dshm\") pod \"sub-test-model-c173ad-kserve-757777b6f-2ph56\" (UID: \"a2a4c6b8-11bf-42f4-81dd-5bd0eae58783\") " pod="ai-tenant-e2e-shared-b-w3-c173ad/sub-test-model-c173ad-kserve-757777b6f-2ph56"
Sep 09 02:54:36.558489 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:36.558456    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2a4c6b8-11bf-42f4-81dd-5bd0eae58783-tmp-dir\") pod \"sub-test-model-c173ad-kserve-757777b6f-2ph56\" (UID: \"a2a4c6b8-11bf-42f4-81dd-5bd0eae58783\") " pod="ai-tenant-e2e-shared-b-w3-c173ad/sub-test-model-c173ad-kserve-757777b6f-2ph56"
Sep 09 02:54:36.558489 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:36.558490    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2a4c6b8-11bf-42f4-81dd-5bd0eae58783-tls-certs\") pod \"sub-test-model-c173ad-kserve-757777b6f-2ph56\" (UID: \"a2a4c6b8-11bf-42f4-81dd-5bd0eae58783\") " pod="ai-tenant-e2e-shared-b-w3-c173ad/sub-test-model-c173ad-kserve-757777b6f-2ph56"
Sep 09 02:54:36.558703 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:36.558512    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2a4c6b8-11bf-42f4-81dd-5bd0eae58783-home\") pod \"sub-test-model-c173ad-kserve-757777b6f-2ph56\" (UID: \"a2a4c6b8-11bf-42f4-81dd-5bd0eae58783\") " pod="ai-tenant-e2e-shared-b-w3-c173ad/sub-test-model-c173ad-kserve-757777b6f-2ph56"
Sep 09 02:54:36.558703 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:36.558532    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2a4c6b8-11bf-42f4-81dd-5bd0eae58783-model-cache\") pod \"sub-test-model-c173ad-kserve-757777b6f-2ph56\" (UID: \"a2a4c6b8-11bf-42f4-81dd-5bd0eae58783\") " pod="ai-tenant-e2e-shared-b-w3-c173ad/sub-test-model-c173ad-kserve-757777b6f-2ph56"
Sep 09 02:54:36.558703 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:36.558554    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvsqf\" (UniqueName: \"kubernetes.io/projected/a2a4c6b8-11bf-42f4-81dd-5bd0eae58783-kube-api-access-kvsqf\") pod \"sub-test-model-c173ad-kserve-757777b6f-2ph56\" (UID: \"a2a4c6b8-11bf-42f4-81dd-5bd0eae58783\") " pod="ai-tenant-e2e-shared-b-w3-c173ad/sub-test-model-c173ad-kserve-757777b6f-2ph56"
Sep 09 02:54:36.558703 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:36.558574    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2a4c6b8-11bf-42f4-81dd-5bd0eae58783-dshm\") pod \"sub-test-model-c173ad-kserve-757777b6f-2ph56\" (UID: \"a2a4c6b8-11bf-42f4-81dd-5bd0eae58783\") " pod="ai-tenant-e2e-shared-b-w3-c173ad/sub-test-model-c173ad-kserve-757777b6f-2ph56"
Sep 09 02:54:36.558921 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:36.558893    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2a4c6b8-11bf-42f4-81dd-5bd0eae58783-home\") pod \"sub-test-model-c173ad-kserve-757777b6f-2ph56\" (UID: \"a2a4c6b8-11bf-42f4-81dd-5bd0eae58783\") " pod="ai-tenant-e2e-shared-b-w3-c173ad/sub-test-model-c173ad-kserve-757777b6f-2ph56"
Sep 09 02:54:36.558962 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:36.558931    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2a4c6b8-11bf-42f4-81dd-5bd0eae58783-model-cache\") pod \"sub-test-model-c173ad-kserve-757777b6f-2ph56\" (UID: \"a2a4c6b8-11bf-42f4-81dd-5bd0eae58783\") " pod="ai-tenant-e2e-shared-b-w3-c173ad/sub-test-model-c173ad-kserve-757777b6f-2ph56"
Sep 09 02:54:36.559030 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:36.559020    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2a4c6b8-11bf-42f4-81dd-5bd0eae58783-tmp-dir\") pod \"sub-test-model-c173ad-kserve-757777b6f-2ph56\" (UID: \"a2a4c6b8-11bf-42f4-81dd-5bd0eae58783\") " pod="ai-tenant-e2e-shared-b-w3-c173ad/sub-test-model-c173ad-kserve-757777b6f-2ph56"
Sep 09 02:54:36.560782 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:36.560760    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2a4c6b8-11bf-42f4-81dd-5bd0eae58783-dshm\") pod \"sub-test-model-c173ad-kserve-757777b6f-2ph56\" (UID: \"a2a4c6b8-11bf-42f4-81dd-5bd0eae58783\") " pod="ai-tenant-e2e-shared-b-w3-c173ad/sub-test-model-c173ad-kserve-757777b6f-2ph56"
Sep 09 02:54:36.561085 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:36.561067    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2a4c6b8-11bf-42f4-81dd-5bd0eae58783-tls-certs\") pod \"sub-test-model-c173ad-kserve-757777b6f-2ph56\" (UID: \"a2a4c6b8-11bf-42f4-81dd-5bd0eae58783\") " pod="ai-tenant-e2e-shared-b-w3-c173ad/sub-test-model-c173ad-kserve-757777b6f-2ph56"
Sep 09 02:54:36.566715 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:36.566692    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvsqf\" (UniqueName: \"kubernetes.io/projected/a2a4c6b8-11bf-42f4-81dd-5bd0eae58783-kube-api-access-kvsqf\") pod \"sub-test-model-c173ad-kserve-757777b6f-2ph56\" (UID: \"a2a4c6b8-11bf-42f4-81dd-5bd0eae58783\") " pod="ai-tenant-e2e-shared-b-w3-c173ad/sub-test-model-c173ad-kserve-757777b6f-2ph56"
Sep 09 02:54:36.665139 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:36.665084    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-c173ad/sub-test-model-c173ad-kserve-757777b6f-2ph56"
Sep 09 02:54:36.934624 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:36.934600    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-c173ad/sub-test-model-c173ad-kserve-757777b6f-2ph56"]
Sep 09 02:54:36.936375 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:54:36.936345    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda2a4c6b8_11bf_42f4_81dd_5bd0eae58783.slice/crio-44f92d2c2b21e261301736bf3871b90117b6cebcc0f2c1a5ee9dd4b34260da19 WatchSource:0}: Error finding container 44f92d2c2b21e261301736bf3871b90117b6cebcc0f2c1a5ee9dd4b34260da19: Status 404 returned error can't find the container with id 44f92d2c2b21e261301736bf3871b90117b6cebcc0f2c1a5ee9dd4b34260da19
Sep 09 02:54:37.180855 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:37.180816    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-c173ad/sub-test-model-c173ad-kserve-757777b6f-2ph56" event={"ID":"a2a4c6b8-11bf-42f4-81dd-5bd0eae58783","Type":"ContainerStarted","Data":"bc41adefd45d1b7475e3b3555b90d8fab63cee9a1f6e0adf647733b8ca7e0b6a"}
Sep 09 02:54:37.181019 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:37.180866    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-c173ad/sub-test-model-c173ad-kserve-757777b6f-2ph56" event={"ID":"a2a4c6b8-11bf-42f4-81dd-5bd0eae58783","Type":"ContainerStarted","Data":"44f92d2c2b21e261301736bf3871b90117b6cebcc0f2c1a5ee9dd4b34260da19"}
Sep 09 02:54:37.199114 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:37.199019    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-c173ad/sub-test-model-c173ad-kserve-757777b6f-2ph56" podStartSLOduration=1.199005016 podStartE2EDuration="1.199005016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:54:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:54:37.1975833 +0000 UTC m=+1459.451886936" watchObservedRunningTime="2026-09-09 02:54:37.199005016 +0000 UTC m=+1459.453308672"
Sep 09 02:54:43.154708 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:43.154664    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-99qjf"]
Sep 09 02:54:46.665803 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:46.665763    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-c173ad/sub-test-model-c173ad-kserve-757777b6f-2ph56"
Sep 09 02:54:46.666223 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:46.665839    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-c173ad/sub-test-model-c173ad-kserve-757777b6f-2ph56"
Sep 09 02:54:46.678254 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:46.678227    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-c173ad/sub-test-model-c173ad-kserve-757777b6f-2ph56"
Sep 09 02:54:47.244551 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:47.244523    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-c173ad/sub-test-model-c173ad-kserve-757777b6f-2ph56"
Sep 09 02:54:58.646750 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:54:58.646714    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-99qjf"]
Sep 09 02:55:03.763494 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:03.763461    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-2b9f0d-openshift-default-94d7f8778-n6qvg"]
Sep 09 02:55:03.763871 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:03.763849    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff129a48-26ae-4893-9ca3-f4501cf6eb00" containerName="istio-proxy"
Sep 09 02:55:03.763871 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:03.763862    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff129a48-26ae-4893-9ca3-f4501cf6eb00" containerName="istio-proxy"
Sep 09 02:55:03.767100 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:03.767084    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2b9f0d-openshift-default-94d7f8778-n6qvg"
Sep 09 02:55:03.769953 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:03.769933    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-2b9f0d-openshift-default-dockercfg-2lzs7\""
Sep 09 02:55:03.779467 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:03.779440    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-2b9f0d-openshift-default-94d7f8778-n6qvg"]
Sep 09 02:55:03.818222 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:03.818188    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j74dd\" (UniqueName: \"kubernetes.io/projected/44eb27d1-e8e8-4689-a39f-42804def79ee-kube-api-access-j74dd\") pod \"e2e-api-a-2b9f0d-openshift-default-94d7f8778-n6qvg\" (UID: \"44eb27d1-e8e8-4689-a39f-42804def79ee\") " pod="openshift-ingress/e2e-api-a-2b9f0d-openshift-default-94d7f8778-n6qvg"
Sep 09 02:55:03.818393 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:03.818235    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44eb27d1-e8e8-4689-a39f-42804def79ee-workload-socket\") pod \"e2e-api-a-2b9f0d-openshift-default-94d7f8778-n6qvg\" (UID: \"44eb27d1-e8e8-4689-a39f-42804def79ee\") " pod="openshift-ingress/e2e-api-a-2b9f0d-openshift-default-94d7f8778-n6qvg"
Sep 09 02:55:03.818393 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:03.818262    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44eb27d1-e8e8-4689-a39f-42804def79ee-istio-token\") pod \"e2e-api-a-2b9f0d-openshift-default-94d7f8778-n6qvg\" (UID: \"44eb27d1-e8e8-4689-a39f-42804def79ee\") " pod="openshift-ingress/e2e-api-a-2b9f0d-openshift-default-94d7f8778-n6qvg"
Sep 09 02:55:03.818393 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:03.818344    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44eb27d1-e8e8-4689-a39f-42804def79ee-credential-socket\") pod \"e2e-api-a-2b9f0d-openshift-default-94d7f8778-n6qvg\" (UID: \"44eb27d1-e8e8-4689-a39f-42804def79ee\") " pod="openshift-ingress/e2e-api-a-2b9f0d-openshift-default-94d7f8778-n6qvg"
Sep 09 02:55:03.818393 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:03.818391    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44eb27d1-e8e8-4689-a39f-42804def79ee-istio-envoy\") pod \"e2e-api-a-2b9f0d-openshift-default-94d7f8778-n6qvg\" (UID: \"44eb27d1-e8e8-4689-a39f-42804def79ee\") " pod="openshift-ingress/e2e-api-a-2b9f0d-openshift-default-94d7f8778-n6qvg"
Sep 09 02:55:03.818624 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:03.818487    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44eb27d1-e8e8-4689-a39f-42804def79ee-workload-certs\") pod \"e2e-api-a-2b9f0d-openshift-default-94d7f8778-n6qvg\" (UID: \"44eb27d1-e8e8-4689-a39f-42804def79ee\") " pod="openshift-ingress/e2e-api-a-2b9f0d-openshift-default-94d7f8778-n6qvg"
Sep 09 02:55:03.818624 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:03.818527    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44eb27d1-e8e8-4689-a39f-42804def79ee-istiod-ca-cert\") pod \"e2e-api-a-2b9f0d-openshift-default-94d7f8778-n6qvg\" (UID: \"44eb27d1-e8e8-4689-a39f-42804def79ee\") " pod="openshift-ingress/e2e-api-a-2b9f0d-openshift-default-94d7f8778-n6qvg"
Sep 09 02:55:03.818624 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:03.818559    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44eb27d1-e8e8-4689-a39f-42804def79ee-istio-podinfo\") pod \"e2e-api-a-2b9f0d-openshift-default-94d7f8778-n6qvg\" (UID: \"44eb27d1-e8e8-4689-a39f-42804def79ee\") " pod="openshift-ingress/e2e-api-a-2b9f0d-openshift-default-94d7f8778-n6qvg"
Sep 09 02:55:03.818624 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:03.818604    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44eb27d1-e8e8-4689-a39f-42804def79ee-istio-data\") pod \"e2e-api-a-2b9f0d-openshift-default-94d7f8778-n6qvg\" (UID: \"44eb27d1-e8e8-4689-a39f-42804def79ee\") " pod="openshift-ingress/e2e-api-a-2b9f0d-openshift-default-94d7f8778-n6qvg"
Sep 09 02:55:03.919989 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:03.919945    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44eb27d1-e8e8-4689-a39f-42804def79ee-workload-certs\") pod \"e2e-api-a-2b9f0d-openshift-default-94d7f8778-n6qvg\" (UID: \"44eb27d1-e8e8-4689-a39f-42804def79ee\") " pod="openshift-ingress/e2e-api-a-2b9f0d-openshift-default-94d7f8778-n6qvg"
Sep 09 02:55:03.920184 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:03.920016    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44eb27d1-e8e8-4689-a39f-42804def79ee-istiod-ca-cert\") pod \"e2e-api-a-2b9f0d-openshift-default-94d7f8778-n6qvg\" (UID: \"44eb27d1-e8e8-4689-a39f-42804def79ee\") " pod="openshift-ingress/e2e-api-a-2b9f0d-openshift-default-94d7f8778-n6qvg"
Sep 09 02:55:03.920184 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:03.920039    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44eb27d1-e8e8-4689-a39f-42804def79ee-istio-podinfo\") pod \"e2e-api-a-2b9f0d-openshift-default-94d7f8778-n6qvg\" (UID: \"44eb27d1-e8e8-4689-a39f-42804def79ee\") " pod="openshift-ingress/e2e-api-a-2b9f0d-openshift-default-94d7f8778-n6qvg"
Sep 09 02:55:03.920184 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:03.920067    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44eb27d1-e8e8-4689-a39f-42804def79ee-istio-data\") pod \"e2e-api-a-2b9f0d-openshift-default-94d7f8778-n6qvg\" (UID: \"44eb27d1-e8e8-4689-a39f-42804def79ee\") " pod="openshift-ingress/e2e-api-a-2b9f0d-openshift-default-94d7f8778-n6qvg"
Sep 09 02:55:03.920184 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:03.920139    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j74dd\" (UniqueName: \"kubernetes.io/projected/44eb27d1-e8e8-4689-a39f-42804def79ee-kube-api-access-j74dd\") pod \"e2e-api-a-2b9f0d-openshift-default-94d7f8778-n6qvg\" (UID: \"44eb27d1-e8e8-4689-a39f-42804def79ee\") " pod="openshift-ingress/e2e-api-a-2b9f0d-openshift-default-94d7f8778-n6qvg"
Sep 09 02:55:03.920480 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:03.920208    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44eb27d1-e8e8-4689-a39f-42804def79ee-workload-socket\") pod \"e2e-api-a-2b9f0d-openshift-default-94d7f8778-n6qvg\" (UID: \"44eb27d1-e8e8-4689-a39f-42804def79ee\") " pod="openshift-ingress/e2e-api-a-2b9f0d-openshift-default-94d7f8778-n6qvg"
Sep 09 02:55:03.920480 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:03.920246    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44eb27d1-e8e8-4689-a39f-42804def79ee-istio-token\") pod \"e2e-api-a-2b9f0d-openshift-default-94d7f8778-n6qvg\" (UID: \"44eb27d1-e8e8-4689-a39f-42804def79ee\") " pod="openshift-ingress/e2e-api-a-2b9f0d-openshift-default-94d7f8778-n6qvg"
Sep 09 02:55:03.920480 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:03.920412    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44eb27d1-e8e8-4689-a39f-42804def79ee-workload-certs\") pod \"e2e-api-a-2b9f0d-openshift-default-94d7f8778-n6qvg\" (UID: \"44eb27d1-e8e8-4689-a39f-42804def79ee\") " pod="openshift-ingress/e2e-api-a-2b9f0d-openshift-default-94d7f8778-n6qvg"
Sep 09 02:55:03.920480 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:03.920439    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44eb27d1-e8e8-4689-a39f-42804def79ee-credential-socket\") pod \"e2e-api-a-2b9f0d-openshift-default-94d7f8778-n6qvg\" (UID: \"44eb27d1-e8e8-4689-a39f-42804def79ee\") " pod="openshift-ingress/e2e-api-a-2b9f0d-openshift-default-94d7f8778-n6qvg"
Sep 09 02:55:03.920694 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:03.920487    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44eb27d1-e8e8-4689-a39f-42804def79ee-istio-envoy\") pod \"e2e-api-a-2b9f0d-openshift-default-94d7f8778-n6qvg\" (UID: \"44eb27d1-e8e8-4689-a39f-42804def79ee\") " pod="openshift-ingress/e2e-api-a-2b9f0d-openshift-default-94d7f8778-n6qvg"
Sep 09 02:55:03.920694 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:03.920583    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44eb27d1-e8e8-4689-a39f-42804def79ee-istio-data\") pod \"e2e-api-a-2b9f0d-openshift-default-94d7f8778-n6qvg\" (UID: \"44eb27d1-e8e8-4689-a39f-42804def79ee\") " pod="openshift-ingress/e2e-api-a-2b9f0d-openshift-default-94d7f8778-n6qvg"
Sep 09 02:55:03.920694 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:03.920637    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44eb27d1-e8e8-4689-a39f-42804def79ee-workload-socket\") pod \"e2e-api-a-2b9f0d-openshift-default-94d7f8778-n6qvg\" (UID: \"44eb27d1-e8e8-4689-a39f-42804def79ee\") " pod="openshift-ingress/e2e-api-a-2b9f0d-openshift-default-94d7f8778-n6qvg"
Sep 09 02:55:03.920859 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:03.920828    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44eb27d1-e8e8-4689-a39f-42804def79ee-credential-socket\") pod \"e2e-api-a-2b9f0d-openshift-default-94d7f8778-n6qvg\" (UID: \"44eb27d1-e8e8-4689-a39f-42804def79ee\") " pod="openshift-ingress/e2e-api-a-2b9f0d-openshift-default-94d7f8778-n6qvg"
Sep 09 02:55:03.920859 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:03.920835    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44eb27d1-e8e8-4689-a39f-42804def79ee-istiod-ca-cert\") pod \"e2e-api-a-2b9f0d-openshift-default-94d7f8778-n6qvg\" (UID: \"44eb27d1-e8e8-4689-a39f-42804def79ee\") " pod="openshift-ingress/e2e-api-a-2b9f0d-openshift-default-94d7f8778-n6qvg"
Sep 09 02:55:03.922554 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:03.922531    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44eb27d1-e8e8-4689-a39f-42804def79ee-istio-podinfo\") pod \"e2e-api-a-2b9f0d-openshift-default-94d7f8778-n6qvg\" (UID: \"44eb27d1-e8e8-4689-a39f-42804def79ee\") " pod="openshift-ingress/e2e-api-a-2b9f0d-openshift-default-94d7f8778-n6qvg"
Sep 09 02:55:03.922662 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:03.922534    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44eb27d1-e8e8-4689-a39f-42804def79ee-istio-envoy\") pod \"e2e-api-a-2b9f0d-openshift-default-94d7f8778-n6qvg\" (UID: \"44eb27d1-e8e8-4689-a39f-42804def79ee\") " pod="openshift-ingress/e2e-api-a-2b9f0d-openshift-default-94d7f8778-n6qvg"
Sep 09 02:55:03.928084 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:03.928060    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44eb27d1-e8e8-4689-a39f-42804def79ee-istio-token\") pod \"e2e-api-a-2b9f0d-openshift-default-94d7f8778-n6qvg\" (UID: \"44eb27d1-e8e8-4689-a39f-42804def79ee\") " pod="openshift-ingress/e2e-api-a-2b9f0d-openshift-default-94d7f8778-n6qvg"
Sep 09 02:55:03.928243 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:03.928225    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j74dd\" (UniqueName: \"kubernetes.io/projected/44eb27d1-e8e8-4689-a39f-42804def79ee-kube-api-access-j74dd\") pod \"e2e-api-a-2b9f0d-openshift-default-94d7f8778-n6qvg\" (UID: \"44eb27d1-e8e8-4689-a39f-42804def79ee\") " pod="openshift-ingress/e2e-api-a-2b9f0d-openshift-default-94d7f8778-n6qvg"
Sep 09 02:55:04.078662 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:04.078566    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2b9f0d-openshift-default-94d7f8778-n6qvg"
Sep 09 02:55:04.356096 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:04.356065    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-2b9f0d-openshift-default-94d7f8778-n6qvg"]
Sep 09 02:55:04.357822 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:55:04.357787    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44eb27d1_e8e8_4689_a39f_42804def79ee.slice/crio-30c4623682ee7e8c9312612e68cd79f6867dc3a1e11043b7ef955a8d9317fc10 WatchSource:0}: Error finding container 30c4623682ee7e8c9312612e68cd79f6867dc3a1e11043b7ef955a8d9317fc10: Status 404 returned error can't find the container with id 30c4623682ee7e8c9312612e68cd79f6867dc3a1e11043b7ef955a8d9317fc10
Sep 09 02:55:04.360111 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:04.360077    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 02:55:04.360194 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:04.360143    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 02:55:04.360194 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:04.360169    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 02:55:05.324778 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:05.324741    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2b9f0d-openshift-default-94d7f8778-n6qvg" event={"ID":"44eb27d1-e8e8-4689-a39f-42804def79ee","Type":"ContainerStarted","Data":"33f5a40e690562f15ff8d4a91f02c7efef70400907b210f076fa68ceb893bdeb"}
Sep 09 02:55:05.325215 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:05.324806    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2b9f0d-openshift-default-94d7f8778-n6qvg" event={"ID":"44eb27d1-e8e8-4689-a39f-42804def79ee","Type":"ContainerStarted","Data":"30c4623682ee7e8c9312612e68cd79f6867dc3a1e11043b7ef955a8d9317fc10"}
Sep 09 02:55:05.346163 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:05.346106    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-2b9f0d-openshift-default-94d7f8778-n6qvg" podStartSLOduration=2.346088238 podStartE2EDuration="2.346088238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:55:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:55:05.345019152 +0000 UTC m=+1487.599322802" watchObservedRunningTime="2026-09-09 02:55:05.346088238 +0000 UTC m=+1487.600391873"
Sep 09 02:55:06.079470 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:06.079430    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-2b9f0d-openshift-default-94d7f8778-n6qvg"
Sep 09 02:55:06.085350 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:06.085324    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-2b9f0d-openshift-default-94d7f8778-n6qvg"
Sep 09 02:55:06.330187 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:06.330106    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-2b9f0d-openshift-default-94d7f8778-n6qvg"
Sep 09 02:55:06.331044 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:06.331026    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-2b9f0d-openshift-default-94d7f8778-n6qvg"
Sep 09 02:55:15.652991 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:15.652933    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-99qjf"]
Sep 09 02:55:26.433829 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:26.433793    2570 generic.go:356] "Generic (PLEG): container finished" podID="44eb27d1-e8e8-4689-a39f-42804def79ee" containerID="33f5a40e690562f15ff8d4a91f02c7efef70400907b210f076fa68ceb893bdeb" exitCode=0
Sep 09 02:55:26.434287 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:26.433847    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2b9f0d-openshift-default-94d7f8778-n6qvg" event={"ID":"44eb27d1-e8e8-4689-a39f-42804def79ee","Type":"ContainerDied","Data":"33f5a40e690562f15ff8d4a91f02c7efef70400907b210f076fa68ceb893bdeb"}
Sep 09 02:55:26.434287 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:26.434242    2570 scope.go:117] "RemoveContainer" containerID="33f5a40e690562f15ff8d4a91f02c7efef70400907b210f076fa68ceb893bdeb"
Sep 09 02:55:26.435418 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:26.435383    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 02:55:26.435533 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:26.435454    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 02:55:26.435533 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:26.435493    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 02:55:26.500094 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:26.500062    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-56dbc0/sub-test-model-56dbc0-kserve-65648d856c-68sn6"]
Sep 09 02:55:27.442616 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:27.442572    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2b9f0d-openshift-default-94d7f8778-n6qvg" event={"ID":"44eb27d1-e8e8-4689-a39f-42804def79ee","Type":"ContainerStarted","Data":"75dd315d6440bfeebe896a9c2269ab7311284dacf8edd087e52e97eb348a0511"}
Sep 09 02:55:27.523532 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:27.523500    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-c173ad/sub-test-model-c173ad-kserve-757777b6f-2ph56"]
Sep 09 02:55:28.078988 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:28.078933    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-2b9f0d-openshift-default-94d7f8778-n6qvg"
Sep 09 02:55:28.079240 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:28.079213    2570 patch_prober.go:28] interesting pod/e2e-api-a-2b9f0d-openshift-default-94d7f8778-n6qvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Sep 09 02:55:28.079331 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:28.079275    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2b9f0d-openshift-default-94d7f8778-n6qvg" podUID="44eb27d1-e8e8-4689-a39f-42804def79ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Sep 09 02:55:29.079065 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:29.079033    2570 patch_prober.go:28] interesting pod/e2e-api-a-2b9f0d-openshift-default-94d7f8778-n6qvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Sep 09 02:55:29.079461 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:29.079089    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2b9f0d-openshift-default-94d7f8778-n6qvg" podUID="44eb27d1-e8e8-4689-a39f-42804def79ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Sep 09 02:55:30.079114 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:30.079073    2570 patch_prober.go:28] interesting pod/e2e-api-a-2b9f0d-openshift-default-94d7f8778-n6qvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Sep 09 02:55:30.079508 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:30.079140    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2b9f0d-openshift-default-94d7f8778-n6qvg" podUID="44eb27d1-e8e8-4689-a39f-42804def79ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Sep 09 02:55:30.146167 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:30.146125    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-99qjf"]
Sep 09 02:55:31.079027 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:31.078961    2570 patch_prober.go:28] interesting pod/e2e-api-a-2b9f0d-openshift-default-94d7f8778-n6qvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Sep 09 02:55:31.079217 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:31.079061    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2b9f0d-openshift-default-94d7f8778-n6qvg" podUID="44eb27d1-e8e8-4689-a39f-42804def79ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Sep 09 02:55:32.079427 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:32.079388    2570 patch_prober.go:28] interesting pod/e2e-api-a-2b9f0d-openshift-default-94d7f8778-n6qvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Sep 09 02:55:32.079839 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:32.079444    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2b9f0d-openshift-default-94d7f8778-n6qvg" podUID="44eb27d1-e8e8-4689-a39f-42804def79ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Sep 09 02:55:32.744823 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:32.744780    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-56dbc0/rate-test-model-56dbc0-kserve-69658dcd7-hbgm6"]
Sep 09 02:55:32.749209 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:32.749176    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-56dbc0/rate-test-model-56dbc0-kserve-69658dcd7-hbgm6"
Sep 09 02:55:32.751951 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:32.751924    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-56dbc0\"/\"rate-test-model-56dbc0-kserve-self-signed-certs\""
Sep 09 02:55:32.758455 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:32.758426    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-56dbc0/rate-test-model-56dbc0-kserve-69658dcd7-hbgm6"]
Sep 09 02:55:32.816427 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:32.816390    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a1cd62f-91d1-4c56-8a2f-4279d7f43a18-tls-certs\") pod \"rate-test-model-56dbc0-kserve-69658dcd7-hbgm6\" (UID: \"8a1cd62f-91d1-4c56-8a2f-4279d7f43a18\") " pod="ai-tenant-e2e-shared-a-w3-56dbc0/rate-test-model-56dbc0-kserve-69658dcd7-hbgm6"
Sep 09 02:55:32.816608 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:32.816430    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a1cd62f-91d1-4c56-8a2f-4279d7f43a18-home\") pod \"rate-test-model-56dbc0-kserve-69658dcd7-hbgm6\" (UID: \"8a1cd62f-91d1-4c56-8a2f-4279d7f43a18\") " pod="ai-tenant-e2e-shared-a-w3-56dbc0/rate-test-model-56dbc0-kserve-69658dcd7-hbgm6"
Sep 09 02:55:32.816608 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:32.816489    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a1cd62f-91d1-4c56-8a2f-4279d7f43a18-model-cache\") pod \"rate-test-model-56dbc0-kserve-69658dcd7-hbgm6\" (UID: \"8a1cd62f-91d1-4c56-8a2f-4279d7f43a18\") " pod="ai-tenant-e2e-shared-a-w3-56dbc0/rate-test-model-56dbc0-kserve-69658dcd7-hbgm6"
Sep 09 02:55:32.816608 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:32.816519    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a1cd62f-91d1-4c56-8a2f-4279d7f43a18-dshm\") pod \"rate-test-model-56dbc0-kserve-69658dcd7-hbgm6\" (UID: \"8a1cd62f-91d1-4c56-8a2f-4279d7f43a18\") " pod="ai-tenant-e2e-shared-a-w3-56dbc0/rate-test-model-56dbc0-kserve-69658dcd7-hbgm6"
Sep 09 02:55:32.816773 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:32.816654    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a1cd62f-91d1-4c56-8a2f-4279d7f43a18-tmp-dir\") pod \"rate-test-model-56dbc0-kserve-69658dcd7-hbgm6\" (UID: \"8a1cd62f-91d1-4c56-8a2f-4279d7f43a18\") " pod="ai-tenant-e2e-shared-a-w3-56dbc0/rate-test-model-56dbc0-kserve-69658dcd7-hbgm6"
Sep 09 02:55:32.816773 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:32.816731    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcllz\" (UniqueName: \"kubernetes.io/projected/8a1cd62f-91d1-4c56-8a2f-4279d7f43a18-kube-api-access-dcllz\") pod \"rate-test-model-56dbc0-kserve-69658dcd7-hbgm6\" (UID: \"8a1cd62f-91d1-4c56-8a2f-4279d7f43a18\") " pod="ai-tenant-e2e-shared-a-w3-56dbc0/rate-test-model-56dbc0-kserve-69658dcd7-hbgm6"
Sep 09 02:55:32.917768 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:32.917730    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a1cd62f-91d1-4c56-8a2f-4279d7f43a18-tmp-dir\") pod \"rate-test-model-56dbc0-kserve-69658dcd7-hbgm6\" (UID: \"8a1cd62f-91d1-4c56-8a2f-4279d7f43a18\") " pod="ai-tenant-e2e-shared-a-w3-56dbc0/rate-test-model-56dbc0-kserve-69658dcd7-hbgm6"
Sep 09 02:55:32.921616 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:32.917789    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcllz\" (UniqueName: \"kubernetes.io/projected/8a1cd62f-91d1-4c56-8a2f-4279d7f43a18-kube-api-access-dcllz\") pod \"rate-test-model-56dbc0-kserve-69658dcd7-hbgm6\" (UID: \"8a1cd62f-91d1-4c56-8a2f-4279d7f43a18\") " pod="ai-tenant-e2e-shared-a-w3-56dbc0/rate-test-model-56dbc0-kserve-69658dcd7-hbgm6"
Sep 09 02:55:32.921616 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:32.917827    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a1cd62f-91d1-4c56-8a2f-4279d7f43a18-tls-certs\") pod \"rate-test-model-56dbc0-kserve-69658dcd7-hbgm6\" (UID: \"8a1cd62f-91d1-4c56-8a2f-4279d7f43a18\") " pod="ai-tenant-e2e-shared-a-w3-56dbc0/rate-test-model-56dbc0-kserve-69658dcd7-hbgm6"
Sep 09 02:55:32.921616 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:32.917847    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a1cd62f-91d1-4c56-8a2f-4279d7f43a18-home\") pod \"rate-test-model-56dbc0-kserve-69658dcd7-hbgm6\" (UID: \"8a1cd62f-91d1-4c56-8a2f-4279d7f43a18\") " pod="ai-tenant-e2e-shared-a-w3-56dbc0/rate-test-model-56dbc0-kserve-69658dcd7-hbgm6"
Sep 09 02:55:32.921616 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:32.917878    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a1cd62f-91d1-4c56-8a2f-4279d7f43a18-model-cache\") pod \"rate-test-model-56dbc0-kserve-69658dcd7-hbgm6\" (UID: \"8a1cd62f-91d1-4c56-8a2f-4279d7f43a18\") " pod="ai-tenant-e2e-shared-a-w3-56dbc0/rate-test-model-56dbc0-kserve-69658dcd7-hbgm6"
Sep 09 02:55:32.921616 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:32.917912    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a1cd62f-91d1-4c56-8a2f-4279d7f43a18-dshm\") pod \"rate-test-model-56dbc0-kserve-69658dcd7-hbgm6\" (UID: \"8a1cd62f-91d1-4c56-8a2f-4279d7f43a18\") " pod="ai-tenant-e2e-shared-a-w3-56dbc0/rate-test-model-56dbc0-kserve-69658dcd7-hbgm6"
Sep 09 02:55:32.921616 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:32.918236    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a1cd62f-91d1-4c56-8a2f-4279d7f43a18-tmp-dir\") pod \"rate-test-model-56dbc0-kserve-69658dcd7-hbgm6\" (UID: \"8a1cd62f-91d1-4c56-8a2f-4279d7f43a18\") " pod="ai-tenant-e2e-shared-a-w3-56dbc0/rate-test-model-56dbc0-kserve-69658dcd7-hbgm6"
Sep 09 02:55:32.921616 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:32.918313    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a1cd62f-91d1-4c56-8a2f-4279d7f43a18-home\") pod \"rate-test-model-56dbc0-kserve-69658dcd7-hbgm6\" (UID: \"8a1cd62f-91d1-4c56-8a2f-4279d7f43a18\") " pod="ai-tenant-e2e-shared-a-w3-56dbc0/rate-test-model-56dbc0-kserve-69658dcd7-hbgm6"
Sep 09 02:55:32.921616 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:32.918506    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a1cd62f-91d1-4c56-8a2f-4279d7f43a18-model-cache\") pod \"rate-test-model-56dbc0-kserve-69658dcd7-hbgm6\" (UID: \"8a1cd62f-91d1-4c56-8a2f-4279d7f43a18\") " pod="ai-tenant-e2e-shared-a-w3-56dbc0/rate-test-model-56dbc0-kserve-69658dcd7-hbgm6"
Sep 09 02:55:32.921616 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:32.921445    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a1cd62f-91d1-4c56-8a2f-4279d7f43a18-tls-certs\") pod \"rate-test-model-56dbc0-kserve-69658dcd7-hbgm6\" (UID: \"8a1cd62f-91d1-4c56-8a2f-4279d7f43a18\") " pod="ai-tenant-e2e-shared-a-w3-56dbc0/rate-test-model-56dbc0-kserve-69658dcd7-hbgm6"
Sep 09 02:55:32.922264 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:32.921953    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a1cd62f-91d1-4c56-8a2f-4279d7f43a18-dshm\") pod \"rate-test-model-56dbc0-kserve-69658dcd7-hbgm6\" (UID: \"8a1cd62f-91d1-4c56-8a2f-4279d7f43a18\") " pod="ai-tenant-e2e-shared-a-w3-56dbc0/rate-test-model-56dbc0-kserve-69658dcd7-hbgm6"
Sep 09 02:55:32.933129 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:32.933062    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcllz\" (UniqueName: \"kubernetes.io/projected/8a1cd62f-91d1-4c56-8a2f-4279d7f43a18-kube-api-access-dcllz\") pod \"rate-test-model-56dbc0-kserve-69658dcd7-hbgm6\" (UID: \"8a1cd62f-91d1-4c56-8a2f-4279d7f43a18\") " pod="ai-tenant-e2e-shared-a-w3-56dbc0/rate-test-model-56dbc0-kserve-69658dcd7-hbgm6"
Sep 09 02:55:33.032020 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:33.031928    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-4bd452-openshift-default-689bdb8f8d-9nmhj"]
Sep 09 02:55:33.035685 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:33.035664    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4bd452-openshift-default-689bdb8f8d-9nmhj"
Sep 09 02:55:33.038486 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:33.038464    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-4bd452-openshift-default-dockercfg-8l925\""
Sep 09 02:55:33.047688 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:33.047658    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-4bd452-openshift-default-689bdb8f8d-9nmhj"]
Sep 09 02:55:33.060448 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:33.060424    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-56dbc0/rate-test-model-56dbc0-kserve-69658dcd7-hbgm6"
Sep 09 02:55:33.083229 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:33.083205    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-2b9f0d-openshift-default-94d7f8778-n6qvg"
Sep 09 02:55:33.083610 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:33.083472    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-2b9f0d-openshift-default-94d7f8778-n6qvg"
Sep 09 02:55:33.084369 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:33.084347    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-2b9f0d-openshift-default-94d7f8778-n6qvg"
Sep 09 02:55:33.120650 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:33.119600    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3a3e1e3-41be-4919-bd49-58fbaf689785-workload-socket\") pod \"e2e-api-b-4bd452-openshift-default-689bdb8f8d-9nmhj\" (UID: \"d3a3e1e3-41be-4919-bd49-58fbaf689785\") " pod="openshift-ingress/e2e-api-b-4bd452-openshift-default-689bdb8f8d-9nmhj"
Sep 09 02:55:33.120650 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:33.119682    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3a3e1e3-41be-4919-bd49-58fbaf689785-istiod-ca-cert\") pod \"e2e-api-b-4bd452-openshift-default-689bdb8f8d-9nmhj\" (UID: \"d3a3e1e3-41be-4919-bd49-58fbaf689785\") " pod="openshift-ingress/e2e-api-b-4bd452-openshift-default-689bdb8f8d-9nmhj"
Sep 09 02:55:33.120650 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:33.119717    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3a3e1e3-41be-4919-bd49-58fbaf689785-istio-envoy\") pod \"e2e-api-b-4bd452-openshift-default-689bdb8f8d-9nmhj\" (UID: \"d3a3e1e3-41be-4919-bd49-58fbaf689785\") " pod="openshift-ingress/e2e-api-b-4bd452-openshift-default-689bdb8f8d-9nmhj"
Sep 09 02:55:33.120650 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:33.119767    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3a3e1e3-41be-4919-bd49-58fbaf689785-istio-data\") pod \"e2e-api-b-4bd452-openshift-default-689bdb8f8d-9nmhj\" (UID: \"d3a3e1e3-41be-4919-bd49-58fbaf689785\") " pod="openshift-ingress/e2e-api-b-4bd452-openshift-default-689bdb8f8d-9nmhj"
Sep 09 02:55:33.120650 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:33.119789    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3a3e1e3-41be-4919-bd49-58fbaf689785-istio-podinfo\") pod \"e2e-api-b-4bd452-openshift-default-689bdb8f8d-9nmhj\" (UID: \"d3a3e1e3-41be-4919-bd49-58fbaf689785\") " pod="openshift-ingress/e2e-api-b-4bd452-openshift-default-689bdb8f8d-9nmhj"
Sep 09 02:55:33.120650 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:33.119839    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2z7m\" (UniqueName: \"kubernetes.io/projected/d3a3e1e3-41be-4919-bd49-58fbaf689785-kube-api-access-m2z7m\") pod \"e2e-api-b-4bd452-openshift-default-689bdb8f8d-9nmhj\" (UID: \"d3a3e1e3-41be-4919-bd49-58fbaf689785\") " pod="openshift-ingress/e2e-api-b-4bd452-openshift-default-689bdb8f8d-9nmhj"
Sep 09 02:55:33.120650 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:33.119880    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3a3e1e3-41be-4919-bd49-58fbaf689785-workload-certs\") pod \"e2e-api-b-4bd452-openshift-default-689bdb8f8d-9nmhj\" (UID: \"d3a3e1e3-41be-4919-bd49-58fbaf689785\") " pod="openshift-ingress/e2e-api-b-4bd452-openshift-default-689bdb8f8d-9nmhj"
Sep 09 02:55:33.120650 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:33.119928    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3a3e1e3-41be-4919-bd49-58fbaf689785-credential-socket\") pod \"e2e-api-b-4bd452-openshift-default-689bdb8f8d-9nmhj\" (UID: \"d3a3e1e3-41be-4919-bd49-58fbaf689785\") " pod="openshift-ingress/e2e-api-b-4bd452-openshift-default-689bdb8f8d-9nmhj"
Sep 09 02:55:33.120650 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:33.119992    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3a3e1e3-41be-4919-bd49-58fbaf689785-istio-token\") pod \"e2e-api-b-4bd452-openshift-default-689bdb8f8d-9nmhj\" (UID: \"d3a3e1e3-41be-4919-bd49-58fbaf689785\") " pod="openshift-ingress/e2e-api-b-4bd452-openshift-default-689bdb8f8d-9nmhj"
Sep 09 02:55:33.221286 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:33.221246    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2z7m\" (UniqueName: \"kubernetes.io/projected/d3a3e1e3-41be-4919-bd49-58fbaf689785-kube-api-access-m2z7m\") pod \"e2e-api-b-4bd452-openshift-default-689bdb8f8d-9nmhj\" (UID: \"d3a3e1e3-41be-4919-bd49-58fbaf689785\") " pod="openshift-ingress/e2e-api-b-4bd452-openshift-default-689bdb8f8d-9nmhj"
Sep 09 02:55:33.221286 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:33.221295    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3a3e1e3-41be-4919-bd49-58fbaf689785-workload-certs\") pod \"e2e-api-b-4bd452-openshift-default-689bdb8f8d-9nmhj\" (UID: \"d3a3e1e3-41be-4919-bd49-58fbaf689785\") " pod="openshift-ingress/e2e-api-b-4bd452-openshift-default-689bdb8f8d-9nmhj"
Sep 09 02:55:33.221505 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:33.221323    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3a3e1e3-41be-4919-bd49-58fbaf689785-credential-socket\") pod \"e2e-api-b-4bd452-openshift-default-689bdb8f8d-9nmhj\" (UID: \"d3a3e1e3-41be-4919-bd49-58fbaf689785\") " pod="openshift-ingress/e2e-api-b-4bd452-openshift-default-689bdb8f8d-9nmhj"
Sep 09 02:55:33.221505 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:33.221483    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3a3e1e3-41be-4919-bd49-58fbaf689785-istio-token\") pod \"e2e-api-b-4bd452-openshift-default-689bdb8f8d-9nmhj\" (UID: \"d3a3e1e3-41be-4919-bd49-58fbaf689785\") " pod="openshift-ingress/e2e-api-b-4bd452-openshift-default-689bdb8f8d-9nmhj"
Sep 09 02:55:33.221599 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:33.221579    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3a3e1e3-41be-4919-bd49-58fbaf689785-workload-socket\") pod \"e2e-api-b-4bd452-openshift-default-689bdb8f8d-9nmhj\" (UID: \"d3a3e1e3-41be-4919-bd49-58fbaf689785\") " pod="openshift-ingress/e2e-api-b-4bd452-openshift-default-689bdb8f8d-9nmhj"
Sep 09 02:55:33.221652 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:33.221641    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3a3e1e3-41be-4919-bd49-58fbaf689785-istiod-ca-cert\") pod \"e2e-api-b-4bd452-openshift-default-689bdb8f8d-9nmhj\" (UID: \"d3a3e1e3-41be-4919-bd49-58fbaf689785\") " pod="openshift-ingress/e2e-api-b-4bd452-openshift-default-689bdb8f8d-9nmhj"
Sep 09 02:55:33.221708 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:33.221677    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3a3e1e3-41be-4919-bd49-58fbaf689785-workload-certs\") pod \"e2e-api-b-4bd452-openshift-default-689bdb8f8d-9nmhj\" (UID: \"d3a3e1e3-41be-4919-bd49-58fbaf689785\") " pod="openshift-ingress/e2e-api-b-4bd452-openshift-default-689bdb8f8d-9nmhj"
Sep 09 02:55:33.221708 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:33.221680    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3a3e1e3-41be-4919-bd49-58fbaf689785-istio-envoy\") pod \"e2e-api-b-4bd452-openshift-default-689bdb8f8d-9nmhj\" (UID: \"d3a3e1e3-41be-4919-bd49-58fbaf689785\") " pod="openshift-ingress/e2e-api-b-4bd452-openshift-default-689bdb8f8d-9nmhj"
Sep 09 02:55:33.221814 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:33.221641    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3a3e1e3-41be-4919-bd49-58fbaf689785-credential-socket\") pod \"e2e-api-b-4bd452-openshift-default-689bdb8f8d-9nmhj\" (UID: \"d3a3e1e3-41be-4919-bd49-58fbaf689785\") " pod="openshift-ingress/e2e-api-b-4bd452-openshift-default-689bdb8f8d-9nmhj"
Sep 09 02:55:33.221814 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:33.221736    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3a3e1e3-41be-4919-bd49-58fbaf689785-istio-data\") pod \"e2e-api-b-4bd452-openshift-default-689bdb8f8d-9nmhj\" (UID: \"d3a3e1e3-41be-4919-bd49-58fbaf689785\") " pod="openshift-ingress/e2e-api-b-4bd452-openshift-default-689bdb8f8d-9nmhj"
Sep 09 02:55:33.221814 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:33.221756    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3a3e1e3-41be-4919-bd49-58fbaf689785-istio-podinfo\") pod \"e2e-api-b-4bd452-openshift-default-689bdb8f8d-9nmhj\" (UID: \"d3a3e1e3-41be-4919-bd49-58fbaf689785\") " pod="openshift-ingress/e2e-api-b-4bd452-openshift-default-689bdb8f8d-9nmhj"
Sep 09 02:55:33.222005 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:33.221930    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3a3e1e3-41be-4919-bd49-58fbaf689785-workload-socket\") pod \"e2e-api-b-4bd452-openshift-default-689bdb8f8d-9nmhj\" (UID: \"d3a3e1e3-41be-4919-bd49-58fbaf689785\") " pod="openshift-ingress/e2e-api-b-4bd452-openshift-default-689bdb8f8d-9nmhj"
Sep 09 02:55:33.222061 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:33.222014    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3a3e1e3-41be-4919-bd49-58fbaf689785-istio-data\") pod \"e2e-api-b-4bd452-openshift-default-689bdb8f8d-9nmhj\" (UID: \"d3a3e1e3-41be-4919-bd49-58fbaf689785\") " pod="openshift-ingress/e2e-api-b-4bd452-openshift-default-689bdb8f8d-9nmhj"
Sep 09 02:55:33.222231 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:33.222205    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3a3e1e3-41be-4919-bd49-58fbaf689785-istiod-ca-cert\") pod \"e2e-api-b-4bd452-openshift-default-689bdb8f8d-9nmhj\" (UID: \"d3a3e1e3-41be-4919-bd49-58fbaf689785\") " pod="openshift-ingress/e2e-api-b-4bd452-openshift-default-689bdb8f8d-9nmhj"
Sep 09 02:55:33.223886 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:33.223862    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3a3e1e3-41be-4919-bd49-58fbaf689785-istio-envoy\") pod \"e2e-api-b-4bd452-openshift-default-689bdb8f8d-9nmhj\" (UID: \"d3a3e1e3-41be-4919-bd49-58fbaf689785\") " pod="openshift-ingress/e2e-api-b-4bd452-openshift-default-689bdb8f8d-9nmhj"
Sep 09 02:55:33.224115 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:33.224097    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3a3e1e3-41be-4919-bd49-58fbaf689785-istio-podinfo\") pod \"e2e-api-b-4bd452-openshift-default-689bdb8f8d-9nmhj\" (UID: \"d3a3e1e3-41be-4919-bd49-58fbaf689785\") " pod="openshift-ingress/e2e-api-b-4bd452-openshift-default-689bdb8f8d-9nmhj"
Sep 09 02:55:33.229303 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:33.229283    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2z7m\" (UniqueName: \"kubernetes.io/projected/d3a3e1e3-41be-4919-bd49-58fbaf689785-kube-api-access-m2z7m\") pod \"e2e-api-b-4bd452-openshift-default-689bdb8f8d-9nmhj\" (UID: \"d3a3e1e3-41be-4919-bd49-58fbaf689785\") " pod="openshift-ingress/e2e-api-b-4bd452-openshift-default-689bdb8f8d-9nmhj"
Sep 09 02:55:33.229790 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:33.229772    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3a3e1e3-41be-4919-bd49-58fbaf689785-istio-token\") pod \"e2e-api-b-4bd452-openshift-default-689bdb8f8d-9nmhj\" (UID: \"d3a3e1e3-41be-4919-bd49-58fbaf689785\") " pod="openshift-ingress/e2e-api-b-4bd452-openshift-default-689bdb8f8d-9nmhj"
Sep 09 02:55:33.334700 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:33.334671    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-56dbc0/rate-test-model-56dbc0-kserve-69658dcd7-hbgm6"]
Sep 09 02:55:33.337208 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:55:33.337159    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8a1cd62f_91d1_4c56_8a2f_4279d7f43a18.slice/crio-ac0b5e1f3424f88941e48afea67aebe7cc46a6d6f8cf10a62f28b19e739de666 WatchSource:0}: Error finding container ac0b5e1f3424f88941e48afea67aebe7cc46a6d6f8cf10a62f28b19e739de666: Status 404 returned error can't find the container with id ac0b5e1f3424f88941e48afea67aebe7cc46a6d6f8cf10a62f28b19e739de666
Sep 09 02:55:33.348250 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:33.348232    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4bd452-openshift-default-689bdb8f8d-9nmhj"
Sep 09 02:55:33.483199 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:33.483165    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-56dbc0/rate-test-model-56dbc0-kserve-69658dcd7-hbgm6" event={"ID":"8a1cd62f-91d1-4c56-8a2f-4279d7f43a18","Type":"ContainerStarted","Data":"129d336bbffd73929f06d9ab2022abc2ea40c73bbe1b8a860acad003fe2eba57"}
Sep 09 02:55:33.483379 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:33.483208    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-56dbc0/rate-test-model-56dbc0-kserve-69658dcd7-hbgm6" event={"ID":"8a1cd62f-91d1-4c56-8a2f-4279d7f43a18","Type":"ContainerStarted","Data":"ac0b5e1f3424f88941e48afea67aebe7cc46a6d6f8cf10a62f28b19e739de666"}
Sep 09 02:55:33.504305 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:33.504261    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-56dbc0/rate-test-model-56dbc0-kserve-69658dcd7-hbgm6" podStartSLOduration=1.504245736 podStartE2EDuration="1.504245736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:55:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:55:33.503486968 +0000 UTC m=+1515.757790605" watchObservedRunningTime="2026-09-09 02:55:33.504245736 +0000 UTC m=+1515.758549372"
Sep 09 02:55:33.625553 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:33.625527    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-4bd452-openshift-default-689bdb8f8d-9nmhj"]
Sep 09 02:55:33.627242 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:55:33.627211    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3a3e1e3_41be_4919_bd49_58fbaf689785.slice/crio-bb5df27eae70dcbe4caade724db42e5fdc37ef9e430c2e367ae9e5560228bcad WatchSource:0}: Error finding container bb5df27eae70dcbe4caade724db42e5fdc37ef9e430c2e367ae9e5560228bcad: Status 404 returned error can't find the container with id bb5df27eae70dcbe4caade724db42e5fdc37ef9e430c2e367ae9e5560228bcad
Sep 09 02:55:33.629619 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:33.629590    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 02:55:33.629697 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:33.629650    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 02:55:33.629697 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:33.629677    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 02:55:34.490607 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:34.490564    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4bd452-openshift-default-689bdb8f8d-9nmhj" event={"ID":"d3a3e1e3-41be-4919-bd49-58fbaf689785","Type":"ContainerStarted","Data":"ee8d33eae14dfed140dfc8f05a0f8c5a718de88159c767b513165da290d5392f"}
Sep 09 02:55:34.491047 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:34.490634    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4bd452-openshift-default-689bdb8f8d-9nmhj" event={"ID":"d3a3e1e3-41be-4919-bd49-58fbaf689785","Type":"ContainerStarted","Data":"bb5df27eae70dcbe4caade724db42e5fdc37ef9e430c2e367ae9e5560228bcad"}
Sep 09 02:55:34.513146 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:34.513093    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-4bd452-openshift-default-689bdb8f8d-9nmhj" podStartSLOduration=1.513077158 podStartE2EDuration="1.513077158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:55:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:55:34.510476536 +0000 UTC m=+1516.764780172" watchObservedRunningTime="2026-09-09 02:55:34.513077158 +0000 UTC m=+1516.767380794"
Sep 09 02:55:35.348477 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:35.348431    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-4bd452-openshift-default-689bdb8f8d-9nmhj"
Sep 09 02:55:35.353678 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:35.353652    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-4bd452-openshift-default-689bdb8f8d-9nmhj"
Sep 09 02:55:35.496308 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:35.496269    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-4bd452-openshift-default-689bdb8f8d-9nmhj"
Sep 09 02:55:35.497234 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:35.497213    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-4bd452-openshift-default-689bdb8f8d-9nmhj"
Sep 09 02:55:41.521132 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:41.521042    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-56dbc0/sub-test-model-56dbc0-kserve-65648d856c-68sn6" podUID="d4488ea6-6df5-4e06-be04-1fc8f8cee5bf" containerName="main" containerID="cri-o://5579821312c8c6a1b1fe851e14019d82128a96545e3b98d861fb74d5bb3db6e4" gracePeriod=45
Sep 09 02:55:41.774618 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:41.774567    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-56dbc0/sub-test-model-56dbc0-kserve-65648d856c-68sn6"
Sep 09 02:55:41.910506 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:41.910469    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4488ea6-6df5-4e06-be04-1fc8f8cee5bf-model-cache\") pod \"d4488ea6-6df5-4e06-be04-1fc8f8cee5bf\" (UID: \"d4488ea6-6df5-4e06-be04-1fc8f8cee5bf\") "
Sep 09 02:55:41.910689 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:41.910591    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pk9lg\" (UniqueName: \"kubernetes.io/projected/d4488ea6-6df5-4e06-be04-1fc8f8cee5bf-kube-api-access-pk9lg\") pod \"d4488ea6-6df5-4e06-be04-1fc8f8cee5bf\" (UID: \"d4488ea6-6df5-4e06-be04-1fc8f8cee5bf\") "
Sep 09 02:55:41.910689 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:41.910650    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4488ea6-6df5-4e06-be04-1fc8f8cee5bf-tls-certs\") pod \"d4488ea6-6df5-4e06-be04-1fc8f8cee5bf\" (UID: \"d4488ea6-6df5-4e06-be04-1fc8f8cee5bf\") "
Sep 09 02:55:41.910689 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:41.910682    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4488ea6-6df5-4e06-be04-1fc8f8cee5bf-home\") pod \"d4488ea6-6df5-4e06-be04-1fc8f8cee5bf\" (UID: \"d4488ea6-6df5-4e06-be04-1fc8f8cee5bf\") "
Sep 09 02:55:41.910862 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:41.910715    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4488ea6-6df5-4e06-be04-1fc8f8cee5bf-dshm\") pod \"d4488ea6-6df5-4e06-be04-1fc8f8cee5bf\" (UID: \"d4488ea6-6df5-4e06-be04-1fc8f8cee5bf\") "
Sep 09 02:55:41.910862 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:41.910742    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4488ea6-6df5-4e06-be04-1fc8f8cee5bf-tmp-dir\") pod \"d4488ea6-6df5-4e06-be04-1fc8f8cee5bf\" (UID: \"d4488ea6-6df5-4e06-be04-1fc8f8cee5bf\") "
Sep 09 02:55:41.910862 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:41.910740    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4488ea6-6df5-4e06-be04-1fc8f8cee5bf-model-cache" (OuterVolumeSpecName: "model-cache") pod "d4488ea6-6df5-4e06-be04-1fc8f8cee5bf" (UID: "d4488ea6-6df5-4e06-be04-1fc8f8cee5bf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 02:55:41.911027 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:41.910953    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4488ea6-6df5-4e06-be04-1fc8f8cee5bf-home" (OuterVolumeSpecName: "home") pod "d4488ea6-6df5-4e06-be04-1fc8f8cee5bf" (UID: "d4488ea6-6df5-4e06-be04-1fc8f8cee5bf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 02:55:41.911112 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:41.911049    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4488ea6-6df5-4e06-be04-1fc8f8cee5bf-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d4488ea6-6df5-4e06-be04-1fc8f8cee5bf" (UID: "d4488ea6-6df5-4e06-be04-1fc8f8cee5bf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 02:55:41.911181 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:41.911166    2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4488ea6-6df5-4e06-be04-1fc8f8cee5bf-home\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:55:41.911241 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:41.911182    2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4488ea6-6df5-4e06-be04-1fc8f8cee5bf-tmp-dir\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:55:41.911241 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:41.911201    2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4488ea6-6df5-4e06-be04-1fc8f8cee5bf-model-cache\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:55:41.912858 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:41.912833    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4488ea6-6df5-4e06-be04-1fc8f8cee5bf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d4488ea6-6df5-4e06-be04-1fc8f8cee5bf" (UID: "d4488ea6-6df5-4e06-be04-1fc8f8cee5bf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 02:55:41.912950 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:41.912850    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4488ea6-6df5-4e06-be04-1fc8f8cee5bf-dshm" (OuterVolumeSpecName: "dshm") pod "d4488ea6-6df5-4e06-be04-1fc8f8cee5bf" (UID: "d4488ea6-6df5-4e06-be04-1fc8f8cee5bf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 02:55:41.912950 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:41.912886    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4488ea6-6df5-4e06-be04-1fc8f8cee5bf-kube-api-access-pk9lg" (OuterVolumeSpecName: "kube-api-access-pk9lg") pod "d4488ea6-6df5-4e06-be04-1fc8f8cee5bf" (UID: "d4488ea6-6df5-4e06-be04-1fc8f8cee5bf"). InnerVolumeSpecName "kube-api-access-pk9lg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 02:55:42.012685 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:42.012657    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pk9lg\" (UniqueName: \"kubernetes.io/projected/d4488ea6-6df5-4e06-be04-1fc8f8cee5bf-kube-api-access-pk9lg\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:55:42.012685 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:42.012683    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4488ea6-6df5-4e06-be04-1fc8f8cee5bf-tls-certs\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:55:42.012855 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:42.012692    2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4488ea6-6df5-4e06-be04-1fc8f8cee5bf-dshm\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:55:42.536198 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:42.536114    2570 generic.go:356] "Generic (PLEG): container finished" podID="d4488ea6-6df5-4e06-be04-1fc8f8cee5bf" containerID="5579821312c8c6a1b1fe851e14019d82128a96545e3b98d861fb74d5bb3db6e4" exitCode=0
Sep 09 02:55:42.536198 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:42.536152    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-56dbc0/sub-test-model-56dbc0-kserve-65648d856c-68sn6" event={"ID":"d4488ea6-6df5-4e06-be04-1fc8f8cee5bf","Type":"ContainerDied","Data":"5579821312c8c6a1b1fe851e14019d82128a96545e3b98d861fb74d5bb3db6e4"}
Sep 09 02:55:42.536198 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:42.536161    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-56dbc0/sub-test-model-56dbc0-kserve-65648d856c-68sn6"
Sep 09 02:55:42.536198 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:42.536179    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-56dbc0/sub-test-model-56dbc0-kserve-65648d856c-68sn6" event={"ID":"d4488ea6-6df5-4e06-be04-1fc8f8cee5bf","Type":"ContainerDied","Data":"a5c6bb67399ad1b8129cdcbdc5749a1037373935453645acc1b19c721ef62fe8"}
Sep 09 02:55:42.536198 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:42.536196    2570 scope.go:117] "RemoveContainer" containerID="5579821312c8c6a1b1fe851e14019d82128a96545e3b98d861fb74d5bb3db6e4"
Sep 09 02:55:42.544758 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:42.544742    2570 scope.go:117] "RemoveContainer" containerID="5579821312c8c6a1b1fe851e14019d82128a96545e3b98d861fb74d5bb3db6e4"
Sep 09 02:55:42.545062 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:55:42.545020    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5579821312c8c6a1b1fe851e14019d82128a96545e3b98d861fb74d5bb3db6e4\": container with ID starting with 5579821312c8c6a1b1fe851e14019d82128a96545e3b98d861fb74d5bb3db6e4 not found: ID does not exist" containerID="5579821312c8c6a1b1fe851e14019d82128a96545e3b98d861fb74d5bb3db6e4"
Sep 09 02:55:42.545136 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:42.545075    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5579821312c8c6a1b1fe851e14019d82128a96545e3b98d861fb74d5bb3db6e4"} err="failed to get container status \"5579821312c8c6a1b1fe851e14019d82128a96545e3b98d861fb74d5bb3db6e4\": rpc error: code = NotFound desc = could not find container \"5579821312c8c6a1b1fe851e14019d82128a96545e3b98d861fb74d5bb3db6e4\": container with ID starting with 5579821312c8c6a1b1fe851e14019d82128a96545e3b98d861fb74d5bb3db6e4 not found: ID does not exist"
Sep 09 02:55:42.553811 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:42.553784    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-56dbc0/sub-test-model-56dbc0-kserve-65648d856c-68sn6"]
Sep 09 02:55:42.555726 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:42.555706    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-56dbc0/sub-test-model-56dbc0-kserve-65648d856c-68sn6"]
Sep 09 02:55:42.569722 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:42.569697    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-c173ad/sub-test-model-c173ad-kserve-757777b6f-2ph56" podUID="a2a4c6b8-11bf-42f4-81dd-5bd0eae58783" containerName="main" containerID="cri-o://bc41adefd45d1b7475e3b3555b90d8fab63cee9a1f6e0adf647733b8ca7e0b6a" gracePeriod=45
Sep 09 02:55:42.819228 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:42.819207    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-c173ad/sub-test-model-c173ad-kserve-757777b6f-2ph56"
Sep 09 02:55:42.922518 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:42.922486    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2a4c6b8-11bf-42f4-81dd-5bd0eae58783-tls-certs\") pod \"a2a4c6b8-11bf-42f4-81dd-5bd0eae58783\" (UID: \"a2a4c6b8-11bf-42f4-81dd-5bd0eae58783\") "
Sep 09 02:55:42.922681 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:42.922569    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2a4c6b8-11bf-42f4-81dd-5bd0eae58783-model-cache\") pod \"a2a4c6b8-11bf-42f4-81dd-5bd0eae58783\" (UID: \"a2a4c6b8-11bf-42f4-81dd-5bd0eae58783\") "
Sep 09 02:55:42.922681 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:42.922606    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kvsqf\" (UniqueName: \"kubernetes.io/projected/a2a4c6b8-11bf-42f4-81dd-5bd0eae58783-kube-api-access-kvsqf\") pod \"a2a4c6b8-11bf-42f4-81dd-5bd0eae58783\" (UID: \"a2a4c6b8-11bf-42f4-81dd-5bd0eae58783\") "
Sep 09 02:55:42.922681 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:42.922627    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2a4c6b8-11bf-42f4-81dd-5bd0eae58783-home\") pod \"a2a4c6b8-11bf-42f4-81dd-5bd0eae58783\" (UID: \"a2a4c6b8-11bf-42f4-81dd-5bd0eae58783\") "
Sep 09 02:55:42.922681 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:42.922661    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2a4c6b8-11bf-42f4-81dd-5bd0eae58783-tmp-dir\") pod \"a2a4c6b8-11bf-42f4-81dd-5bd0eae58783\" (UID: \"a2a4c6b8-11bf-42f4-81dd-5bd0eae58783\") "
Sep 09 02:55:42.922901 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:42.922698    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2a4c6b8-11bf-42f4-81dd-5bd0eae58783-dshm\") pod \"a2a4c6b8-11bf-42f4-81dd-5bd0eae58783\" (UID: \"a2a4c6b8-11bf-42f4-81dd-5bd0eae58783\") "
Sep 09 02:55:42.922901 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:42.922874    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2a4c6b8-11bf-42f4-81dd-5bd0eae58783-model-cache" (OuterVolumeSpecName: "model-cache") pod "a2a4c6b8-11bf-42f4-81dd-5bd0eae58783" (UID: "a2a4c6b8-11bf-42f4-81dd-5bd0eae58783"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 02:55:42.923045 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:42.922911    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2a4c6b8-11bf-42f4-81dd-5bd0eae58783-home" (OuterVolumeSpecName: "home") pod "a2a4c6b8-11bf-42f4-81dd-5bd0eae58783" (UID: "a2a4c6b8-11bf-42f4-81dd-5bd0eae58783"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 02:55:42.923045 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:42.922955    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2a4c6b8-11bf-42f4-81dd-5bd0eae58783-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a2a4c6b8-11bf-42f4-81dd-5bd0eae58783" (UID: "a2a4c6b8-11bf-42f4-81dd-5bd0eae58783"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 02:55:42.923202 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:42.923155    2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2a4c6b8-11bf-42f4-81dd-5bd0eae58783-model-cache\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:55:42.923202 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:42.923177    2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2a4c6b8-11bf-42f4-81dd-5bd0eae58783-home\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:55:42.923202 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:42.923190    2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2a4c6b8-11bf-42f4-81dd-5bd0eae58783-tmp-dir\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:55:42.924696 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:42.924669    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2a4c6b8-11bf-42f4-81dd-5bd0eae58783-kube-api-access-kvsqf" (OuterVolumeSpecName: "kube-api-access-kvsqf") pod "a2a4c6b8-11bf-42f4-81dd-5bd0eae58783" (UID: "a2a4c6b8-11bf-42f4-81dd-5bd0eae58783"). InnerVolumeSpecName "kube-api-access-kvsqf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 02:55:42.924790 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:42.924716    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2a4c6b8-11bf-42f4-81dd-5bd0eae58783-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a2a4c6b8-11bf-42f4-81dd-5bd0eae58783" (UID: "a2a4c6b8-11bf-42f4-81dd-5bd0eae58783"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 02:55:42.924790 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:42.924719    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2a4c6b8-11bf-42f4-81dd-5bd0eae58783-dshm" (OuterVolumeSpecName: "dshm") pod "a2a4c6b8-11bf-42f4-81dd-5bd0eae58783" (UID: "a2a4c6b8-11bf-42f4-81dd-5bd0eae58783"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 02:55:43.024109 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:43.024081    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvsqf\" (UniqueName: \"kubernetes.io/projected/a2a4c6b8-11bf-42f4-81dd-5bd0eae58783-kube-api-access-kvsqf\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:55:43.024109 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:43.024107    2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2a4c6b8-11bf-42f4-81dd-5bd0eae58783-dshm\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:55:43.024250 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:43.024118    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2a4c6b8-11bf-42f4-81dd-5bd0eae58783-tls-certs\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:55:43.061397 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:43.061294    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-56dbc0/rate-test-model-56dbc0-kserve-69658dcd7-hbgm6"
Sep 09 02:55:43.061397 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:43.061384    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-56dbc0/rate-test-model-56dbc0-kserve-69658dcd7-hbgm6"
Sep 09 02:55:43.073488 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:43.073461    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-56dbc0/rate-test-model-56dbc0-kserve-69658dcd7-hbgm6"
Sep 09 02:55:43.543458 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:43.543421    2570 generic.go:356] "Generic (PLEG): container finished" podID="a2a4c6b8-11bf-42f4-81dd-5bd0eae58783" containerID="bc41adefd45d1b7475e3b3555b90d8fab63cee9a1f6e0adf647733b8ca7e0b6a" exitCode=0
Sep 09 02:55:43.543827 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:43.543464    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-c173ad/sub-test-model-c173ad-kserve-757777b6f-2ph56"
Sep 09 02:55:43.543827 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:43.543503    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-c173ad/sub-test-model-c173ad-kserve-757777b6f-2ph56" event={"ID":"a2a4c6b8-11bf-42f4-81dd-5bd0eae58783","Type":"ContainerDied","Data":"bc41adefd45d1b7475e3b3555b90d8fab63cee9a1f6e0adf647733b8ca7e0b6a"}
Sep 09 02:55:43.543827 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:43.543539    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-c173ad/sub-test-model-c173ad-kserve-757777b6f-2ph56" event={"ID":"a2a4c6b8-11bf-42f4-81dd-5bd0eae58783","Type":"ContainerDied","Data":"44f92d2c2b21e261301736bf3871b90117b6cebcc0f2c1a5ee9dd4b34260da19"}
Sep 09 02:55:43.543827 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:43.543557    2570 scope.go:117] "RemoveContainer" containerID="bc41adefd45d1b7475e3b3555b90d8fab63cee9a1f6e0adf647733b8ca7e0b6a"
Sep 09 02:55:43.552743 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:43.552727    2570 scope.go:117] "RemoveContainer" containerID="bc41adefd45d1b7475e3b3555b90d8fab63cee9a1f6e0adf647733b8ca7e0b6a"
Sep 09 02:55:43.553049 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:55:43.553021    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc41adefd45d1b7475e3b3555b90d8fab63cee9a1f6e0adf647733b8ca7e0b6a\": container with ID starting with bc41adefd45d1b7475e3b3555b90d8fab63cee9a1f6e0adf647733b8ca7e0b6a not found: ID does not exist" containerID="bc41adefd45d1b7475e3b3555b90d8fab63cee9a1f6e0adf647733b8ca7e0b6a"
Sep 09 02:55:43.553189 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:43.553059    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc41adefd45d1b7475e3b3555b90d8fab63cee9a1f6e0adf647733b8ca7e0b6a"} err="failed to get container status \"bc41adefd45d1b7475e3b3555b90d8fab63cee9a1f6e0adf647733b8ca7e0b6a\": rpc error: code = NotFound desc = could not find container \"bc41adefd45d1b7475e3b3555b90d8fab63cee9a1f6e0adf647733b8ca7e0b6a\": container with ID starting with bc41adefd45d1b7475e3b3555b90d8fab63cee9a1f6e0adf647733b8ca7e0b6a not found: ID does not exist"
Sep 09 02:55:43.558136 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:43.558108    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-56dbc0/rate-test-model-56dbc0-kserve-69658dcd7-hbgm6"
Sep 09 02:55:43.567692 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:43.567671    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-c173ad/sub-test-model-c173ad-kserve-757777b6f-2ph56"]
Sep 09 02:55:43.571292 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:43.571272    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-c173ad/sub-test-model-c173ad-kserve-757777b6f-2ph56"]
Sep 09 02:55:44.223730 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:44.223696    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2a4c6b8-11bf-42f4-81dd-5bd0eae58783" path="/var/lib/kubelet/pods/a2a4c6b8-11bf-42f4-81dd-5bd0eae58783/volumes"
Sep 09 02:55:44.224178 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:44.224163    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4488ea6-6df5-4e06-be04-1fc8f8cee5bf" path="/var/lib/kubelet/pods/d4488ea6-6df5-4e06-be04-1fc8f8cee5bf/volumes"
Sep 09 02:55:44.552022 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:44.551913    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-99qjf"]
Sep 09 02:55:48.929940 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:48.929908    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-c173ad/rate-test-model-c173ad-kserve-7d5d757b79-qwnrm"]
Sep 09 02:55:48.930377 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:48.930328    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4488ea6-6df5-4e06-be04-1fc8f8cee5bf" containerName="main"
Sep 09 02:55:48.930377 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:48.930340    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4488ea6-6df5-4e06-be04-1fc8f8cee5bf" containerName="main"
Sep 09 02:55:48.930377 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:48.930352    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2a4c6b8-11bf-42f4-81dd-5bd0eae58783" containerName="main"
Sep 09 02:55:48.930377 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:48.930366    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2a4c6b8-11bf-42f4-81dd-5bd0eae58783" containerName="main"
Sep 09 02:55:48.930502 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:48.930427    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2a4c6b8-11bf-42f4-81dd-5bd0eae58783" containerName="main"
Sep 09 02:55:48.930502 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:48.930438    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4488ea6-6df5-4e06-be04-1fc8f8cee5bf" containerName="main"
Sep 09 02:55:48.935003 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:48.934965    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-c173ad/rate-test-model-c173ad-kserve-7d5d757b79-qwnrm"
Sep 09 02:55:48.937773 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:48.937751    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-c173ad\"/\"kube-root-ca.crt\""
Sep 09 02:55:48.938842 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:48.938816    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-c173ad\"/\"default-dockercfg-99zkq\""
Sep 09 02:55:48.938988 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:48.938948    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-c173ad\"/\"openshift-service-ca.crt\""
Sep 09 02:55:48.939315 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:48.938970    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-c173ad\"/\"rate-test-model-c173ad-kserve-self-signed-certs\""
Sep 09 02:55:48.971284 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:48.971214    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-c173ad/rate-test-model-c173ad-kserve-7d5d757b79-qwnrm"]
Sep 09 02:55:49.090030 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:49.090001    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c63dafa5-8b37-4602-b41a-d3b4171e0cda-model-cache\") pod \"rate-test-model-c173ad-kserve-7d5d757b79-qwnrm\" (UID: \"c63dafa5-8b37-4602-b41a-d3b4171e0cda\") " pod="ai-tenant-e2e-shared-b-w3-c173ad/rate-test-model-c173ad-kserve-7d5d757b79-qwnrm"
Sep 09 02:55:49.090188 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:49.090050    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c63dafa5-8b37-4602-b41a-d3b4171e0cda-tmp-dir\") pod \"rate-test-model-c173ad-kserve-7d5d757b79-qwnrm\" (UID: \"c63dafa5-8b37-4602-b41a-d3b4171e0cda\") " pod="ai-tenant-e2e-shared-b-w3-c173ad/rate-test-model-c173ad-kserve-7d5d757b79-qwnrm"
Sep 09 02:55:49.090188 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:49.090153    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c63dafa5-8b37-4602-b41a-d3b4171e0cda-tls-certs\") pod \"rate-test-model-c173ad-kserve-7d5d757b79-qwnrm\" (UID: \"c63dafa5-8b37-4602-b41a-d3b4171e0cda\") " pod="ai-tenant-e2e-shared-b-w3-c173ad/rate-test-model-c173ad-kserve-7d5d757b79-qwnrm"
Sep 09 02:55:49.090264 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:49.090189    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c63dafa5-8b37-4602-b41a-d3b4171e0cda-home\") pod \"rate-test-model-c173ad-kserve-7d5d757b79-qwnrm\" (UID: \"c63dafa5-8b37-4602-b41a-d3b4171e0cda\") " pod="ai-tenant-e2e-shared-b-w3-c173ad/rate-test-model-c173ad-kserve-7d5d757b79-qwnrm"
Sep 09 02:55:49.090264 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:49.090239    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4b8t\" (UniqueName: \"kubernetes.io/projected/c63dafa5-8b37-4602-b41a-d3b4171e0cda-kube-api-access-h4b8t\") pod \"rate-test-model-c173ad-kserve-7d5d757b79-qwnrm\" (UID: \"c63dafa5-8b37-4602-b41a-d3b4171e0cda\") " pod="ai-tenant-e2e-shared-b-w3-c173ad/rate-test-model-c173ad-kserve-7d5d757b79-qwnrm"
Sep 09 02:55:49.090333 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:49.090264    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c63dafa5-8b37-4602-b41a-d3b4171e0cda-dshm\") pod \"rate-test-model-c173ad-kserve-7d5d757b79-qwnrm\" (UID: \"c63dafa5-8b37-4602-b41a-d3b4171e0cda\") " pod="ai-tenant-e2e-shared-b-w3-c173ad/rate-test-model-c173ad-kserve-7d5d757b79-qwnrm"
Sep 09 02:55:49.191897 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:49.191803    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c63dafa5-8b37-4602-b41a-d3b4171e0cda-model-cache\") pod \"rate-test-model-c173ad-kserve-7d5d757b79-qwnrm\" (UID: \"c63dafa5-8b37-4602-b41a-d3b4171e0cda\") " pod="ai-tenant-e2e-shared-b-w3-c173ad/rate-test-model-c173ad-kserve-7d5d757b79-qwnrm"
Sep 09 02:55:49.191897 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:49.191863    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c63dafa5-8b37-4602-b41a-d3b4171e0cda-tmp-dir\") pod \"rate-test-model-c173ad-kserve-7d5d757b79-qwnrm\" (UID: \"c63dafa5-8b37-4602-b41a-d3b4171e0cda\") " pod="ai-tenant-e2e-shared-b-w3-c173ad/rate-test-model-c173ad-kserve-7d5d757b79-qwnrm"
Sep 09 02:55:49.192169 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:49.191917    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c63dafa5-8b37-4602-b41a-d3b4171e0cda-tls-certs\") pod \"rate-test-model-c173ad-kserve-7d5d757b79-qwnrm\" (UID: \"c63dafa5-8b37-4602-b41a-d3b4171e0cda\") " pod="ai-tenant-e2e-shared-b-w3-c173ad/rate-test-model-c173ad-kserve-7d5d757b79-qwnrm"
Sep 09 02:55:49.192169 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:49.191934    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c63dafa5-8b37-4602-b41a-d3b4171e0cda-home\") pod \"rate-test-model-c173ad-kserve-7d5d757b79-qwnrm\" (UID: \"c63dafa5-8b37-4602-b41a-d3b4171e0cda\") " pod="ai-tenant-e2e-shared-b-w3-c173ad/rate-test-model-c173ad-kserve-7d5d757b79-qwnrm"
Sep 09 02:55:49.192169 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:49.192000    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4b8t\" (UniqueName: \"kubernetes.io/projected/c63dafa5-8b37-4602-b41a-d3b4171e0cda-kube-api-access-h4b8t\") pod \"rate-test-model-c173ad-kserve-7d5d757b79-qwnrm\" (UID: \"c63dafa5-8b37-4602-b41a-d3b4171e0cda\") " pod="ai-tenant-e2e-shared-b-w3-c173ad/rate-test-model-c173ad-kserve-7d5d757b79-qwnrm"
Sep 09 02:55:49.192169 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:49.192025    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c63dafa5-8b37-4602-b41a-d3b4171e0cda-dshm\") pod \"rate-test-model-c173ad-kserve-7d5d757b79-qwnrm\" (UID: \"c63dafa5-8b37-4602-b41a-d3b4171e0cda\") " pod="ai-tenant-e2e-shared-b-w3-c173ad/rate-test-model-c173ad-kserve-7d5d757b79-qwnrm"
Sep 09 02:55:49.192369 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:49.192225    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c63dafa5-8b37-4602-b41a-d3b4171e0cda-tmp-dir\") pod \"rate-test-model-c173ad-kserve-7d5d757b79-qwnrm\" (UID: \"c63dafa5-8b37-4602-b41a-d3b4171e0cda\") " pod="ai-tenant-e2e-shared-b-w3-c173ad/rate-test-model-c173ad-kserve-7d5d757b79-qwnrm"
Sep 09 02:55:49.192369 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:49.192294    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c63dafa5-8b37-4602-b41a-d3b4171e0cda-model-cache\") pod \"rate-test-model-c173ad-kserve-7d5d757b79-qwnrm\" (UID: \"c63dafa5-8b37-4602-b41a-d3b4171e0cda\") " pod="ai-tenant-e2e-shared-b-w3-c173ad/rate-test-model-c173ad-kserve-7d5d757b79-qwnrm"
Sep 09 02:55:49.192435 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:49.192415    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c63dafa5-8b37-4602-b41a-d3b4171e0cda-home\") pod \"rate-test-model-c173ad-kserve-7d5d757b79-qwnrm\" (UID: \"c63dafa5-8b37-4602-b41a-d3b4171e0cda\") " pod="ai-tenant-e2e-shared-b-w3-c173ad/rate-test-model-c173ad-kserve-7d5d757b79-qwnrm"
Sep 09 02:55:49.194375 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:49.194340    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c63dafa5-8b37-4602-b41a-d3b4171e0cda-dshm\") pod \"rate-test-model-c173ad-kserve-7d5d757b79-qwnrm\" (UID: \"c63dafa5-8b37-4602-b41a-d3b4171e0cda\") " pod="ai-tenant-e2e-shared-b-w3-c173ad/rate-test-model-c173ad-kserve-7d5d757b79-qwnrm"
Sep 09 02:55:49.194682 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:49.194660    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c63dafa5-8b37-4602-b41a-d3b4171e0cda-tls-certs\") pod \"rate-test-model-c173ad-kserve-7d5d757b79-qwnrm\" (UID: \"c63dafa5-8b37-4602-b41a-d3b4171e0cda\") " pod="ai-tenant-e2e-shared-b-w3-c173ad/rate-test-model-c173ad-kserve-7d5d757b79-qwnrm"
Sep 09 02:55:49.200487 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:49.200466    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4b8t\" (UniqueName: \"kubernetes.io/projected/c63dafa5-8b37-4602-b41a-d3b4171e0cda-kube-api-access-h4b8t\") pod \"rate-test-model-c173ad-kserve-7d5d757b79-qwnrm\" (UID: \"c63dafa5-8b37-4602-b41a-d3b4171e0cda\") " pod="ai-tenant-e2e-shared-b-w3-c173ad/rate-test-model-c173ad-kserve-7d5d757b79-qwnrm"
Sep 09 02:55:49.271041 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:49.271006    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-c173ad/rate-test-model-c173ad-kserve-7d5d757b79-qwnrm"
Sep 09 02:55:49.545908 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:49.545875    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-c173ad/rate-test-model-c173ad-kserve-7d5d757b79-qwnrm"]
Sep 09 02:55:49.583942 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:49.583914    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-c173ad/rate-test-model-c173ad-kserve-7d5d757b79-qwnrm" event={"ID":"c63dafa5-8b37-4602-b41a-d3b4171e0cda","Type":"ContainerStarted","Data":"8ae5c29e46edeaa96f20b25d38c1bff049b8e565e6214133abb9fe9ccc462377"}
Sep 09 02:55:50.596894 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:50.596858    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-c173ad/rate-test-model-c173ad-kserve-7d5d757b79-qwnrm" event={"ID":"c63dafa5-8b37-4602-b41a-d3b4171e0cda","Type":"ContainerStarted","Data":"d6738928314ade24cf9ad18024e9638e8189a5101bb13b215b3e7622b7a130d5"}
Sep 09 02:55:50.628654 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:50.628604    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-c173ad/rate-test-model-c173ad-kserve-7d5d757b79-qwnrm" podStartSLOduration=2.62858963 podStartE2EDuration="2.62858963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:55:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:55:50.627099938 +0000 UTC m=+1532.881403575" watchObservedRunningTime="2026-09-09 02:55:50.62858963 +0000 UTC m=+1532.882893297"
Sep 09 02:55:54.621807 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:54.621775    2570 generic.go:356] "Generic (PLEG): container finished" podID="d3a3e1e3-41be-4919-bd49-58fbaf689785" containerID="ee8d33eae14dfed140dfc8f05a0f8c5a718de88159c767b513165da290d5392f" exitCode=0
Sep 09 02:55:54.622224 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:54.621847    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4bd452-openshift-default-689bdb8f8d-9nmhj" event={"ID":"d3a3e1e3-41be-4919-bd49-58fbaf689785","Type":"ContainerDied","Data":"ee8d33eae14dfed140dfc8f05a0f8c5a718de88159c767b513165da290d5392f"}
Sep 09 02:55:54.622289 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:54.622274    2570 scope.go:117] "RemoveContainer" containerID="ee8d33eae14dfed140dfc8f05a0f8c5a718de88159c767b513165da290d5392f"
Sep 09 02:55:54.623413 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:54.623387    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 02:55:54.623484 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:54.623458    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 02:55:54.623529 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:54.623490    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 02:55:55.628359 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:55.628322    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4bd452-openshift-default-689bdb8f8d-9nmhj" event={"ID":"d3a3e1e3-41be-4919-bd49-58fbaf689785","Type":"ContainerStarted","Data":"725ef857685bc3f7eb3256f5e7a4d6bdf93b112046e98d27ce48883954b52c52"}
Sep 09 02:55:56.349123 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:56.349082    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-4bd452-openshift-default-689bdb8f8d-9nmhj"
Sep 09 02:55:56.349329 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:56.349224    2570 patch_prober.go:28] interesting pod/e2e-api-b-4bd452-openshift-default-689bdb8f8d-9nmhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Sep 09 02:55:56.349329 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:56.349274    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4bd452-openshift-default-689bdb8f8d-9nmhj" podUID="d3a3e1e3-41be-4919-bd49-58fbaf689785" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Sep 09 02:55:57.348929 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:57.348893    2570 patch_prober.go:28] interesting pod/e2e-api-b-4bd452-openshift-default-689bdb8f8d-9nmhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Sep 09 02:55:57.349634 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:57.349601    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4bd452-openshift-default-689bdb8f8d-9nmhj" podUID="d3a3e1e3-41be-4919-bd49-58fbaf689785" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Sep 09 02:55:58.348859 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:58.348823    2570 patch_prober.go:28] interesting pod/e2e-api-b-4bd452-openshift-default-689bdb8f8d-9nmhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Sep 09 02:55:58.349181 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:58.349148    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4bd452-openshift-default-689bdb8f8d-9nmhj" podUID="d3a3e1e3-41be-4919-bd49-58fbaf689785" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Sep 09 02:55:59.271298 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:59.271258    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-c173ad/rate-test-model-c173ad-kserve-7d5d757b79-qwnrm"
Sep 09 02:55:59.271490 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:59.271335    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-c173ad/rate-test-model-c173ad-kserve-7d5d757b79-qwnrm"
Sep 09 02:55:59.284019 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:59.283995    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-c173ad/rate-test-model-c173ad-kserve-7d5d757b79-qwnrm"
Sep 09 02:55:59.349314 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:59.349280    2570 patch_prober.go:28] interesting pod/e2e-api-b-4bd452-openshift-default-689bdb8f8d-9nmhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Sep 09 02:55:59.349825 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:59.349795    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4bd452-openshift-default-689bdb8f8d-9nmhj" podUID="d3a3e1e3-41be-4919-bd49-58fbaf689785" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Sep 09 02:55:59.674895 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:55:59.674861    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-c173ad/rate-test-model-c173ad-kserve-7d5d757b79-qwnrm"
Sep 09 02:56:00.348807 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:00.348770    2570 patch_prober.go:28] interesting pod/e2e-api-b-4bd452-openshift-default-689bdb8f8d-9nmhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Sep 09 02:56:00.349133 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:00.349102    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4bd452-openshift-default-689bdb8f8d-9nmhj" podUID="d3a3e1e3-41be-4919-bd49-58fbaf689785" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Sep 09 02:56:01.540693 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:01.540620    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-4bd452-openshift-default-689bdb8f8d-9nmhj"
Sep 09 02:56:01.541209 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:01.541185    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-4bd452-openshift-default-689bdb8f8d-9nmhj"
Sep 09 02:56:01.542702 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:01.542673    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-4bd452-openshift-default-689bdb8f8d-9nmhj"
Sep 09 02:56:03.642357 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:03.642324    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-99qjf"]
Sep 09 02:56:05.245626 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:05.245593    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2b9f0d-szhhx"]
Sep 09 02:56:05.250047 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:05.250032    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2b9f0d-szhhx"
Sep 09 02:56:05.252490 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:05.252464    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 02:56:05.252590 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:05.252499    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vbr52\""
Sep 09 02:56:05.256599 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:05.256562    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2b9f0d-szhhx"]
Sep 09 02:56:05.361569 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:05.361526    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c28efc1-fc7a-4d12-aa4a-37706f7f7862-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2b9f0d-szhhx\" (UID: \"0c28efc1-fc7a-4d12-aa4a-37706f7f7862\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2b9f0d-szhhx"
Sep 09 02:56:05.462414 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:05.462356    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c28efc1-fc7a-4d12-aa4a-37706f7f7862-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2b9f0d-szhhx\" (UID: \"0c28efc1-fc7a-4d12-aa4a-37706f7f7862\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2b9f0d-szhhx"
Sep 09 02:56:05.463086 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:05.463066    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c28efc1-fc7a-4d12-aa4a-37706f7f7862-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2b9f0d-szhhx\" (UID: \"0c28efc1-fc7a-4d12-aa4a-37706f7f7862\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2b9f0d-szhhx"
Sep 09 02:56:05.562858 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:05.562783    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2b9f0d-szhhx"
Sep 09 02:56:05.835044 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:05.835014    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2b9f0d-szhhx"]
Sep 09 02:56:05.838392 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:56:05.838358    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c28efc1_fc7a_4d12_aa4a_37706f7f7862.slice/crio-d5a677be82e8c7f42bfab0217b69fa8de09e165ad53f798f1c076d6071db7404 WatchSource:0}: Error finding container d5a677be82e8c7f42bfab0217b69fa8de09e165ad53f798f1c076d6071db7404: Status 404 returned error can't find the container with id d5a677be82e8c7f42bfab0217b69fa8de09e165ad53f798f1c076d6071db7404
Sep 09 02:56:06.700451 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:06.700420    2570 generic.go:356] "Generic (PLEG): container finished" podID="0c28efc1-fc7a-4d12-aa4a-37706f7f7862" containerID="11d2e797335d0d584000678d078e2a864758213ea9730859e41e849f3e5d60c2" exitCode=0
Sep 09 02:56:06.700856 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:06.700484    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2b9f0d-szhhx" event={"ID":"0c28efc1-fc7a-4d12-aa4a-37706f7f7862","Type":"ContainerDied","Data":"11d2e797335d0d584000678d078e2a864758213ea9730859e41e849f3e5d60c2"}
Sep 09 02:56:06.700856 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:06.700537    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2b9f0d-szhhx" event={"ID":"0c28efc1-fc7a-4d12-aa4a-37706f7f7862","Type":"ContainerStarted","Data":"d5a677be82e8c7f42bfab0217b69fa8de09e165ad53f798f1c076d6071db7404"}
Sep 09 02:56:07.842454 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:07.842430    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2b9f0d-szhhx"
Sep 09 02:56:07.885347 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:07.885309    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c28efc1-fc7a-4d12-aa4a-37706f7f7862-maas-api-service-ca\") pod \"0c28efc1-fc7a-4d12-aa4a-37706f7f7862\" (UID: \"0c28efc1-fc7a-4d12-aa4a-37706f7f7862\") "
Sep 09 02:56:07.885706 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:07.885681    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c28efc1-fc7a-4d12-aa4a-37706f7f7862-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0c28efc1-fc7a-4d12-aa4a-37706f7f7862" (UID: "0c28efc1-fc7a-4d12-aa4a-37706f7f7862"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 02:56:07.986281 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:07.986207    2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c28efc1-fc7a-4d12-aa4a-37706f7f7862-maas-api-service-ca\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:56:08.712251 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:08.712224    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2b9f0d-szhhx"
Sep 09 02:56:08.712422 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:08.712219    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2b9f0d-szhhx" event={"ID":"0c28efc1-fc7a-4d12-aa4a-37706f7f7862","Type":"ContainerDied","Data":"d5a677be82e8c7f42bfab0217b69fa8de09e165ad53f798f1c076d6071db7404"}
Sep 09 02:56:08.712422 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:08.712354    2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d5a677be82e8c7f42bfab0217b69fa8de09e165ad53f798f1c076d6071db7404"
Sep 09 02:56:16.746056 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:16.746021    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-99qjf"]
Sep 09 02:56:18.468167 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:18.468136    2570 scope.go:117] "RemoveContainer" containerID="656cfaa336d42466098cdd1934d64d82b19e0d5d441012643f779c07c9388cdf"
Sep 09 02:56:18.476957 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:18.476937    2570 scope.go:117] "RemoveContainer" containerID="de3906f816701a277ea78083186b07ac0048ec79c11de1132b391d723d23bbc5"
Sep 09 02:56:21.348274 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:21.348241    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2b9f0d-szhhx"]
Sep 09 02:56:21.350647 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:21.350623    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2b9f0d-szhhx"]
Sep 09 02:56:22.223291 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:22.223254    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c28efc1-fc7a-4d12-aa4a-37706f7f7862" path="/var/lib/kubelet/pods/0c28efc1-fc7a-4d12-aa4a-37706f7f7862/volumes"
Sep 09 02:56:27.769361 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:27.769325    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-2b9f0d-openshift-default-94d7f8778-n6qvg"]
Sep 09 02:56:27.769737 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:27.769583    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-2b9f0d-openshift-default-94d7f8778-n6qvg" podUID="44eb27d1-e8e8-4689-a39f-42804def79ee" containerName="istio-proxy" containerID="cri-o://75dd315d6440bfeebe896a9c2269ab7311284dacf8edd087e52e97eb348a0511" gracePeriod=30
Sep 09 02:56:28.144384 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:28.144347    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4bd452-k6b8w"]
Sep 09 02:56:28.144742 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:28.144731    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c28efc1-fc7a-4d12-aa4a-37706f7f7862" containerName="revoke-keys"
Sep 09 02:56:28.144792 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:28.144744    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c28efc1-fc7a-4d12-aa4a-37706f7f7862" containerName="revoke-keys"
Sep 09 02:56:28.144830 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:28.144802    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c28efc1-fc7a-4d12-aa4a-37706f7f7862" containerName="revoke-keys"
Sep 09 02:56:28.148861 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:28.148846    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4bd452-k6b8w"
Sep 09 02:56:28.151556 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:28.151536    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 02:56:28.151770 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:28.151742    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vbr52\""
Sep 09 02:56:28.155296 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:28.155272    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4bd452-k6b8w"]
Sep 09 02:56:28.281868 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:28.281836    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2ff1459-b763-4c3a-a919-ec9a0fe9ee3a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4bd452-k6b8w\" (UID: \"c2ff1459-b763-4c3a-a919-ec9a0fe9ee3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4bd452-k6b8w"
Sep 09 02:56:28.383309 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:28.383271    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2ff1459-b763-4c3a-a919-ec9a0fe9ee3a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4bd452-k6b8w\" (UID: \"c2ff1459-b763-4c3a-a919-ec9a0fe9ee3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4bd452-k6b8w"
Sep 09 02:56:28.384165 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:28.384143    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2ff1459-b763-4c3a-a919-ec9a0fe9ee3a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4bd452-k6b8w\" (UID: \"c2ff1459-b763-4c3a-a919-ec9a0fe9ee3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4bd452-k6b8w"
Sep 09 02:56:28.461025 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:28.460936    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4bd452-k6b8w"
Sep 09 02:56:28.739772 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:28.739750    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4bd452-k6b8w"]
Sep 09 02:56:28.742125 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:56:28.742095    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2ff1459_b763_4c3a_a919_ec9a0fe9ee3a.slice/crio-97e7f623ad60757da774a0ec1447c6de70678869cab38d364040b1133526b45e WatchSource:0}: Error finding container 97e7f623ad60757da774a0ec1447c6de70678869cab38d364040b1133526b45e: Status 404 returned error can't find the container with id 97e7f623ad60757da774a0ec1447c6de70678869cab38d364040b1133526b45e
Sep 09 02:56:28.820121 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:28.820092    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4bd452-k6b8w" event={"ID":"c2ff1459-b763-4c3a-a919-ec9a0fe9ee3a","Type":"ContainerStarted","Data":"97e7f623ad60757da774a0ec1447c6de70678869cab38d364040b1133526b45e"}
Sep 09 02:56:29.826244 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:29.826163    2570 generic.go:356] "Generic (PLEG): container finished" podID="c2ff1459-b763-4c3a-a919-ec9a0fe9ee3a" containerID="4ba54d49846d53cc891454474cee7dd8983d27972e878bfbb2d2db7f645f3032" exitCode=0
Sep 09 02:56:29.826580 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:29.826245    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4bd452-k6b8w" event={"ID":"c2ff1459-b763-4c3a-a919-ec9a0fe9ee3a","Type":"ContainerDied","Data":"4ba54d49846d53cc891454474cee7dd8983d27972e878bfbb2d2db7f645f3032"}
Sep 09 02:56:30.976472 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:30.976449    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4bd452-k6b8w"
Sep 09 02:56:31.005910 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:31.005884    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2ff1459-b763-4c3a-a919-ec9a0fe9ee3a-maas-api-service-ca\") pod \"c2ff1459-b763-4c3a-a919-ec9a0fe9ee3a\" (UID: \"c2ff1459-b763-4c3a-a919-ec9a0fe9ee3a\") "
Sep 09 02:56:31.006211 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:31.006191    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2ff1459-b763-4c3a-a919-ec9a0fe9ee3a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c2ff1459-b763-4c3a-a919-ec9a0fe9ee3a" (UID: "c2ff1459-b763-4c3a-a919-ec9a0fe9ee3a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 02:56:31.107175 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:31.107143    2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2ff1459-b763-4c3a-a919-ec9a0fe9ee3a-maas-api-service-ca\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:56:31.841466 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:31.841437    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4bd452-k6b8w"
Sep 09 02:56:31.841638 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:31.841461    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4bd452-k6b8w" event={"ID":"c2ff1459-b763-4c3a-a919-ec9a0fe9ee3a","Type":"ContainerDied","Data":"97e7f623ad60757da774a0ec1447c6de70678869cab38d364040b1133526b45e"}
Sep 09 02:56:31.841638 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:31.841514    2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="97e7f623ad60757da774a0ec1447c6de70678869cab38d364040b1133526b45e"
Sep 09 02:56:33.014956 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:33.014932    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2b9f0d-openshift-default-94d7f8778-n6qvg"
Sep 09 02:56:33.022781 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:33.022762    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44eb27d1-e8e8-4689-a39f-42804def79ee-workload-socket\") pod \"44eb27d1-e8e8-4689-a39f-42804def79ee\" (UID: \"44eb27d1-e8e8-4689-a39f-42804def79ee\") "
Sep 09 02:56:33.022836 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:33.022801    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44eb27d1-e8e8-4689-a39f-42804def79ee-istio-podinfo\") pod \"44eb27d1-e8e8-4689-a39f-42804def79ee\" (UID: \"44eb27d1-e8e8-4689-a39f-42804def79ee\") "
Sep 09 02:56:33.022836 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:33.022821    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44eb27d1-e8e8-4689-a39f-42804def79ee-istio-token\") pod \"44eb27d1-e8e8-4689-a39f-42804def79ee\" (UID: \"44eb27d1-e8e8-4689-a39f-42804def79ee\") "
Sep 09 02:56:33.022939 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:33.022841    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44eb27d1-e8e8-4689-a39f-42804def79ee-istiod-ca-cert\") pod \"44eb27d1-e8e8-4689-a39f-42804def79ee\" (UID: \"44eb27d1-e8e8-4689-a39f-42804def79ee\") "
Sep 09 02:56:33.022939 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:33.022857    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44eb27d1-e8e8-4689-a39f-42804def79ee-istio-data\") pod \"44eb27d1-e8e8-4689-a39f-42804def79ee\" (UID: \"44eb27d1-e8e8-4689-a39f-42804def79ee\") "
Sep 09 02:56:33.022939 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:33.022877    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j74dd\" (UniqueName: \"kubernetes.io/projected/44eb27d1-e8e8-4689-a39f-42804def79ee-kube-api-access-j74dd\") pod \"44eb27d1-e8e8-4689-a39f-42804def79ee\" (UID: \"44eb27d1-e8e8-4689-a39f-42804def79ee\") "
Sep 09 02:56:33.022939 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:33.022923    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44eb27d1-e8e8-4689-a39f-42804def79ee-credential-socket\") pod \"44eb27d1-e8e8-4689-a39f-42804def79ee\" (UID: \"44eb27d1-e8e8-4689-a39f-42804def79ee\") "
Sep 09 02:56:33.023143 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:33.022953    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44eb27d1-e8e8-4689-a39f-42804def79ee-istio-envoy\") pod \"44eb27d1-e8e8-4689-a39f-42804def79ee\" (UID: \"44eb27d1-e8e8-4689-a39f-42804def79ee\") "
Sep 09 02:56:33.023143 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:33.023012    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44eb27d1-e8e8-4689-a39f-42804def79ee-workload-certs\") pod \"44eb27d1-e8e8-4689-a39f-42804def79ee\" (UID: \"44eb27d1-e8e8-4689-a39f-42804def79ee\") "
Sep 09 02:56:33.023143 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:33.023081    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44eb27d1-e8e8-4689-a39f-42804def79ee-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "44eb27d1-e8e8-4689-a39f-42804def79ee" (UID: "44eb27d1-e8e8-4689-a39f-42804def79ee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 02:56:33.023304 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:33.023200    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44eb27d1-e8e8-4689-a39f-42804def79ee-istio-data" (OuterVolumeSpecName: "istio-data") pod "44eb27d1-e8e8-4689-a39f-42804def79ee" (UID: "44eb27d1-e8e8-4689-a39f-42804def79ee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 02:56:33.023304 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:33.023295    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44eb27d1-e8e8-4689-a39f-42804def79ee-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "44eb27d1-e8e8-4689-a39f-42804def79ee" (UID: "44eb27d1-e8e8-4689-a39f-42804def79ee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 02:56:33.023406 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:33.023326    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44eb27d1-e8e8-4689-a39f-42804def79ee-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "44eb27d1-e8e8-4689-a39f-42804def79ee" (UID: "44eb27d1-e8e8-4689-a39f-42804def79ee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 02:56:33.023539 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:33.023440    2570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44eb27d1-e8e8-4689-a39f-42804def79ee-istiod-ca-cert\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:56:33.023539 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:33.023498    2570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44eb27d1-e8e8-4689-a39f-42804def79ee-istio-data\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:56:33.023539 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:33.023511    2570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44eb27d1-e8e8-4689-a39f-42804def79ee-credential-socket\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:56:33.023539 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:33.023518    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44eb27d1-e8e8-4689-a39f-42804def79ee-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "44eb27d1-e8e8-4689-a39f-42804def79ee" (UID: "44eb27d1-e8e8-4689-a39f-42804def79ee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 02:56:33.023695 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:33.023525    2570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44eb27d1-e8e8-4689-a39f-42804def79ee-workload-socket\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:56:33.025036 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:33.025013    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44eb27d1-e8e8-4689-a39f-42804def79ee-istio-token" (OuterVolumeSpecName: "istio-token") pod "44eb27d1-e8e8-4689-a39f-42804def79ee" (UID: "44eb27d1-e8e8-4689-a39f-42804def79ee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 02:56:33.025292 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:33.025262    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44eb27d1-e8e8-4689-a39f-42804def79ee-kube-api-access-j74dd" (OuterVolumeSpecName: "kube-api-access-j74dd") pod "44eb27d1-e8e8-4689-a39f-42804def79ee" (UID: "44eb27d1-e8e8-4689-a39f-42804def79ee"). InnerVolumeSpecName "kube-api-access-j74dd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 02:56:33.025425 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:33.025399    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/44eb27d1-e8e8-4689-a39f-42804def79ee-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "44eb27d1-e8e8-4689-a39f-42804def79ee" (UID: "44eb27d1-e8e8-4689-a39f-42804def79ee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 02:56:33.025483 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:33.025451    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44eb27d1-e8e8-4689-a39f-42804def79ee-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "44eb27d1-e8e8-4689-a39f-42804def79ee" (UID: "44eb27d1-e8e8-4689-a39f-42804def79ee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 02:56:33.125158 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:33.125129    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j74dd\" (UniqueName: \"kubernetes.io/projected/44eb27d1-e8e8-4689-a39f-42804def79ee-kube-api-access-j74dd\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:56:33.125158 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:33.125156    2570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44eb27d1-e8e8-4689-a39f-42804def79ee-istio-envoy\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:56:33.125341 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:33.125166    2570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44eb27d1-e8e8-4689-a39f-42804def79ee-workload-certs\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:56:33.125341 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:33.125174    2570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44eb27d1-e8e8-4689-a39f-42804def79ee-istio-podinfo\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:56:33.125341 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:33.125182    2570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44eb27d1-e8e8-4689-a39f-42804def79ee-istio-token\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:56:33.854051 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:33.854015    2570 generic.go:356] "Generic (PLEG): container finished" podID="44eb27d1-e8e8-4689-a39f-42804def79ee" containerID="75dd315d6440bfeebe896a9c2269ab7311284dacf8edd087e52e97eb348a0511" exitCode=0
Sep 09 02:56:33.854225 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:33.854063    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2b9f0d-openshift-default-94d7f8778-n6qvg" event={"ID":"44eb27d1-e8e8-4689-a39f-42804def79ee","Type":"ContainerDied","Data":"75dd315d6440bfeebe896a9c2269ab7311284dacf8edd087e52e97eb348a0511"}
Sep 09 02:56:33.854225 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:33.854105    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2b9f0d-openshift-default-94d7f8778-n6qvg" event={"ID":"44eb27d1-e8e8-4689-a39f-42804def79ee","Type":"ContainerDied","Data":"30c4623682ee7e8c9312612e68cd79f6867dc3a1e11043b7ef955a8d9317fc10"}
Sep 09 02:56:33.854225 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:33.854126    2570 scope.go:117] "RemoveContainer" containerID="75dd315d6440bfeebe896a9c2269ab7311284dacf8edd087e52e97eb348a0511"
Sep 09 02:56:33.854225 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:33.854069    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2b9f0d-openshift-default-94d7f8778-n6qvg"
Sep 09 02:56:33.863647 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:33.863631    2570 scope.go:117] "RemoveContainer" containerID="33f5a40e690562f15ff8d4a91f02c7efef70400907b210f076fa68ceb893bdeb"
Sep 09 02:56:33.871445 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:33.871430    2570 scope.go:117] "RemoveContainer" containerID="75dd315d6440bfeebe896a9c2269ab7311284dacf8edd087e52e97eb348a0511"
Sep 09 02:56:33.871682 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:56:33.871659    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75dd315d6440bfeebe896a9c2269ab7311284dacf8edd087e52e97eb348a0511\": container with ID starting with 75dd315d6440bfeebe896a9c2269ab7311284dacf8edd087e52e97eb348a0511 not found: ID does not exist" containerID="75dd315d6440bfeebe896a9c2269ab7311284dacf8edd087e52e97eb348a0511"
Sep 09 02:56:33.871775 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:33.871687    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75dd315d6440bfeebe896a9c2269ab7311284dacf8edd087e52e97eb348a0511"} err="failed to get container status \"75dd315d6440bfeebe896a9c2269ab7311284dacf8edd087e52e97eb348a0511\": rpc error: code = NotFound desc = could not find container \"75dd315d6440bfeebe896a9c2269ab7311284dacf8edd087e52e97eb348a0511\": container with ID starting with 75dd315d6440bfeebe896a9c2269ab7311284dacf8edd087e52e97eb348a0511 not found: ID does not exist"
Sep 09 02:56:33.871775 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:33.871702    2570 scope.go:117] "RemoveContainer" containerID="33f5a40e690562f15ff8d4a91f02c7efef70400907b210f076fa68ceb893bdeb"
Sep 09 02:56:33.871923 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:56:33.871907    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33f5a40e690562f15ff8d4a91f02c7efef70400907b210f076fa68ceb893bdeb\": container with ID starting with 33f5a40e690562f15ff8d4a91f02c7efef70400907b210f076fa68ceb893bdeb not found: ID does not exist" containerID="33f5a40e690562f15ff8d4a91f02c7efef70400907b210f076fa68ceb893bdeb"
Sep 09 02:56:33.871961 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:33.871929    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33f5a40e690562f15ff8d4a91f02c7efef70400907b210f076fa68ceb893bdeb"} err="failed to get container status \"33f5a40e690562f15ff8d4a91f02c7efef70400907b210f076fa68ceb893bdeb\": rpc error: code = NotFound desc = could not find container \"33f5a40e690562f15ff8d4a91f02c7efef70400907b210f076fa68ceb893bdeb\": container with ID starting with 33f5a40e690562f15ff8d4a91f02c7efef70400907b210f076fa68ceb893bdeb not found: ID does not exist"
Sep 09 02:56:33.877296 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:33.877274    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-2b9f0d-openshift-default-94d7f8778-n6qvg"]
Sep 09 02:56:33.883158 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:33.883137    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-2b9f0d-openshift-default-94d7f8778-n6qvg"]
Sep 09 02:56:34.222960 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:34.222884    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44eb27d1-e8e8-4689-a39f-42804def79ee" path="/var/lib/kubelet/pods/44eb27d1-e8e8-4689-a39f-42804def79ee/volumes"
Sep 09 02:56:34.441782 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:34.441751    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-99qjf"]
Sep 09 02:56:38.957812 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:38.957778    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-56dbc0/rate-test-model-56dbc0-kserve-69658dcd7-hbgm6"]
Sep 09 02:56:39.853417 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:39.853385    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-c173ad/rate-test-model-c173ad-kserve-7d5d757b79-qwnrm"]
Sep 09 02:56:40.647920 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:40.647886    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-2085ee-openshift-default-7fb75b87b8-nggqw"]
Sep 09 02:56:40.648289 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:40.648271    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2ff1459-b763-4c3a-a919-ec9a0fe9ee3a" containerName="revoke-keys"
Sep 09 02:56:40.648289 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:40.648283    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2ff1459-b763-4c3a-a919-ec9a0fe9ee3a" containerName="revoke-keys"
Sep 09 02:56:40.648361 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:40.648292    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44eb27d1-e8e8-4689-a39f-42804def79ee" containerName="istio-proxy"
Sep 09 02:56:40.648361 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:40.648297    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="44eb27d1-e8e8-4689-a39f-42804def79ee" containerName="istio-proxy"
Sep 09 02:56:40.648361 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:40.648304    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44eb27d1-e8e8-4689-a39f-42804def79ee" containerName="istio-proxy"
Sep 09 02:56:40.648361 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:40.648310    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="44eb27d1-e8e8-4689-a39f-42804def79ee" containerName="istio-proxy"
Sep 09 02:56:40.648538 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:40.648377    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="44eb27d1-e8e8-4689-a39f-42804def79ee" containerName="istio-proxy"
Sep 09 02:56:40.648538 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:40.648386    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2ff1459-b763-4c3a-a919-ec9a0fe9ee3a" containerName="revoke-keys"
Sep 09 02:56:40.652968 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:40.652950    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2085ee-openshift-default-7fb75b87b8-nggqw"
Sep 09 02:56:40.655552 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:40.655531    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-2085ee-openshift-default-dockercfg-hjwtb\""
Sep 09 02:56:40.662739 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:40.662714    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-2085ee-openshift-default-7fb75b87b8-nggqw"]
Sep 09 02:56:40.701124 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:40.701099    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d70260f-d945-437c-9fe6-432561b37d70-credential-socket\") pod \"e2e-ipp-a-2085ee-openshift-default-7fb75b87b8-nggqw\" (UID: \"8d70260f-d945-437c-9fe6-432561b37d70\") " pod="openshift-ingress/e2e-ipp-a-2085ee-openshift-default-7fb75b87b8-nggqw"
Sep 09 02:56:40.701216 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:40.701135    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d70260f-d945-437c-9fe6-432561b37d70-istio-data\") pod \"e2e-ipp-a-2085ee-openshift-default-7fb75b87b8-nggqw\" (UID: \"8d70260f-d945-437c-9fe6-432561b37d70\") " pod="openshift-ingress/e2e-ipp-a-2085ee-openshift-default-7fb75b87b8-nggqw"
Sep 09 02:56:40.701216 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:40.701177    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d70260f-d945-437c-9fe6-432561b37d70-workload-certs\") pod \"e2e-ipp-a-2085ee-openshift-default-7fb75b87b8-nggqw\" (UID: \"8d70260f-d945-437c-9fe6-432561b37d70\") " pod="openshift-ingress/e2e-ipp-a-2085ee-openshift-default-7fb75b87b8-nggqw"
Sep 09 02:56:40.701216 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:40.701194    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d70260f-d945-437c-9fe6-432561b37d70-istio-token\") pod \"e2e-ipp-a-2085ee-openshift-default-7fb75b87b8-nggqw\" (UID: \"8d70260f-d945-437c-9fe6-432561b37d70\") " pod="openshift-ingress/e2e-ipp-a-2085ee-openshift-default-7fb75b87b8-nggqw"
Sep 09 02:56:40.701334 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:40.701254    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpjdm\" (UniqueName: \"kubernetes.io/projected/8d70260f-d945-437c-9fe6-432561b37d70-kube-api-access-tpjdm\") pod \"e2e-ipp-a-2085ee-openshift-default-7fb75b87b8-nggqw\" (UID: \"8d70260f-d945-437c-9fe6-432561b37d70\") " pod="openshift-ingress/e2e-ipp-a-2085ee-openshift-default-7fb75b87b8-nggqw"
Sep 09 02:56:40.701334 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:40.701297    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d70260f-d945-437c-9fe6-432561b37d70-istio-podinfo\") pod \"e2e-ipp-a-2085ee-openshift-default-7fb75b87b8-nggqw\" (UID: \"8d70260f-d945-437c-9fe6-432561b37d70\") " pod="openshift-ingress/e2e-ipp-a-2085ee-openshift-default-7fb75b87b8-nggqw"
Sep 09 02:56:40.701394 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:40.701361    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d70260f-d945-437c-9fe6-432561b37d70-istio-envoy\") pod \"e2e-ipp-a-2085ee-openshift-default-7fb75b87b8-nggqw\" (UID: \"8d70260f-d945-437c-9fe6-432561b37d70\") " pod="openshift-ingress/e2e-ipp-a-2085ee-openshift-default-7fb75b87b8-nggqw"
Sep 09 02:56:40.701428 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:40.701405    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d70260f-d945-437c-9fe6-432561b37d70-istiod-ca-cert\") pod \"e2e-ipp-a-2085ee-openshift-default-7fb75b87b8-nggqw\" (UID: \"8d70260f-d945-437c-9fe6-432561b37d70\") " pod="openshift-ingress/e2e-ipp-a-2085ee-openshift-default-7fb75b87b8-nggqw"
Sep 09 02:56:40.701470 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:40.701432    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d70260f-d945-437c-9fe6-432561b37d70-workload-socket\") pod \"e2e-ipp-a-2085ee-openshift-default-7fb75b87b8-nggqw\" (UID: \"8d70260f-d945-437c-9fe6-432561b37d70\") " pod="openshift-ingress/e2e-ipp-a-2085ee-openshift-default-7fb75b87b8-nggqw"
Sep 09 02:56:40.802987 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:40.802953    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d70260f-d945-437c-9fe6-432561b37d70-istio-envoy\") pod \"e2e-ipp-a-2085ee-openshift-default-7fb75b87b8-nggqw\" (UID: \"8d70260f-d945-437c-9fe6-432561b37d70\") " pod="openshift-ingress/e2e-ipp-a-2085ee-openshift-default-7fb75b87b8-nggqw"
Sep 09 02:56:40.803139 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:40.803101    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d70260f-d945-437c-9fe6-432561b37d70-istiod-ca-cert\") pod \"e2e-ipp-a-2085ee-openshift-default-7fb75b87b8-nggqw\" (UID: \"8d70260f-d945-437c-9fe6-432561b37d70\") " pod="openshift-ingress/e2e-ipp-a-2085ee-openshift-default-7fb75b87b8-nggqw"
Sep 09 02:56:40.803139 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:40.803131    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d70260f-d945-437c-9fe6-432561b37d70-workload-socket\") pod \"e2e-ipp-a-2085ee-openshift-default-7fb75b87b8-nggqw\" (UID: \"8d70260f-d945-437c-9fe6-432561b37d70\") " pod="openshift-ingress/e2e-ipp-a-2085ee-openshift-default-7fb75b87b8-nggqw"
Sep 09 02:56:40.803303 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:40.803280    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d70260f-d945-437c-9fe6-432561b37d70-credential-socket\") pod \"e2e-ipp-a-2085ee-openshift-default-7fb75b87b8-nggqw\" (UID: \"8d70260f-d945-437c-9fe6-432561b37d70\") " pod="openshift-ingress/e2e-ipp-a-2085ee-openshift-default-7fb75b87b8-nggqw"
Sep 09 02:56:40.803357 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:40.803334    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d70260f-d945-437c-9fe6-432561b37d70-istio-data\") pod \"e2e-ipp-a-2085ee-openshift-default-7fb75b87b8-nggqw\" (UID: \"8d70260f-d945-437c-9fe6-432561b37d70\") " pod="openshift-ingress/e2e-ipp-a-2085ee-openshift-default-7fb75b87b8-nggqw"
Sep 09 02:56:40.803413 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:40.803376    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d70260f-d945-437c-9fe6-432561b37d70-workload-certs\") pod \"e2e-ipp-a-2085ee-openshift-default-7fb75b87b8-nggqw\" (UID: \"8d70260f-d945-437c-9fe6-432561b37d70\") " pod="openshift-ingress/e2e-ipp-a-2085ee-openshift-default-7fb75b87b8-nggqw"
Sep 09 02:56:40.803413 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:40.803398    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d70260f-d945-437c-9fe6-432561b37d70-istio-token\") pod \"e2e-ipp-a-2085ee-openshift-default-7fb75b87b8-nggqw\" (UID: \"8d70260f-d945-437c-9fe6-432561b37d70\") " pod="openshift-ingress/e2e-ipp-a-2085ee-openshift-default-7fb75b87b8-nggqw"
Sep 09 02:56:40.803521 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:40.803471    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpjdm\" (UniqueName: \"kubernetes.io/projected/8d70260f-d945-437c-9fe6-432561b37d70-kube-api-access-tpjdm\") pod \"e2e-ipp-a-2085ee-openshift-default-7fb75b87b8-nggqw\" (UID: \"8d70260f-d945-437c-9fe6-432561b37d70\") " pod="openshift-ingress/e2e-ipp-a-2085ee-openshift-default-7fb75b87b8-nggqw"
Sep 09 02:56:40.803521 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:40.803506    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d70260f-d945-437c-9fe6-432561b37d70-workload-socket\") pod \"e2e-ipp-a-2085ee-openshift-default-7fb75b87b8-nggqw\" (UID: \"8d70260f-d945-437c-9fe6-432561b37d70\") " pod="openshift-ingress/e2e-ipp-a-2085ee-openshift-default-7fb75b87b8-nggqw"
Sep 09 02:56:40.803632 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:40.803525    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d70260f-d945-437c-9fe6-432561b37d70-istio-podinfo\") pod \"e2e-ipp-a-2085ee-openshift-default-7fb75b87b8-nggqw\" (UID: \"8d70260f-d945-437c-9fe6-432561b37d70\") " pod="openshift-ingress/e2e-ipp-a-2085ee-openshift-default-7fb75b87b8-nggqw"
Sep 09 02:56:40.803690 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:40.803630    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d70260f-d945-437c-9fe6-432561b37d70-credential-socket\") pod \"e2e-ipp-a-2085ee-openshift-default-7fb75b87b8-nggqw\" (UID: \"8d70260f-d945-437c-9fe6-432561b37d70\") " pod="openshift-ingress/e2e-ipp-a-2085ee-openshift-default-7fb75b87b8-nggqw"
Sep 09 02:56:40.803745 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:40.803687    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d70260f-d945-437c-9fe6-432561b37d70-workload-certs\") pod \"e2e-ipp-a-2085ee-openshift-default-7fb75b87b8-nggqw\" (UID: \"8d70260f-d945-437c-9fe6-432561b37d70\") " pod="openshift-ingress/e2e-ipp-a-2085ee-openshift-default-7fb75b87b8-nggqw"
Sep 09 02:56:40.803745 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:40.803727    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d70260f-d945-437c-9fe6-432561b37d70-istio-data\") pod \"e2e-ipp-a-2085ee-openshift-default-7fb75b87b8-nggqw\" (UID: \"8d70260f-d945-437c-9fe6-432561b37d70\") " pod="openshift-ingress/e2e-ipp-a-2085ee-openshift-default-7fb75b87b8-nggqw"
Sep 09 02:56:40.803893 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:40.803872    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d70260f-d945-437c-9fe6-432561b37d70-istiod-ca-cert\") pod \"e2e-ipp-a-2085ee-openshift-default-7fb75b87b8-nggqw\" (UID: \"8d70260f-d945-437c-9fe6-432561b37d70\") " pod="openshift-ingress/e2e-ipp-a-2085ee-openshift-default-7fb75b87b8-nggqw"
Sep 09 02:56:40.805304 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:40.805280    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d70260f-d945-437c-9fe6-432561b37d70-istio-envoy\") pod \"e2e-ipp-a-2085ee-openshift-default-7fb75b87b8-nggqw\" (UID: \"8d70260f-d945-437c-9fe6-432561b37d70\") " pod="openshift-ingress/e2e-ipp-a-2085ee-openshift-default-7fb75b87b8-nggqw"
Sep 09 02:56:40.805943 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:40.805926    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d70260f-d945-437c-9fe6-432561b37d70-istio-podinfo\") pod \"e2e-ipp-a-2085ee-openshift-default-7fb75b87b8-nggqw\" (UID: \"8d70260f-d945-437c-9fe6-432561b37d70\") " pod="openshift-ingress/e2e-ipp-a-2085ee-openshift-default-7fb75b87b8-nggqw"
Sep 09 02:56:40.811454 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:40.811432    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d70260f-d945-437c-9fe6-432561b37d70-istio-token\") pod \"e2e-ipp-a-2085ee-openshift-default-7fb75b87b8-nggqw\" (UID: \"8d70260f-d945-437c-9fe6-432561b37d70\") " pod="openshift-ingress/e2e-ipp-a-2085ee-openshift-default-7fb75b87b8-nggqw"
Sep 09 02:56:40.811635 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:40.811614    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpjdm\" (UniqueName: \"kubernetes.io/projected/8d70260f-d945-437c-9fe6-432561b37d70-kube-api-access-tpjdm\") pod \"e2e-ipp-a-2085ee-openshift-default-7fb75b87b8-nggqw\" (UID: \"8d70260f-d945-437c-9fe6-432561b37d70\") " pod="openshift-ingress/e2e-ipp-a-2085ee-openshift-default-7fb75b87b8-nggqw"
Sep 09 02:56:40.964218 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:40.964136    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2085ee-openshift-default-7fb75b87b8-nggqw"
Sep 09 02:56:41.243711 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:41.243625    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-2085ee-openshift-default-7fb75b87b8-nggqw"]
Sep 09 02:56:41.246585 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:56:41.246558    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d70260f_d945_437c_9fe6_432561b37d70.slice/crio-e02d9a449ecdf1646a82926af13c253577952abb03f3f2ccfba9b2e7d268b09d WatchSource:0}: Error finding container e02d9a449ecdf1646a82926af13c253577952abb03f3f2ccfba9b2e7d268b09d: Status 404 returned error can't find the container with id e02d9a449ecdf1646a82926af13c253577952abb03f3f2ccfba9b2e7d268b09d
Sep 09 02:56:41.249110 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:41.249075    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 02:56:41.249186 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:41.249166    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 02:56:41.249231 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:41.249215    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 02:56:41.910141 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:41.910104    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2085ee-openshift-default-7fb75b87b8-nggqw" event={"ID":"8d70260f-d945-437c-9fe6-432561b37d70","Type":"ContainerStarted","Data":"96b1ad4a2f9bd24d32375e0c1b7c47cee11e44b638c9a4fa747c1291d08cda17"}
Sep 09 02:56:41.910523 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:41.910162    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2085ee-openshift-default-7fb75b87b8-nggqw" event={"ID":"8d70260f-d945-437c-9fe6-432561b37d70","Type":"ContainerStarted","Data":"e02d9a449ecdf1646a82926af13c253577952abb03f3f2ccfba9b2e7d268b09d"}
Sep 09 02:56:41.931158 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:41.931113    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-2085ee-openshift-default-7fb75b87b8-nggqw" podStartSLOduration=1.9310987050000001 podStartE2EDuration="1.931098705s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:56:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:56:41.930161011 +0000 UTC m=+1584.184464646" watchObservedRunningTime="2026-09-09 02:56:41.931098705 +0000 UTC m=+1584.185402340"
Sep 09 02:56:41.964916 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:41.964890    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-2085ee-openshift-default-7fb75b87b8-nggqw"
Sep 09 02:56:42.968908 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:42.968871    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-2085ee-openshift-default-7fb75b87b8-nggqw"
Sep 09 02:56:43.922432 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:43.922396    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-2085ee-openshift-default-7fb75b87b8-nggqw"
Sep 09 02:56:43.923370 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:43.923347    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-2085ee-openshift-default-7fb75b87b8-nggqw"
Sep 09 02:56:44.087957 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:44.087918    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4bd452-k6b8w"]
Sep 09 02:56:44.092421 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:44.092391    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4bd452-k6b8w"]
Sep 09 02:56:44.229547 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:44.229461    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2ff1459-b763-4c3a-a919-ec9a0fe9ee3a" path="/var/lib/kubelet/pods/c2ff1459-b763-4c3a-a919-ec9a0fe9ee3a/volumes"
Sep 09 02:56:50.370706 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:50.370662    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-99qjf"]
Sep 09 02:56:50.520687 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:50.520653    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-4bd452-openshift-default-689bdb8f8d-9nmhj"]
Sep 09 02:56:50.521154 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:50.521108    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-4bd452-openshift-default-689bdb8f8d-9nmhj" podUID="d3a3e1e3-41be-4919-bd49-58fbaf689785" containerName="istio-proxy" containerID="cri-o://725ef857685bc3f7eb3256f5e7a4d6bdf93b112046e98d27ce48883954b52c52" gracePeriod=30
Sep 09 02:56:53.978460 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:53.978416    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-56dbc0/rate-test-model-56dbc0-kserve-69658dcd7-hbgm6" podUID="8a1cd62f-91d1-4c56-8a2f-4279d7f43a18" containerName="main" containerID="cri-o://129d336bbffd73929f06d9ab2022abc2ea40c73bbe1b8a860acad003fe2eba57" gracePeriod=45
Sep 09 02:56:54.269812 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:54.269785    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-56dbc0/rate-test-model-56dbc0-kserve-69658dcd7-hbgm6"
Sep 09 02:56:54.364290 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:54.364257    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a1cd62f-91d1-4c56-8a2f-4279d7f43a18-dshm\") pod \"8a1cd62f-91d1-4c56-8a2f-4279d7f43a18\" (UID: \"8a1cd62f-91d1-4c56-8a2f-4279d7f43a18\") "
Sep 09 02:56:54.364481 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:54.364350    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a1cd62f-91d1-4c56-8a2f-4279d7f43a18-home\") pod \"8a1cd62f-91d1-4c56-8a2f-4279d7f43a18\" (UID: \"8a1cd62f-91d1-4c56-8a2f-4279d7f43a18\") "
Sep 09 02:56:54.364481 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:54.364414    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a1cd62f-91d1-4c56-8a2f-4279d7f43a18-tls-certs\") pod \"8a1cd62f-91d1-4c56-8a2f-4279d7f43a18\" (UID: \"8a1cd62f-91d1-4c56-8a2f-4279d7f43a18\") "
Sep 09 02:56:54.364481 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:54.364476    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a1cd62f-91d1-4c56-8a2f-4279d7f43a18-tmp-dir\") pod \"8a1cd62f-91d1-4c56-8a2f-4279d7f43a18\" (UID: \"8a1cd62f-91d1-4c56-8a2f-4279d7f43a18\") "
Sep 09 02:56:54.364667 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:54.364519    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a1cd62f-91d1-4c56-8a2f-4279d7f43a18-model-cache\") pod \"8a1cd62f-91d1-4c56-8a2f-4279d7f43a18\" (UID: \"8a1cd62f-91d1-4c56-8a2f-4279d7f43a18\") "
Sep 09 02:56:54.364667 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:54.364545    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dcllz\" (UniqueName: \"kubernetes.io/projected/8a1cd62f-91d1-4c56-8a2f-4279d7f43a18-kube-api-access-dcllz\") pod \"8a1cd62f-91d1-4c56-8a2f-4279d7f43a18\" (UID: \"8a1cd62f-91d1-4c56-8a2f-4279d7f43a18\") "
Sep 09 02:56:54.364667 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:54.364599    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a1cd62f-91d1-4c56-8a2f-4279d7f43a18-home" (OuterVolumeSpecName: "home") pod "8a1cd62f-91d1-4c56-8a2f-4279d7f43a18" (UID: "8a1cd62f-91d1-4c56-8a2f-4279d7f43a18"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 02:56:54.364878 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:54.364848    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a1cd62f-91d1-4c56-8a2f-4279d7f43a18-model-cache" (OuterVolumeSpecName: "model-cache") pod "8a1cd62f-91d1-4c56-8a2f-4279d7f43a18" (UID: "8a1cd62f-91d1-4c56-8a2f-4279d7f43a18"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 02:56:54.365096 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:54.365075    2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a1cd62f-91d1-4c56-8a2f-4279d7f43a18-model-cache\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:56:54.365211 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:54.365100    2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a1cd62f-91d1-4c56-8a2f-4279d7f43a18-home\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:56:54.365434 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:54.365408    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a1cd62f-91d1-4c56-8a2f-4279d7f43a18-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8a1cd62f-91d1-4c56-8a2f-4279d7f43a18" (UID: "8a1cd62f-91d1-4c56-8a2f-4279d7f43a18"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 02:56:54.366616 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:54.366586    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a1cd62f-91d1-4c56-8a2f-4279d7f43a18-dshm" (OuterVolumeSpecName: "dshm") pod "8a1cd62f-91d1-4c56-8a2f-4279d7f43a18" (UID: "8a1cd62f-91d1-4c56-8a2f-4279d7f43a18"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 02:56:54.366711 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:54.366667    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a1cd62f-91d1-4c56-8a2f-4279d7f43a18-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8a1cd62f-91d1-4c56-8a2f-4279d7f43a18" (UID: "8a1cd62f-91d1-4c56-8a2f-4279d7f43a18"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 02:56:54.366760 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:54.366711    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a1cd62f-91d1-4c56-8a2f-4279d7f43a18-kube-api-access-dcllz" (OuterVolumeSpecName: "kube-api-access-dcllz") pod "8a1cd62f-91d1-4c56-8a2f-4279d7f43a18" (UID: "8a1cd62f-91d1-4c56-8a2f-4279d7f43a18"). InnerVolumeSpecName "kube-api-access-dcllz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 02:56:54.466174 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:54.466141    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a1cd62f-91d1-4c56-8a2f-4279d7f43a18-tls-certs\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:56:54.466174 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:54.466176    2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a1cd62f-91d1-4c56-8a2f-4279d7f43a18-tmp-dir\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:56:54.466385 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:54.466190    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcllz\" (UniqueName: \"kubernetes.io/projected/8a1cd62f-91d1-4c56-8a2f-4279d7f43a18-kube-api-access-dcllz\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:56:54.466385 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:54.466204    2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a1cd62f-91d1-4c56-8a2f-4279d7f43a18-dshm\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:56:54.873147 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:54.873102    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-c173ad/rate-test-model-c173ad-kserve-7d5d757b79-qwnrm" podUID="c63dafa5-8b37-4602-b41a-d3b4171e0cda" containerName="main" containerID="cri-o://d6738928314ade24cf9ad18024e9638e8189a5101bb13b215b3e7622b7a130d5" gracePeriod=45
Sep 09 02:56:54.990126 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:54.990095    2570 generic.go:356] "Generic (PLEG): container finished" podID="8a1cd62f-91d1-4c56-8a2f-4279d7f43a18" containerID="129d336bbffd73929f06d9ab2022abc2ea40c73bbe1b8a860acad003fe2eba57" exitCode=0
Sep 09 02:56:54.990536 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:54.990141    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-56dbc0/rate-test-model-56dbc0-kserve-69658dcd7-hbgm6"
Sep 09 02:56:54.990536 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:54.990171    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-56dbc0/rate-test-model-56dbc0-kserve-69658dcd7-hbgm6" event={"ID":"8a1cd62f-91d1-4c56-8a2f-4279d7f43a18","Type":"ContainerDied","Data":"129d336bbffd73929f06d9ab2022abc2ea40c73bbe1b8a860acad003fe2eba57"}
Sep 09 02:56:54.990536 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:54.990218    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-56dbc0/rate-test-model-56dbc0-kserve-69658dcd7-hbgm6" event={"ID":"8a1cd62f-91d1-4c56-8a2f-4279d7f43a18","Type":"ContainerDied","Data":"ac0b5e1f3424f88941e48afea67aebe7cc46a6d6f8cf10a62f28b19e739de666"}
Sep 09 02:56:54.990536 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:54.990240    2570 scope.go:117] "RemoveContainer" containerID="129d336bbffd73929f06d9ab2022abc2ea40c73bbe1b8a860acad003fe2eba57"
Sep 09 02:56:54.992145 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:54.992120    2570 generic.go:356] "Generic (PLEG): container finished" podID="c63dafa5-8b37-4602-b41a-d3b4171e0cda" containerID="d6738928314ade24cf9ad18024e9638e8189a5101bb13b215b3e7622b7a130d5" exitCode=0
Sep 09 02:56:54.992260 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:54.992166    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-c173ad/rate-test-model-c173ad-kserve-7d5d757b79-qwnrm" event={"ID":"c63dafa5-8b37-4602-b41a-d3b4171e0cda","Type":"ContainerDied","Data":"d6738928314ade24cf9ad18024e9638e8189a5101bb13b215b3e7622b7a130d5"}
Sep 09 02:56:55.000135 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:55.000089    2570 scope.go:117] "RemoveContainer" containerID="129d336bbffd73929f06d9ab2022abc2ea40c73bbe1b8a860acad003fe2eba57"
Sep 09 02:56:55.000415 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:56:55.000390    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"129d336bbffd73929f06d9ab2022abc2ea40c73bbe1b8a860acad003fe2eba57\": container with ID starting with 129d336bbffd73929f06d9ab2022abc2ea40c73bbe1b8a860acad003fe2eba57 not found: ID does not exist" containerID="129d336bbffd73929f06d9ab2022abc2ea40c73bbe1b8a860acad003fe2eba57"
Sep 09 02:56:55.000513 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:55.000428    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"129d336bbffd73929f06d9ab2022abc2ea40c73bbe1b8a860acad003fe2eba57"} err="failed to get container status \"129d336bbffd73929f06d9ab2022abc2ea40c73bbe1b8a860acad003fe2eba57\": rpc error: code = NotFound desc = could not find container \"129d336bbffd73929f06d9ab2022abc2ea40c73bbe1b8a860acad003fe2eba57\": container with ID starting with 129d336bbffd73929f06d9ab2022abc2ea40c73bbe1b8a860acad003fe2eba57 not found: ID does not exist"
Sep 09 02:56:55.016954 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:55.016924    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-56dbc0/rate-test-model-56dbc0-kserve-69658dcd7-hbgm6"]
Sep 09 02:56:55.020851 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:55.020829    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-56dbc0/rate-test-model-56dbc0-kserve-69658dcd7-hbgm6"]
Sep 09 02:56:55.130434 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:55.130378    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-c173ad/rate-test-model-c173ad-kserve-7d5d757b79-qwnrm"
Sep 09 02:56:55.272681 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:55.272650    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c63dafa5-8b37-4602-b41a-d3b4171e0cda-model-cache\") pod \"c63dafa5-8b37-4602-b41a-d3b4171e0cda\" (UID: \"c63dafa5-8b37-4602-b41a-d3b4171e0cda\") "
Sep 09 02:56:55.272818 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:55.272703    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c63dafa5-8b37-4602-b41a-d3b4171e0cda-tls-certs\") pod \"c63dafa5-8b37-4602-b41a-d3b4171e0cda\" (UID: \"c63dafa5-8b37-4602-b41a-d3b4171e0cda\") "
Sep 09 02:56:55.272818 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:55.272757    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c63dafa5-8b37-4602-b41a-d3b4171e0cda-home\") pod \"c63dafa5-8b37-4602-b41a-d3b4171e0cda\" (UID: \"c63dafa5-8b37-4602-b41a-d3b4171e0cda\") "
Sep 09 02:56:55.272818 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:55.272795    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h4b8t\" (UniqueName: \"kubernetes.io/projected/c63dafa5-8b37-4602-b41a-d3b4171e0cda-kube-api-access-h4b8t\") pod \"c63dafa5-8b37-4602-b41a-d3b4171e0cda\" (UID: \"c63dafa5-8b37-4602-b41a-d3b4171e0cda\") "
Sep 09 02:56:55.273017 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:55.272835    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c63dafa5-8b37-4602-b41a-d3b4171e0cda-dshm\") pod \"c63dafa5-8b37-4602-b41a-d3b4171e0cda\" (UID: \"c63dafa5-8b37-4602-b41a-d3b4171e0cda\") "
Sep 09 02:56:55.273017 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:55.272864    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c63dafa5-8b37-4602-b41a-d3b4171e0cda-tmp-dir\") pod \"c63dafa5-8b37-4602-b41a-d3b4171e0cda\" (UID: \"c63dafa5-8b37-4602-b41a-d3b4171e0cda\") "
Sep 09 02:56:55.273017 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:55.272921    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c63dafa5-8b37-4602-b41a-d3b4171e0cda-model-cache" (OuterVolumeSpecName: "model-cache") pod "c63dafa5-8b37-4602-b41a-d3b4171e0cda" (UID: "c63dafa5-8b37-4602-b41a-d3b4171e0cda"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 02:56:55.273263 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:55.273049    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c63dafa5-8b37-4602-b41a-d3b4171e0cda-home" (OuterVolumeSpecName: "home") pod "c63dafa5-8b37-4602-b41a-d3b4171e0cda" (UID: "c63dafa5-8b37-4602-b41a-d3b4171e0cda"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 02:56:55.273263 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:55.273124    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c63dafa5-8b37-4602-b41a-d3b4171e0cda-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c63dafa5-8b37-4602-b41a-d3b4171e0cda" (UID: "c63dafa5-8b37-4602-b41a-d3b4171e0cda"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 02:56:55.273263 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:55.273244    2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c63dafa5-8b37-4602-b41a-d3b4171e0cda-model-cache\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:56:55.273263 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:55.273263    2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c63dafa5-8b37-4602-b41a-d3b4171e0cda-home\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:56:55.273413 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:55.273274    2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c63dafa5-8b37-4602-b41a-d3b4171e0cda-tmp-dir\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:56:55.274932 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:55.274905    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c63dafa5-8b37-4602-b41a-d3b4171e0cda-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c63dafa5-8b37-4602-b41a-d3b4171e0cda" (UID: "c63dafa5-8b37-4602-b41a-d3b4171e0cda"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 02:56:55.275080 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:55.274955    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c63dafa5-8b37-4602-b41a-d3b4171e0cda-dshm" (OuterVolumeSpecName: "dshm") pod "c63dafa5-8b37-4602-b41a-d3b4171e0cda" (UID: "c63dafa5-8b37-4602-b41a-d3b4171e0cda"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 02:56:55.275080 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:55.274965    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c63dafa5-8b37-4602-b41a-d3b4171e0cda-kube-api-access-h4b8t" (OuterVolumeSpecName: "kube-api-access-h4b8t") pod "c63dafa5-8b37-4602-b41a-d3b4171e0cda" (UID: "c63dafa5-8b37-4602-b41a-d3b4171e0cda"). InnerVolumeSpecName "kube-api-access-h4b8t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 02:56:55.374119 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:55.374082    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c63dafa5-8b37-4602-b41a-d3b4171e0cda-tls-certs\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:56:55.374119 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:55.374108    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4b8t\" (UniqueName: \"kubernetes.io/projected/c63dafa5-8b37-4602-b41a-d3b4171e0cda-kube-api-access-h4b8t\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:56:55.374119 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:55.374123    2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c63dafa5-8b37-4602-b41a-d3b4171e0cda-dshm\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:56:55.795054 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:55.795025    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4bd452-openshift-default-689bdb8f8d-9nmhj"
Sep 09 02:56:55.979361 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:55.979327    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3a3e1e3-41be-4919-bd49-58fbaf689785-credential-socket\") pod \"d3a3e1e3-41be-4919-bd49-58fbaf689785\" (UID: \"d3a3e1e3-41be-4919-bd49-58fbaf689785\") "
Sep 09 02:56:55.979361 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:55.979368    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3a3e1e3-41be-4919-bd49-58fbaf689785-istio-envoy\") pod \"d3a3e1e3-41be-4919-bd49-58fbaf689785\" (UID: \"d3a3e1e3-41be-4919-bd49-58fbaf689785\") "
Sep 09 02:56:55.979574 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:55.979391    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3a3e1e3-41be-4919-bd49-58fbaf689785-workload-socket\") pod \"d3a3e1e3-41be-4919-bd49-58fbaf689785\" (UID: \"d3a3e1e3-41be-4919-bd49-58fbaf689785\") "
Sep 09 02:56:55.979574 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:55.979447    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m2z7m\" (UniqueName: \"kubernetes.io/projected/d3a3e1e3-41be-4919-bd49-58fbaf689785-kube-api-access-m2z7m\") pod \"d3a3e1e3-41be-4919-bd49-58fbaf689785\" (UID: \"d3a3e1e3-41be-4919-bd49-58fbaf689785\") "
Sep 09 02:56:55.979574 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:55.979514    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3a3e1e3-41be-4919-bd49-58fbaf689785-istio-token\") pod \"d3a3e1e3-41be-4919-bd49-58fbaf689785\" (UID: \"d3a3e1e3-41be-4919-bd49-58fbaf689785\") "
Sep 09 02:56:55.979574 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:55.979538    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3a3e1e3-41be-4919-bd49-58fbaf689785-istio-data\") pod \"d3a3e1e3-41be-4919-bd49-58fbaf689785\" (UID: \"d3a3e1e3-41be-4919-bd49-58fbaf689785\") "
Sep 09 02:56:55.979574 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:55.979559    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3a3e1e3-41be-4919-bd49-58fbaf689785-istio-podinfo\") pod \"d3a3e1e3-41be-4919-bd49-58fbaf689785\" (UID: \"d3a3e1e3-41be-4919-bd49-58fbaf689785\") "
Sep 09 02:56:55.979772 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:55.979588    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3a3e1e3-41be-4919-bd49-58fbaf689785-workload-certs\") pod \"d3a3e1e3-41be-4919-bd49-58fbaf689785\" (UID: \"d3a3e1e3-41be-4919-bd49-58fbaf689785\") "
Sep 09 02:56:55.979995 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:55.979955    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3a3e1e3-41be-4919-bd49-58fbaf689785-istiod-ca-cert\") pod \"d3a3e1e3-41be-4919-bd49-58fbaf689785\" (UID: \"d3a3e1e3-41be-4919-bd49-58fbaf689785\") "
Sep 09 02:56:55.980078 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:55.979659    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3a3e1e3-41be-4919-bd49-58fbaf689785-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d3a3e1e3-41be-4919-bd49-58fbaf689785" (UID: "d3a3e1e3-41be-4919-bd49-58fbaf689785"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 02:56:55.980078 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:55.979723    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3a3e1e3-41be-4919-bd49-58fbaf689785-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d3a3e1e3-41be-4919-bd49-58fbaf689785" (UID: "d3a3e1e3-41be-4919-bd49-58fbaf689785"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 02:56:55.980187 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:55.979760    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3a3e1e3-41be-4919-bd49-58fbaf689785-istio-data" (OuterVolumeSpecName: "istio-data") pod "d3a3e1e3-41be-4919-bd49-58fbaf689785" (UID: "d3a3e1e3-41be-4919-bd49-58fbaf689785"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 02:56:55.980187 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:55.979917    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3a3e1e3-41be-4919-bd49-58fbaf689785-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d3a3e1e3-41be-4919-bd49-58fbaf689785" (UID: "d3a3e1e3-41be-4919-bd49-58fbaf689785"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 02:56:55.980343 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:55.980321    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3a3e1e3-41be-4919-bd49-58fbaf689785-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d3a3e1e3-41be-4919-bd49-58fbaf689785" (UID: "d3a3e1e3-41be-4919-bd49-58fbaf689785"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 02:56:55.980436 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:55.980420    2570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3a3e1e3-41be-4919-bd49-58fbaf689785-istio-data\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:56:55.980493 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:55.980443    2570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3a3e1e3-41be-4919-bd49-58fbaf689785-workload-certs\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:56:55.980493 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:55.980456    2570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3a3e1e3-41be-4919-bd49-58fbaf689785-istiod-ca-cert\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:56:55.980493 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:55.980469    2570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3a3e1e3-41be-4919-bd49-58fbaf689785-credential-socket\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:56:55.980493 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:55.980482    2570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3a3e1e3-41be-4919-bd49-58fbaf689785-workload-socket\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:56:55.981825 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:55.981795    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3a3e1e3-41be-4919-bd49-58fbaf689785-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d3a3e1e3-41be-4919-bd49-58fbaf689785" (UID: "d3a3e1e3-41be-4919-bd49-58fbaf689785"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 02:56:55.981909 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:55.981828    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3a3e1e3-41be-4919-bd49-58fbaf689785-kube-api-access-m2z7m" (OuterVolumeSpecName: "kube-api-access-m2z7m") pod "d3a3e1e3-41be-4919-bd49-58fbaf689785" (UID: "d3a3e1e3-41be-4919-bd49-58fbaf689785"). InnerVolumeSpecName "kube-api-access-m2z7m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 02:56:55.982135 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:55.982107    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d3a3e1e3-41be-4919-bd49-58fbaf689785-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d3a3e1e3-41be-4919-bd49-58fbaf689785" (UID: "d3a3e1e3-41be-4919-bd49-58fbaf689785"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 02:56:55.982135 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:55.982112    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3a3e1e3-41be-4919-bd49-58fbaf689785-istio-token" (OuterVolumeSpecName: "istio-token") pod "d3a3e1e3-41be-4919-bd49-58fbaf689785" (UID: "d3a3e1e3-41be-4919-bd49-58fbaf689785"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 02:56:55.999453 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:55.999429    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-c173ad/rate-test-model-c173ad-kserve-7d5d757b79-qwnrm" event={"ID":"c63dafa5-8b37-4602-b41a-d3b4171e0cda","Type":"ContainerDied","Data":"8ae5c29e46edeaa96f20b25d38c1bff049b8e565e6214133abb9fe9ccc462377"}
Sep 09 02:56:55.999453 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:55.999447    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-c173ad/rate-test-model-c173ad-kserve-7d5d757b79-qwnrm"
Sep 09 02:56:55.999820 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:55.999473    2570 scope.go:117] "RemoveContainer" containerID="d6738928314ade24cf9ad18024e9638e8189a5101bb13b215b3e7622b7a130d5"
Sep 09 02:56:56.001279 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:56.001258    2570 generic.go:356] "Generic (PLEG): container finished" podID="d3a3e1e3-41be-4919-bd49-58fbaf689785" containerID="725ef857685bc3f7eb3256f5e7a4d6bdf93b112046e98d27ce48883954b52c52" exitCode=0
Sep 09 02:56:56.001361 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:56.001289    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4bd452-openshift-default-689bdb8f8d-9nmhj" event={"ID":"d3a3e1e3-41be-4919-bd49-58fbaf689785","Type":"ContainerDied","Data":"725ef857685bc3f7eb3256f5e7a4d6bdf93b112046e98d27ce48883954b52c52"}
Sep 09 02:56:56.001361 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:56.001325    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4bd452-openshift-default-689bdb8f8d-9nmhj"
Sep 09 02:56:56.001465 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:56.001330    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4bd452-openshift-default-689bdb8f8d-9nmhj" event={"ID":"d3a3e1e3-41be-4919-bd49-58fbaf689785","Type":"ContainerDied","Data":"bb5df27eae70dcbe4caade724db42e5fdc37ef9e430c2e367ae9e5560228bcad"}
Sep 09 02:56:56.009906 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:56.009886    2570 scope.go:117] "RemoveContainer" containerID="725ef857685bc3f7eb3256f5e7a4d6bdf93b112046e98d27ce48883954b52c52"
Sep 09 02:56:56.022541 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:56.022514    2570 scope.go:117] "RemoveContainer" containerID="ee8d33eae14dfed140dfc8f05a0f8c5a718de88159c767b513165da290d5392f"
Sep 09 02:56:56.028224 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:56.028199    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-c173ad/rate-test-model-c173ad-kserve-7d5d757b79-qwnrm"]
Sep 09 02:56:56.030572 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:56.030554    2570 scope.go:117] "RemoveContainer" containerID="725ef857685bc3f7eb3256f5e7a4d6bdf93b112046e98d27ce48883954b52c52"
Sep 09 02:56:56.030866 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:56:56.030843    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"725ef857685bc3f7eb3256f5e7a4d6bdf93b112046e98d27ce48883954b52c52\": container with ID starting with 725ef857685bc3f7eb3256f5e7a4d6bdf93b112046e98d27ce48883954b52c52 not found: ID does not exist" containerID="725ef857685bc3f7eb3256f5e7a4d6bdf93b112046e98d27ce48883954b52c52"
Sep 09 02:56:56.031064 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:56.030869    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"725ef857685bc3f7eb3256f5e7a4d6bdf93b112046e98d27ce48883954b52c52"} err="failed to get container status \"725ef857685bc3f7eb3256f5e7a4d6bdf93b112046e98d27ce48883954b52c52\": rpc error: code = NotFound desc = could not find container \"725ef857685bc3f7eb3256f5e7a4d6bdf93b112046e98d27ce48883954b52c52\": container with ID starting with 725ef857685bc3f7eb3256f5e7a4d6bdf93b112046e98d27ce48883954b52c52 not found: ID does not exist"
Sep 09 02:56:56.031064 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:56.030884    2570 scope.go:117] "RemoveContainer" containerID="ee8d33eae14dfed140dfc8f05a0f8c5a718de88159c767b513165da290d5392f"
Sep 09 02:56:56.031201 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:56:56.031134    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee8d33eae14dfed140dfc8f05a0f8c5a718de88159c767b513165da290d5392f\": container with ID starting with ee8d33eae14dfed140dfc8f05a0f8c5a718de88159c767b513165da290d5392f not found: ID does not exist" containerID="ee8d33eae14dfed140dfc8f05a0f8c5a718de88159c767b513165da290d5392f"
Sep 09 02:56:56.031201 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:56.031151    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee8d33eae14dfed140dfc8f05a0f8c5a718de88159c767b513165da290d5392f"} err="failed to get container status \"ee8d33eae14dfed140dfc8f05a0f8c5a718de88159c767b513165da290d5392f\": rpc error: code = NotFound desc = could not find container \"ee8d33eae14dfed140dfc8f05a0f8c5a718de88159c767b513165da290d5392f\": container with ID starting with ee8d33eae14dfed140dfc8f05a0f8c5a718de88159c767b513165da290d5392f not found: ID does not exist"
Sep 09 02:56:56.031795 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:56.031772    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-c173ad/rate-test-model-c173ad-kserve-7d5d757b79-qwnrm"]
Sep 09 02:56:56.060100 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:56.060031    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-4bd452-openshift-default-689bdb8f8d-9nmhj"]
Sep 09 02:56:56.065248 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:56.065227    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-4bd452-openshift-default-689bdb8f8d-9nmhj"]
Sep 09 02:56:56.081236 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:56.081215    2570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3a3e1e3-41be-4919-bd49-58fbaf689785-istio-envoy\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:56:56.081317 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:56.081239    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2z7m\" (UniqueName: \"kubernetes.io/projected/d3a3e1e3-41be-4919-bd49-58fbaf689785-kube-api-access-m2z7m\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:56:56.081317 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:56.081255    2570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3a3e1e3-41be-4919-bd49-58fbaf689785-istio-token\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:56:56.081317 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:56.081268    2570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3a3e1e3-41be-4919-bd49-58fbaf689785-istio-podinfo\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:56:56.222859 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:56.222826    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a1cd62f-91d1-4c56-8a2f-4279d7f43a18" path="/var/lib/kubelet/pods/8a1cd62f-91d1-4c56-8a2f-4279d7f43a18/volumes"
Sep 09 02:56:56.223310 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:56.223294    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c63dafa5-8b37-4602-b41a-d3b4171e0cda" path="/var/lib/kubelet/pods/c63dafa5-8b37-4602-b41a-d3b4171e0cda/volumes"
Sep 09 02:56:56.223686 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:56.223674    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3a3e1e3-41be-4919-bd49-58fbaf689785" path="/var/lib/kubelet/pods/d3a3e1e3-41be-4919-bd49-58fbaf689785/volumes"
Sep 09 02:56:59.964114 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:59.964078    2570 patch_prober.go:28] interesting pod/e2e-ipp-a-2085ee-openshift-default-7fb75b87b8-nggqw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 09 02:56:59.964572 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:56:59.964135    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-2085ee-openshift-default-7fb75b87b8-nggqw" podUID="8d70260f-d945-437c-9fe6-432561b37d70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 09 02:57:01.031021 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:01.030964    2570 generic.go:356] "Generic (PLEG): container finished" podID="8d70260f-d945-437c-9fe6-432561b37d70" containerID="96b1ad4a2f9bd24d32375e0c1b7c47cee11e44b638c9a4fa747c1291d08cda17" exitCode=0
Sep 09 02:57:01.031476 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:01.031045    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2085ee-openshift-default-7fb75b87b8-nggqw" event={"ID":"8d70260f-d945-437c-9fe6-432561b37d70","Type":"ContainerDied","Data":"96b1ad4a2f9bd24d32375e0c1b7c47cee11e44b638c9a4fa747c1291d08cda17"}
Sep 09 02:57:01.031608 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:01.031589    2570 scope.go:117] "RemoveContainer" containerID="96b1ad4a2f9bd24d32375e0c1b7c47cee11e44b638c9a4fa747c1291d08cda17"
Sep 09 02:57:01.032839 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:01.032807    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 02:57:01.032948 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:01.032893    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 02:57:01.033036 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:01.032943    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 02:57:02.043118 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:02.043073    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2085ee-openshift-default-7fb75b87b8-nggqw" event={"ID":"8d70260f-d945-437c-9fe6-432561b37d70","Type":"ContainerStarted","Data":"17a49bac98918b39d38558e406879f2c301302438aa34df53d3977cf94cce3e0"}
Sep 09 02:57:02.964369 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:02.964319    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-2085ee-openshift-default-7fb75b87b8-nggqw"
Sep 09 02:57:02.964611 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:02.964581    2570 patch_prober.go:28] interesting pod/e2e-ipp-a-2085ee-openshift-default-7fb75b87b8-nggqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Sep 09 02:57:02.964691 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:02.964649    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2085ee-openshift-default-7fb75b87b8-nggqw" podUID="8d70260f-d945-437c-9fe6-432561b37d70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Sep 09 02:57:03.965097 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:03.965056    2570 patch_prober.go:28] interesting pod/e2e-ipp-a-2085ee-openshift-default-7fb75b87b8-nggqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Sep 09 02:57:03.965601 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:03.965123    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2085ee-openshift-default-7fb75b87b8-nggqw" podUID="8d70260f-d945-437c-9fe6-432561b37d70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Sep 09 02:57:04.965036 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:04.964993    2570 patch_prober.go:28] interesting pod/e2e-ipp-a-2085ee-openshift-default-7fb75b87b8-nggqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Sep 09 02:57:04.965228 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:04.965054    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2085ee-openshift-default-7fb75b87b8-nggqw" podUID="8d70260f-d945-437c-9fe6-432561b37d70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Sep 09 02:57:05.964779 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:05.964732    2570 patch_prober.go:28] interesting pod/e2e-ipp-a-2085ee-openshift-default-7fb75b87b8-nggqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Sep 09 02:57:05.964998 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:05.964798    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2085ee-openshift-default-7fb75b87b8-nggqw" podUID="8d70260f-d945-437c-9fe6-432561b37d70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Sep 09 02:57:06.965023 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:06.964953    2570 patch_prober.go:28] interesting pod/e2e-ipp-a-2085ee-openshift-default-7fb75b87b8-nggqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Sep 09 02:57:06.965414 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:06.965054    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2085ee-openshift-default-7fb75b87b8-nggqw" podUID="8d70260f-d945-437c-9fe6-432561b37d70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Sep 09 02:57:07.045763 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:07.045725    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-99qjf"]
Sep 09 02:57:08.008837 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:08.008802    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-2085ee-openshift-default-7fb75b87b8-nggqw"
Sep 09 02:57:08.009294 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:08.009179    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-2085ee-openshift-default-7fb75b87b8-nggqw"
Sep 09 02:57:08.009824 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:08.009808    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-2085ee-openshift-default-7fb75b87b8-nggqw"
Sep 09 02:57:08.288542 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:08.288455    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-d838aa-openshift-default-6989bbb4f6-gj56v"]
Sep 09 02:57:08.288747 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:08.288720    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-d838aa-openshift-default-6989bbb4f6-gj56v" podUID="cb43df80-abb3-4a89-870a-dc6587abf0fa" containerName="istio-proxy" containerID="cri-o://e825f79926523f2902e7cbbf9b3068fd4841d3ed4ed7aec0f75721deff0d4345" gracePeriod=30
Sep 09 02:57:08.698774 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:08.698745    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f5aebb-7ckh4"]
Sep 09 02:57:08.699169 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:08.699155    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3a3e1e3-41be-4919-bd49-58fbaf689785" containerName="istio-proxy"
Sep 09 02:57:08.699233 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:08.699171    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3a3e1e3-41be-4919-bd49-58fbaf689785" containerName="istio-proxy"
Sep 09 02:57:08.699233 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:08.699180    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a1cd62f-91d1-4c56-8a2f-4279d7f43a18" containerName="main"
Sep 09 02:57:08.699233 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:08.699185    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a1cd62f-91d1-4c56-8a2f-4279d7f43a18" containerName="main"
Sep 09 02:57:08.699233 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:08.699194    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c63dafa5-8b37-4602-b41a-d3b4171e0cda" containerName="main"
Sep 09 02:57:08.699233 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:08.699199    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="c63dafa5-8b37-4602-b41a-d3b4171e0cda" containerName="main"
Sep 09 02:57:08.699233 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:08.699209    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3a3e1e3-41be-4919-bd49-58fbaf689785" containerName="istio-proxy"
Sep 09 02:57:08.699233 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:08.699214    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3a3e1e3-41be-4919-bd49-58fbaf689785" containerName="istio-proxy"
Sep 09 02:57:08.699451 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:08.699275    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="44eb27d1-e8e8-4689-a39f-42804def79ee" containerName="istio-proxy"
Sep 09 02:57:08.699451 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:08.699283    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3a3e1e3-41be-4919-bd49-58fbaf689785" containerName="istio-proxy"
Sep 09 02:57:08.699451 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:08.699291    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a1cd62f-91d1-4c56-8a2f-4279d7f43a18" containerName="main"
Sep 09 02:57:08.699451 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:08.699298    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="c63dafa5-8b37-4602-b41a-d3b4171e0cda" containerName="main"
Sep 09 02:57:08.699451 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:08.699304    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3a3e1e3-41be-4919-bd49-58fbaf689785" containerName="istio-proxy"
Sep 09 02:57:08.701251 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:08.701237    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f5aebb-7ckh4"
Sep 09 02:57:08.703638 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:08.703623    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 02:57:08.703902 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:08.703883    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vbr52\""
Sep 09 02:57:08.710550 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:08.710524    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f5aebb-7ckh4"]
Sep 09 02:57:08.814233 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:08.814202    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2e584dc-69ad-43db-9a53-e6be4f66640a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-f5aebb-7ckh4\" (UID: \"c2e584dc-69ad-43db-9a53-e6be4f66640a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f5aebb-7ckh4"
Sep 09 02:57:08.915267 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:08.915241    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2e584dc-69ad-43db-9a53-e6be4f66640a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-f5aebb-7ckh4\" (UID: \"c2e584dc-69ad-43db-9a53-e6be4f66640a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f5aebb-7ckh4"
Sep 09 02:57:08.915864 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:08.915836    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2e584dc-69ad-43db-9a53-e6be4f66640a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-f5aebb-7ckh4\" (UID: \"c2e584dc-69ad-43db-9a53-e6be4f66640a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f5aebb-7ckh4"
Sep 09 02:57:09.012414 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:09.012343    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f5aebb-7ckh4"
Sep 09 02:57:09.281436 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:09.281372    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f5aebb-7ckh4"]
Sep 09 02:57:09.283740 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:57:09.283712    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2e584dc_69ad_43db_9a53_e6be4f66640a.slice/crio-ba49c5944a1f3962162070c4bd911522885a56c5dd60bff9a30b000e4c6695c3 WatchSource:0}: Error finding container ba49c5944a1f3962162070c4bd911522885a56c5dd60bff9a30b000e4c6695c3: Status 404 returned error can't find the container with id ba49c5944a1f3962162070c4bd911522885a56c5dd60bff9a30b000e4c6695c3
Sep 09 02:57:10.098229 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:10.098193    2570 generic.go:356] "Generic (PLEG): container finished" podID="c2e584dc-69ad-43db-9a53-e6be4f66640a" containerID="0b5473fcd1e047e4d01dd02bae3d1e66b43cc51fe2edb030c21727f9463f6a69" exitCode=0
Sep 09 02:57:10.098615 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:10.098276    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f5aebb-7ckh4" event={"ID":"c2e584dc-69ad-43db-9a53-e6be4f66640a","Type":"ContainerDied","Data":"0b5473fcd1e047e4d01dd02bae3d1e66b43cc51fe2edb030c21727f9463f6a69"}
Sep 09 02:57:10.098615 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:10.098323    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f5aebb-7ckh4" event={"ID":"c2e584dc-69ad-43db-9a53-e6be4f66640a","Type":"ContainerStarted","Data":"ba49c5944a1f3962162070c4bd911522885a56c5dd60bff9a30b000e4c6695c3"}
Sep 09 02:57:10.130073 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:10.130045    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-621ac9-openshift-default-549f68cf56-dl2vr"]
Sep 09 02:57:10.132166 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:10.132149    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-621ac9-openshift-default-549f68cf56-dl2vr"
Sep 09 02:57:10.134719 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:10.134698    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-621ac9-openshift-default-dockercfg-4gq6g\""
Sep 09 02:57:10.145456 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:10.145429    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-621ac9-openshift-default-549f68cf56-dl2vr"]
Sep 09 02:57:10.227180 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:10.227096    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72a3ba64-14d5-4c42-b59a-b70e1296f504-istiod-ca-cert\") pod \"e2e-ipp-b-621ac9-openshift-default-549f68cf56-dl2vr\" (UID: \"72a3ba64-14d5-4c42-b59a-b70e1296f504\") " pod="openshift-ingress/e2e-ipp-b-621ac9-openshift-default-549f68cf56-dl2vr"
Sep 09 02:57:10.227180 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:10.227154    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvglw\" (UniqueName: \"kubernetes.io/projected/72a3ba64-14d5-4c42-b59a-b70e1296f504-kube-api-access-jvglw\") pod \"e2e-ipp-b-621ac9-openshift-default-549f68cf56-dl2vr\" (UID: \"72a3ba64-14d5-4c42-b59a-b70e1296f504\") " pod="openshift-ingress/e2e-ipp-b-621ac9-openshift-default-549f68cf56-dl2vr"
Sep 09 02:57:10.227364 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:10.227198    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72a3ba64-14d5-4c42-b59a-b70e1296f504-workload-certs\") pod \"e2e-ipp-b-621ac9-openshift-default-549f68cf56-dl2vr\" (UID: \"72a3ba64-14d5-4c42-b59a-b70e1296f504\") " pod="openshift-ingress/e2e-ipp-b-621ac9-openshift-default-549f68cf56-dl2vr"
Sep 09 02:57:10.227364 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:10.227221    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72a3ba64-14d5-4c42-b59a-b70e1296f504-istio-envoy\") pod \"e2e-ipp-b-621ac9-openshift-default-549f68cf56-dl2vr\" (UID: \"72a3ba64-14d5-4c42-b59a-b70e1296f504\") " pod="openshift-ingress/e2e-ipp-b-621ac9-openshift-default-549f68cf56-dl2vr"
Sep 09 02:57:10.227364 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:10.227240    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72a3ba64-14d5-4c42-b59a-b70e1296f504-istio-podinfo\") pod \"e2e-ipp-b-621ac9-openshift-default-549f68cf56-dl2vr\" (UID: \"72a3ba64-14d5-4c42-b59a-b70e1296f504\") " pod="openshift-ingress/e2e-ipp-b-621ac9-openshift-default-549f68cf56-dl2vr"
Sep 09 02:57:10.227364 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:10.227263    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72a3ba64-14d5-4c42-b59a-b70e1296f504-credential-socket\") pod \"e2e-ipp-b-621ac9-openshift-default-549f68cf56-dl2vr\" (UID: \"72a3ba64-14d5-4c42-b59a-b70e1296f504\") " pod="openshift-ingress/e2e-ipp-b-621ac9-openshift-default-549f68cf56-dl2vr"
Sep 09 02:57:10.227364 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:10.227293    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72a3ba64-14d5-4c42-b59a-b70e1296f504-istio-data\") pod \"e2e-ipp-b-621ac9-openshift-default-549f68cf56-dl2vr\" (UID: \"72a3ba64-14d5-4c42-b59a-b70e1296f504\") " pod="openshift-ingress/e2e-ipp-b-621ac9-openshift-default-549f68cf56-dl2vr"
Sep 09 02:57:10.227364 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:10.227314    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72a3ba64-14d5-4c42-b59a-b70e1296f504-workload-socket\") pod \"e2e-ipp-b-621ac9-openshift-default-549f68cf56-dl2vr\" (UID: \"72a3ba64-14d5-4c42-b59a-b70e1296f504\") " pod="openshift-ingress/e2e-ipp-b-621ac9-openshift-default-549f68cf56-dl2vr"
Sep 09 02:57:10.227544 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:10.227429    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72a3ba64-14d5-4c42-b59a-b70e1296f504-istio-token\") pod \"e2e-ipp-b-621ac9-openshift-default-549f68cf56-dl2vr\" (UID: \"72a3ba64-14d5-4c42-b59a-b70e1296f504\") " pod="openshift-ingress/e2e-ipp-b-621ac9-openshift-default-549f68cf56-dl2vr"
Sep 09 02:57:10.328926 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:10.328888    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72a3ba64-14d5-4c42-b59a-b70e1296f504-istio-token\") pod \"e2e-ipp-b-621ac9-openshift-default-549f68cf56-dl2vr\" (UID: \"72a3ba64-14d5-4c42-b59a-b70e1296f504\") " pod="openshift-ingress/e2e-ipp-b-621ac9-openshift-default-549f68cf56-dl2vr"
Sep 09 02:57:10.329105 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:10.328934    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72a3ba64-14d5-4c42-b59a-b70e1296f504-istiod-ca-cert\") pod \"e2e-ipp-b-621ac9-openshift-default-549f68cf56-dl2vr\" (UID: \"72a3ba64-14d5-4c42-b59a-b70e1296f504\") " pod="openshift-ingress/e2e-ipp-b-621ac9-openshift-default-549f68cf56-dl2vr"
Sep 09 02:57:10.329105 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:10.328995    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvglw\" (UniqueName: \"kubernetes.io/projected/72a3ba64-14d5-4c42-b59a-b70e1296f504-kube-api-access-jvglw\") pod \"e2e-ipp-b-621ac9-openshift-default-549f68cf56-dl2vr\" (UID: \"72a3ba64-14d5-4c42-b59a-b70e1296f504\") " pod="openshift-ingress/e2e-ipp-b-621ac9-openshift-default-549f68cf56-dl2vr"
Sep 09 02:57:10.329105 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:10.329022    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72a3ba64-14d5-4c42-b59a-b70e1296f504-workload-certs\") pod \"e2e-ipp-b-621ac9-openshift-default-549f68cf56-dl2vr\" (UID: \"72a3ba64-14d5-4c42-b59a-b70e1296f504\") " pod="openshift-ingress/e2e-ipp-b-621ac9-openshift-default-549f68cf56-dl2vr"
Sep 09 02:57:10.329105 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:10.329042    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72a3ba64-14d5-4c42-b59a-b70e1296f504-istio-envoy\") pod \"e2e-ipp-b-621ac9-openshift-default-549f68cf56-dl2vr\" (UID: \"72a3ba64-14d5-4c42-b59a-b70e1296f504\") " pod="openshift-ingress/e2e-ipp-b-621ac9-openshift-default-549f68cf56-dl2vr"
Sep 09 02:57:10.329105 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:10.329072    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72a3ba64-14d5-4c42-b59a-b70e1296f504-istio-podinfo\") pod \"e2e-ipp-b-621ac9-openshift-default-549f68cf56-dl2vr\" (UID: \"72a3ba64-14d5-4c42-b59a-b70e1296f504\") " pod="openshift-ingress/e2e-ipp-b-621ac9-openshift-default-549f68cf56-dl2vr"
Sep 09 02:57:10.329105 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:10.329104    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72a3ba64-14d5-4c42-b59a-b70e1296f504-credential-socket\") pod \"e2e-ipp-b-621ac9-openshift-default-549f68cf56-dl2vr\" (UID: \"72a3ba64-14d5-4c42-b59a-b70e1296f504\") " pod="openshift-ingress/e2e-ipp-b-621ac9-openshift-default-549f68cf56-dl2vr"
Sep 09 02:57:10.329458 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:10.329132    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72a3ba64-14d5-4c42-b59a-b70e1296f504-istio-data\") pod \"e2e-ipp-b-621ac9-openshift-default-549f68cf56-dl2vr\" (UID: \"72a3ba64-14d5-4c42-b59a-b70e1296f504\") " pod="openshift-ingress/e2e-ipp-b-621ac9-openshift-default-549f68cf56-dl2vr"
Sep 09 02:57:10.329458 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:10.329160    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72a3ba64-14d5-4c42-b59a-b70e1296f504-workload-socket\") pod \"e2e-ipp-b-621ac9-openshift-default-549f68cf56-dl2vr\" (UID: \"72a3ba64-14d5-4c42-b59a-b70e1296f504\") " pod="openshift-ingress/e2e-ipp-b-621ac9-openshift-default-549f68cf56-dl2vr"
Sep 09 02:57:10.329567 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:10.329485    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72a3ba64-14d5-4c42-b59a-b70e1296f504-workload-certs\") pod \"e2e-ipp-b-621ac9-openshift-default-549f68cf56-dl2vr\" (UID: \"72a3ba64-14d5-4c42-b59a-b70e1296f504\") " pod="openshift-ingress/e2e-ipp-b-621ac9-openshift-default-549f68cf56-dl2vr"
Sep 09 02:57:10.329567 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:10.329523    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72a3ba64-14d5-4c42-b59a-b70e1296f504-credential-socket\") pod \"e2e-ipp-b-621ac9-openshift-default-549f68cf56-dl2vr\" (UID: \"72a3ba64-14d5-4c42-b59a-b70e1296f504\") " pod="openshift-ingress/e2e-ipp-b-621ac9-openshift-default-549f68cf56-dl2vr"
Sep 09 02:57:10.329657 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:10.329631    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72a3ba64-14d5-4c42-b59a-b70e1296f504-istio-data\") pod \"e2e-ipp-b-621ac9-openshift-default-549f68cf56-dl2vr\" (UID: \"72a3ba64-14d5-4c42-b59a-b70e1296f504\") " pod="openshift-ingress/e2e-ipp-b-621ac9-openshift-default-549f68cf56-dl2vr"
Sep 09 02:57:10.329728 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:10.329712    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72a3ba64-14d5-4c42-b59a-b70e1296f504-workload-socket\") pod \"e2e-ipp-b-621ac9-openshift-default-549f68cf56-dl2vr\" (UID: \"72a3ba64-14d5-4c42-b59a-b70e1296f504\") " pod="openshift-ingress/e2e-ipp-b-621ac9-openshift-default-549f68cf56-dl2vr"
Sep 09 02:57:10.329781 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:10.329758    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72a3ba64-14d5-4c42-b59a-b70e1296f504-istiod-ca-cert\") pod \"e2e-ipp-b-621ac9-openshift-default-549f68cf56-dl2vr\" (UID: \"72a3ba64-14d5-4c42-b59a-b70e1296f504\") " pod="openshift-ingress/e2e-ipp-b-621ac9-openshift-default-549f68cf56-dl2vr"
Sep 09 02:57:10.331583 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:10.331559    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72a3ba64-14d5-4c42-b59a-b70e1296f504-istio-envoy\") pod \"e2e-ipp-b-621ac9-openshift-default-549f68cf56-dl2vr\" (UID: \"72a3ba64-14d5-4c42-b59a-b70e1296f504\") " pod="openshift-ingress/e2e-ipp-b-621ac9-openshift-default-549f68cf56-dl2vr"
Sep 09 02:57:10.331707 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:10.331688    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72a3ba64-14d5-4c42-b59a-b70e1296f504-istio-podinfo\") pod \"e2e-ipp-b-621ac9-openshift-default-549f68cf56-dl2vr\" (UID: \"72a3ba64-14d5-4c42-b59a-b70e1296f504\") " pod="openshift-ingress/e2e-ipp-b-621ac9-openshift-default-549f68cf56-dl2vr"
Sep 09 02:57:10.337521 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:10.337497    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72a3ba64-14d5-4c42-b59a-b70e1296f504-istio-token\") pod \"e2e-ipp-b-621ac9-openshift-default-549f68cf56-dl2vr\" (UID: \"72a3ba64-14d5-4c42-b59a-b70e1296f504\") " pod="openshift-ingress/e2e-ipp-b-621ac9-openshift-default-549f68cf56-dl2vr"
Sep 09 02:57:10.337608 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:10.337575    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvglw\" (UniqueName: \"kubernetes.io/projected/72a3ba64-14d5-4c42-b59a-b70e1296f504-kube-api-access-jvglw\") pod \"e2e-ipp-b-621ac9-openshift-default-549f68cf56-dl2vr\" (UID: \"72a3ba64-14d5-4c42-b59a-b70e1296f504\") " pod="openshift-ingress/e2e-ipp-b-621ac9-openshift-default-549f68cf56-dl2vr"
Sep 09 02:57:10.444213 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:10.444173    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-621ac9-openshift-default-549f68cf56-dl2vr"
Sep 09 02:57:10.723701 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:10.723674    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-621ac9-openshift-default-549f68cf56-dl2vr"]
Sep 09 02:57:10.725427 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:57:10.725396    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72a3ba64_14d5_4c42_b59a_b70e1296f504.slice/crio-6a2d402d7196dd0a05893e60f84d2d3719c378426f8d860c2d7d24931e52436a WatchSource:0}: Error finding container 6a2d402d7196dd0a05893e60f84d2d3719c378426f8d860c2d7d24931e52436a: Status 404 returned error can't find the container with id 6a2d402d7196dd0a05893e60f84d2d3719c378426f8d860c2d7d24931e52436a
Sep 09 02:57:10.727777 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:10.727741    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 02:57:10.727851 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:10.727809    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 02:57:10.727851 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:10.727843    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 02:57:11.105527 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:11.105495    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-621ac9-openshift-default-549f68cf56-dl2vr" event={"ID":"72a3ba64-14d5-4c42-b59a-b70e1296f504","Type":"ContainerStarted","Data":"eae43e49b12736a2c2425cb562a6b984b6db4aa379939f253bdbeeff49780550"}
Sep 09 02:57:11.105930 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:11.105547    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-621ac9-openshift-default-549f68cf56-dl2vr" event={"ID":"72a3ba64-14d5-4c42-b59a-b70e1296f504","Type":"ContainerStarted","Data":"6a2d402d7196dd0a05893e60f84d2d3719c378426f8d860c2d7d24931e52436a"}
Sep 09 02:57:11.107910 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:11.107882    2570 patch_prober.go:28] interesting pod/e2e-shared-a-w2-d838aa-openshift-default-6989bbb4f6-gj56v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 09 02:57:11.108048 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:11.107941    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-d838aa-openshift-default-6989bbb4f6-gj56v" podUID="cb43df80-abb3-4a89-870a-dc6587abf0fa" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 02:57:11.125584 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:11.125541    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-621ac9-openshift-default-549f68cf56-dl2vr" podStartSLOduration=1.1255261029999999 podStartE2EDuration="1.125526103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:57:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:57:11.124365686 +0000 UTC m=+1613.378669324" watchObservedRunningTime="2026-09-09 02:57:11.125526103 +0000 UTC m=+1613.379829739"
Sep 09 02:57:11.222180 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:11.222155    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f5aebb-7ckh4"
Sep 09 02:57:11.348425 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:11.348401    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2e584dc-69ad-43db-9a53-e6be4f66640a-maas-api-service-ca\") pod \"c2e584dc-69ad-43db-9a53-e6be4f66640a\" (UID: \"c2e584dc-69ad-43db-9a53-e6be4f66640a\") "
Sep 09 02:57:11.348750 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:11.348726    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2e584dc-69ad-43db-9a53-e6be4f66640a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c2e584dc-69ad-43db-9a53-e6be4f66640a" (UID: "c2e584dc-69ad-43db-9a53-e6be4f66640a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 02:57:11.445237 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:11.445161    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-621ac9-openshift-default-549f68cf56-dl2vr"
Sep 09 02:57:11.449772 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:11.449750    2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2e584dc-69ad-43db-9a53-e6be4f66640a-maas-api-service-ca\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:57:11.449870 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:11.449793    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-621ac9-openshift-default-549f68cf56-dl2vr"
Sep 09 02:57:12.111531 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:12.111498    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f5aebb-7ckh4"
Sep 09 02:57:12.111899 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:12.111529    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f5aebb-7ckh4" event={"ID":"c2e584dc-69ad-43db-9a53-e6be4f66640a","Type":"ContainerDied","Data":"ba49c5944a1f3962162070c4bd911522885a56c5dd60bff9a30b000e4c6695c3"}
Sep 09 02:57:12.111899 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:12.111581    2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba49c5944a1f3962162070c4bd911522885a56c5dd60bff9a30b000e4c6695c3"
Sep 09 02:57:12.111899 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:12.111876    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-621ac9-openshift-default-549f68cf56-dl2vr"
Sep 09 02:57:12.112773 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:12.112758    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-621ac9-openshift-default-549f68cf56-dl2vr"
Sep 09 02:57:13.536604 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:13.536577    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-d838aa-openshift-default-6989bbb4f6-gj56v"
Sep 09 02:57:13.670200 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:13.670171    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4sll9\" (UniqueName: \"kubernetes.io/projected/cb43df80-abb3-4a89-870a-dc6587abf0fa-kube-api-access-4sll9\") pod \"cb43df80-abb3-4a89-870a-dc6587abf0fa\" (UID: \"cb43df80-abb3-4a89-870a-dc6587abf0fa\") "
Sep 09 02:57:13.670339 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:13.670207    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb43df80-abb3-4a89-870a-dc6587abf0fa-istio-token\") pod \"cb43df80-abb3-4a89-870a-dc6587abf0fa\" (UID: \"cb43df80-abb3-4a89-870a-dc6587abf0fa\") "
Sep 09 02:57:13.670339 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:13.670240    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb43df80-abb3-4a89-870a-dc6587abf0fa-istiod-ca-cert\") pod \"cb43df80-abb3-4a89-870a-dc6587abf0fa\" (UID: \"cb43df80-abb3-4a89-870a-dc6587abf0fa\") "
Sep 09 02:57:13.670339 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:13.670259    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb43df80-abb3-4a89-870a-dc6587abf0fa-workload-socket\") pod \"cb43df80-abb3-4a89-870a-dc6587abf0fa\" (UID: \"cb43df80-abb3-4a89-870a-dc6587abf0fa\") "
Sep 09 02:57:13.670339 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:13.670335    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb43df80-abb3-4a89-870a-dc6587abf0fa-istio-envoy\") pod \"cb43df80-abb3-4a89-870a-dc6587abf0fa\" (UID: \"cb43df80-abb3-4a89-870a-dc6587abf0fa\") "
Sep 09 02:57:13.670527 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:13.670367    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb43df80-abb3-4a89-870a-dc6587abf0fa-istio-podinfo\") pod \"cb43df80-abb3-4a89-870a-dc6587abf0fa\" (UID: \"cb43df80-abb3-4a89-870a-dc6587abf0fa\") "
Sep 09 02:57:13.670527 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:13.670390    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb43df80-abb3-4a89-870a-dc6587abf0fa-workload-certs\") pod \"cb43df80-abb3-4a89-870a-dc6587abf0fa\" (UID: \"cb43df80-abb3-4a89-870a-dc6587abf0fa\") "
Sep 09 02:57:13.670527 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:13.670412    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb43df80-abb3-4a89-870a-dc6587abf0fa-istio-data\") pod \"cb43df80-abb3-4a89-870a-dc6587abf0fa\" (UID: \"cb43df80-abb3-4a89-870a-dc6587abf0fa\") "
Sep 09 02:57:13.670527 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:13.670523    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb43df80-abb3-4a89-870a-dc6587abf0fa-credential-socket\") pod \"cb43df80-abb3-4a89-870a-dc6587abf0fa\" (UID: \"cb43df80-abb3-4a89-870a-dc6587abf0fa\") "
Sep 09 02:57:13.670746 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:13.670566    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb43df80-abb3-4a89-870a-dc6587abf0fa-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "cb43df80-abb3-4a89-870a-dc6587abf0fa" (UID: "cb43df80-abb3-4a89-870a-dc6587abf0fa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 02:57:13.670746 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:13.670620    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb43df80-abb3-4a89-870a-dc6587abf0fa-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "cb43df80-abb3-4a89-870a-dc6587abf0fa" (UID: "cb43df80-abb3-4a89-870a-dc6587abf0fa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 02:57:13.670746 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:13.670717    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb43df80-abb3-4a89-870a-dc6587abf0fa-istio-data" (OuterVolumeSpecName: "istio-data") pod "cb43df80-abb3-4a89-870a-dc6587abf0fa" (UID: "cb43df80-abb3-4a89-870a-dc6587abf0fa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 02:57:13.670746 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:13.670730    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb43df80-abb3-4a89-870a-dc6587abf0fa-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "cb43df80-abb3-4a89-870a-dc6587abf0fa" (UID: "cb43df80-abb3-4a89-870a-dc6587abf0fa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 02:57:13.670955 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:13.670866    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb43df80-abb3-4a89-870a-dc6587abf0fa-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "cb43df80-abb3-4a89-870a-dc6587abf0fa" (UID: "cb43df80-abb3-4a89-870a-dc6587abf0fa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 02:57:13.671199 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:13.671178    2570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb43df80-abb3-4a89-870a-dc6587abf0fa-istiod-ca-cert\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:57:13.671296 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:13.671205    2570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb43df80-abb3-4a89-870a-dc6587abf0fa-workload-socket\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:57:13.671296 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:13.671220    2570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb43df80-abb3-4a89-870a-dc6587abf0fa-workload-certs\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:57:13.671296 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:13.671237    2570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb43df80-abb3-4a89-870a-dc6587abf0fa-istio-data\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:57:13.671296 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:13.671251    2570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb43df80-abb3-4a89-870a-dc6587abf0fa-credential-socket\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:57:13.672507 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:13.672484    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb43df80-abb3-4a89-870a-dc6587abf0fa-istio-token" (OuterVolumeSpecName: "istio-token") pod "cb43df80-abb3-4a89-870a-dc6587abf0fa" (UID: "cb43df80-abb3-4a89-870a-dc6587abf0fa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 02:57:13.672594 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:13.672520    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb43df80-abb3-4a89-870a-dc6587abf0fa-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "cb43df80-abb3-4a89-870a-dc6587abf0fa" (UID: "cb43df80-abb3-4a89-870a-dc6587abf0fa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 02:57:13.672594 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:13.672523    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cb43df80-abb3-4a89-870a-dc6587abf0fa-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "cb43df80-abb3-4a89-870a-dc6587abf0fa" (UID: "cb43df80-abb3-4a89-870a-dc6587abf0fa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 02:57:13.672791 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:13.672774    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb43df80-abb3-4a89-870a-dc6587abf0fa-kube-api-access-4sll9" (OuterVolumeSpecName: "kube-api-access-4sll9") pod "cb43df80-abb3-4a89-870a-dc6587abf0fa" (UID: "cb43df80-abb3-4a89-870a-dc6587abf0fa"). InnerVolumeSpecName "kube-api-access-4sll9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 02:57:13.772075 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:13.772051    2570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb43df80-abb3-4a89-870a-dc6587abf0fa-istio-token\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:57:13.772075 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:13.772073    2570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb43df80-abb3-4a89-870a-dc6587abf0fa-istio-envoy\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:57:13.772200 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:13.772081    2570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb43df80-abb3-4a89-870a-dc6587abf0fa-istio-podinfo\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:57:13.772200 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:13.772090    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4sll9\" (UniqueName: \"kubernetes.io/projected/cb43df80-abb3-4a89-870a-dc6587abf0fa-kube-api-access-4sll9\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:57:14.123787 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:14.123748    2570 generic.go:356] "Generic (PLEG): container finished" podID="cb43df80-abb3-4a89-870a-dc6587abf0fa" containerID="e825f79926523f2902e7cbbf9b3068fd4841d3ed4ed7aec0f75721deff0d4345" exitCode=0
Sep 09 02:57:14.123964 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:14.123798    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-d838aa-openshift-default-6989bbb4f6-gj56v"
Sep 09 02:57:14.123964 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:14.123834    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-d838aa-openshift-default-6989bbb4f6-gj56v" event={"ID":"cb43df80-abb3-4a89-870a-dc6587abf0fa","Type":"ContainerDied","Data":"e825f79926523f2902e7cbbf9b3068fd4841d3ed4ed7aec0f75721deff0d4345"}
Sep 09 02:57:14.123964 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:14.123888    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-d838aa-openshift-default-6989bbb4f6-gj56v" event={"ID":"cb43df80-abb3-4a89-870a-dc6587abf0fa","Type":"ContainerDied","Data":"c6066d8369142baa3c571400667a39be1ed1706020417d9c427f3234e7490666"}
Sep 09 02:57:14.123964 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:14.123911    2570 scope.go:117] "RemoveContainer" containerID="e825f79926523f2902e7cbbf9b3068fd4841d3ed4ed7aec0f75721deff0d4345"
Sep 09 02:57:14.133969 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:14.133952    2570 scope.go:117] "RemoveContainer" containerID="53bf0dd407af569b1c526edc5e56a2d679ba6b239bd737d75637550054786114"
Sep 09 02:57:14.144769 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:14.144747    2570 scope.go:117] "RemoveContainer" containerID="e825f79926523f2902e7cbbf9b3068fd4841d3ed4ed7aec0f75721deff0d4345"
Sep 09 02:57:14.145116 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:57:14.145088    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e825f79926523f2902e7cbbf9b3068fd4841d3ed4ed7aec0f75721deff0d4345\": container with ID starting with e825f79926523f2902e7cbbf9b3068fd4841d3ed4ed7aec0f75721deff0d4345 not found: ID does not exist" containerID="e825f79926523f2902e7cbbf9b3068fd4841d3ed4ed7aec0f75721deff0d4345"
Sep 09 02:57:14.145251 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:14.145129    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e825f79926523f2902e7cbbf9b3068fd4841d3ed4ed7aec0f75721deff0d4345"} err="failed to get container status \"e825f79926523f2902e7cbbf9b3068fd4841d3ed4ed7aec0f75721deff0d4345\": rpc error: code = NotFound desc = could not find container \"e825f79926523f2902e7cbbf9b3068fd4841d3ed4ed7aec0f75721deff0d4345\": container with ID starting with e825f79926523f2902e7cbbf9b3068fd4841d3ed4ed7aec0f75721deff0d4345 not found: ID does not exist"
Sep 09 02:57:14.145251 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:14.145152    2570 scope.go:117] "RemoveContainer" containerID="53bf0dd407af569b1c526edc5e56a2d679ba6b239bd737d75637550054786114"
Sep 09 02:57:14.145441 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:57:14.145420    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53bf0dd407af569b1c526edc5e56a2d679ba6b239bd737d75637550054786114\": container with ID starting with 53bf0dd407af569b1c526edc5e56a2d679ba6b239bd737d75637550054786114 not found: ID does not exist" containerID="53bf0dd407af569b1c526edc5e56a2d679ba6b239bd737d75637550054786114"
Sep 09 02:57:14.145501 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:14.145447    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53bf0dd407af569b1c526edc5e56a2d679ba6b239bd737d75637550054786114"} err="failed to get container status \"53bf0dd407af569b1c526edc5e56a2d679ba6b239bd737d75637550054786114\": rpc error: code = NotFound desc = could not find container \"53bf0dd407af569b1c526edc5e56a2d679ba6b239bd737d75637550054786114\": container with ID starting with 53bf0dd407af569b1c526edc5e56a2d679ba6b239bd737d75637550054786114 not found: ID does not exist"
Sep 09 02:57:14.147921 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:14.147903    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-d838aa-openshift-default-6989bbb4f6-gj56v"]
Sep 09 02:57:14.151323 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:14.151302    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-d838aa-openshift-default-6989bbb4f6-gj56v"]
Sep 09 02:57:14.223107 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:14.223086    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb43df80-abb3-4a89-870a-dc6587abf0fa" path="/var/lib/kubelet/pods/cb43df80-abb3-4a89-870a-dc6587abf0fa/volumes"
Sep 09 02:57:19.632303 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:19.632266    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f5aebb-7ckh4"]
Sep 09 02:57:19.635765 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:19.635735    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f5aebb-7ckh4"]
Sep 09 02:57:20.223296 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:20.223263    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2e584dc-69ad-43db-9a53-e6be4f66640a" path="/var/lib/kubelet/pods/c2e584dc-69ad-43db-9a53-e6be4f66640a/volumes"
Sep 09 02:57:20.456176 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:20.456140    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-99qjf"]
Sep 09 02:57:25.950744 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:25.950705    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-f5aebb-openshift-default-7c76c475d-r9lxr"]
Sep 09 02:57:25.951637 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:25.951588    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-f5aebb-openshift-default-7c76c475d-r9lxr" podUID="7463a0d2-3ecb-479f-b47b-b4b5e8190396" containerName="istio-proxy" containerID="cri-o://f8533d367700abbbe51870e99aa0628352be9d5ccd943215c616f30dd05d07fb" gracePeriod=30
Sep 09 02:57:30.448319 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:30.448276    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-99qjf"]
Sep 09 02:57:31.220107 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:31.220077    2570 generic.go:356] "Generic (PLEG): container finished" podID="7463a0d2-3ecb-479f-b47b-b4b5e8190396" containerID="f8533d367700abbbe51870e99aa0628352be9d5ccd943215c616f30dd05d07fb" exitCode=0
Sep 09 02:57:31.220272 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:31.220145    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-f5aebb-openshift-default-7c76c475d-r9lxr" event={"ID":"7463a0d2-3ecb-479f-b47b-b4b5e8190396","Type":"ContainerDied","Data":"f8533d367700abbbe51870e99aa0628352be9d5ccd943215c616f30dd05d07fb"}
Sep 09 02:57:31.220272 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:31.220196    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-f5aebb-openshift-default-7c76c475d-r9lxr" event={"ID":"7463a0d2-3ecb-479f-b47b-b4b5e8190396","Type":"ContainerDied","Data":"6634994ba1653b5e02bb3998490ba62aac410d532ef5ab79fcd9f665548f9fa3"}
Sep 09 02:57:31.220272 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:31.220216    2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6634994ba1653b5e02bb3998490ba62aac410d532ef5ab79fcd9f665548f9fa3"
Sep 09 02:57:31.220272 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:31.220232    2570 scope.go:117] "RemoveContainer" containerID="99723178ceaf97f1e67cf2df749aee119e4fab4187313d3023476ba2c50c5e11"
Sep 09 02:57:31.221880 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:31.221858    2570 generic.go:356] "Generic (PLEG): container finished" podID="72a3ba64-14d5-4c42-b59a-b70e1296f504" containerID="eae43e49b12736a2c2425cb562a6b984b6db4aa379939f253bdbeeff49780550" exitCode=0
Sep 09 02:57:31.222001 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:31.221890    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-621ac9-openshift-default-549f68cf56-dl2vr" event={"ID":"72a3ba64-14d5-4c42-b59a-b70e1296f504","Type":"ContainerDied","Data":"eae43e49b12736a2c2425cb562a6b984b6db4aa379939f253bdbeeff49780550"}
Sep 09 02:57:31.222386 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:31.222363    2570 scope.go:117] "RemoveContainer" containerID="eae43e49b12736a2c2425cb562a6b984b6db4aa379939f253bdbeeff49780550"
Sep 09 02:57:31.223501 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:31.223474    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 02:57:31.223617 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:31.223552    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 02:57:31.223617 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:31.223599    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 02:57:31.225539 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:31.225515    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-f5aebb-openshift-default-7c76c475d-r9lxr"
Sep 09 02:57:31.244518 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:31.244489    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7463a0d2-3ecb-479f-b47b-b4b5e8190396-istio-podinfo\") pod \"7463a0d2-3ecb-479f-b47b-b4b5e8190396\" (UID: \"7463a0d2-3ecb-479f-b47b-b4b5e8190396\") "
Sep 09 02:57:31.244626 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:31.244530    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7463a0d2-3ecb-479f-b47b-b4b5e8190396-istio-data\") pod \"7463a0d2-3ecb-479f-b47b-b4b5e8190396\" (UID: \"7463a0d2-3ecb-479f-b47b-b4b5e8190396\") "
Sep 09 02:57:31.244626 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:31.244551    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7463a0d2-3ecb-479f-b47b-b4b5e8190396-workload-socket\") pod \"7463a0d2-3ecb-479f-b47b-b4b5e8190396\" (UID: \"7463a0d2-3ecb-479f-b47b-b4b5e8190396\") "
Sep 09 02:57:31.244626 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:31.244579    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7463a0d2-3ecb-479f-b47b-b4b5e8190396-istio-envoy\") pod \"7463a0d2-3ecb-479f-b47b-b4b5e8190396\" (UID: \"7463a0d2-3ecb-479f-b47b-b4b5e8190396\") "
Sep 09 02:57:31.244792 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:31.244658    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7463a0d2-3ecb-479f-b47b-b4b5e8190396-istio-token\") pod \"7463a0d2-3ecb-479f-b47b-b4b5e8190396\" (UID: \"7463a0d2-3ecb-479f-b47b-b4b5e8190396\") "
Sep 09 02:57:31.244792 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:31.244686    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7463a0d2-3ecb-479f-b47b-b4b5e8190396-credential-socket\") pod \"7463a0d2-3ecb-479f-b47b-b4b5e8190396\" (UID: \"7463a0d2-3ecb-479f-b47b-b4b5e8190396\") "
Sep 09 02:57:31.244792 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:31.244718    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wcvcw\" (UniqueName: \"kubernetes.io/projected/7463a0d2-3ecb-479f-b47b-b4b5e8190396-kube-api-access-wcvcw\") pod \"7463a0d2-3ecb-479f-b47b-b4b5e8190396\" (UID: \"7463a0d2-3ecb-479f-b47b-b4b5e8190396\") "
Sep 09 02:57:31.244792 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:31.244766    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7463a0d2-3ecb-479f-b47b-b4b5e8190396-workload-certs\") pod \"7463a0d2-3ecb-479f-b47b-b4b5e8190396\" (UID: \"7463a0d2-3ecb-479f-b47b-b4b5e8190396\") "
Sep 09 02:57:31.244792 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:31.244784    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7463a0d2-3ecb-479f-b47b-b4b5e8190396-istio-data" (OuterVolumeSpecName: "istio-data") pod "7463a0d2-3ecb-479f-b47b-b4b5e8190396" (UID: "7463a0d2-3ecb-479f-b47b-b4b5e8190396"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 02:57:31.245077 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:31.244789    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7463a0d2-3ecb-479f-b47b-b4b5e8190396-istiod-ca-cert\") pod \"7463a0d2-3ecb-479f-b47b-b4b5e8190396\" (UID: \"7463a0d2-3ecb-479f-b47b-b4b5e8190396\") "
Sep 09 02:57:31.245134 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:31.245089    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7463a0d2-3ecb-479f-b47b-b4b5e8190396-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7463a0d2-3ecb-479f-b47b-b4b5e8190396" (UID: "7463a0d2-3ecb-479f-b47b-b4b5e8190396"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 02:57:31.245867 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:31.245223    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7463a0d2-3ecb-479f-b47b-b4b5e8190396-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7463a0d2-3ecb-479f-b47b-b4b5e8190396" (UID: "7463a0d2-3ecb-479f-b47b-b4b5e8190396"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 02:57:31.245867 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:31.245251    2570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7463a0d2-3ecb-479f-b47b-b4b5e8190396-istiod-ca-cert\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:57:31.245867 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:31.245269    2570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7463a0d2-3ecb-479f-b47b-b4b5e8190396-istio-data\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:57:31.245867 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:31.245413    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7463a0d2-3ecb-479f-b47b-b4b5e8190396-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7463a0d2-3ecb-479f-b47b-b4b5e8190396" (UID: "7463a0d2-3ecb-479f-b47b-b4b5e8190396"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 02:57:31.245867 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:31.245526    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7463a0d2-3ecb-479f-b47b-b4b5e8190396-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7463a0d2-3ecb-479f-b47b-b4b5e8190396" (UID: "7463a0d2-3ecb-479f-b47b-b4b5e8190396"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 02:57:31.247444 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:31.247402    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7463a0d2-3ecb-479f-b47b-b4b5e8190396-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7463a0d2-3ecb-479f-b47b-b4b5e8190396" (UID: "7463a0d2-3ecb-479f-b47b-b4b5e8190396"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 02:57:31.247827 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:31.247802    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7463a0d2-3ecb-479f-b47b-b4b5e8190396-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7463a0d2-3ecb-479f-b47b-b4b5e8190396" (UID: "7463a0d2-3ecb-479f-b47b-b4b5e8190396"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 02:57:31.248943 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:31.248919    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7463a0d2-3ecb-479f-b47b-b4b5e8190396-istio-token" (OuterVolumeSpecName: "istio-token") pod "7463a0d2-3ecb-479f-b47b-b4b5e8190396" (UID: "7463a0d2-3ecb-479f-b47b-b4b5e8190396"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 02:57:31.249450 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:31.249430    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7463a0d2-3ecb-479f-b47b-b4b5e8190396-kube-api-access-wcvcw" (OuterVolumeSpecName: "kube-api-access-wcvcw") pod "7463a0d2-3ecb-479f-b47b-b4b5e8190396" (UID: "7463a0d2-3ecb-479f-b47b-b4b5e8190396"). InnerVolumeSpecName "kube-api-access-wcvcw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 02:57:31.346808 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:31.346769    2570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7463a0d2-3ecb-479f-b47b-b4b5e8190396-istio-token\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:57:31.346808 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:31.346811    2570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7463a0d2-3ecb-479f-b47b-b4b5e8190396-credential-socket\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:57:31.347012 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:31.346829    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcvcw\" (UniqueName: \"kubernetes.io/projected/7463a0d2-3ecb-479f-b47b-b4b5e8190396-kube-api-access-wcvcw\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:57:31.347012 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:31.346843    2570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7463a0d2-3ecb-479f-b47b-b4b5e8190396-workload-certs\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:57:31.347012 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:31.346858    2570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7463a0d2-3ecb-479f-b47b-b4b5e8190396-istio-podinfo\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:57:31.347012 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:31.346871    2570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7463a0d2-3ecb-479f-b47b-b4b5e8190396-workload-socket\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:57:31.347012 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:31.346884    2570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7463a0d2-3ecb-479f-b47b-b4b5e8190396-istio-envoy\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:57:32.235706 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:32.235643    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-621ac9-openshift-default-549f68cf56-dl2vr" event={"ID":"72a3ba64-14d5-4c42-b59a-b70e1296f504","Type":"ContainerStarted","Data":"c99421f21bfd42fc7ead7bc143bc03314d67e7cd00e23731916e267050876cba"}
Sep 09 02:57:32.237318 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:32.237300    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-f5aebb-openshift-default-7c76c475d-r9lxr"
Sep 09 02:57:32.305221 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:32.305188    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-f5aebb-openshift-default-7c76c475d-r9lxr"]
Sep 09 02:57:32.309365 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:32.309328    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-f5aebb-openshift-default-7c76c475d-r9lxr"]
Sep 09 02:57:32.444839 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:32.444800    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-621ac9-openshift-default-549f68cf56-dl2vr"
Sep 09 02:57:32.445275 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:32.445240    2570 patch_prober.go:28] interesting pod/e2e-ipp-b-621ac9-openshift-default-549f68cf56-dl2vr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Sep 09 02:57:32.445384 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:32.445324    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-621ac9-openshift-default-549f68cf56-dl2vr" podUID="72a3ba64-14d5-4c42-b59a-b70e1296f504" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Sep 09 02:57:33.445207 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:33.445171    2570 patch_prober.go:28] interesting pod/e2e-ipp-b-621ac9-openshift-default-549f68cf56-dl2vr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Sep 09 02:57:33.445776 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:33.445745    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-621ac9-openshift-default-549f68cf56-dl2vr" podUID="72a3ba64-14d5-4c42-b59a-b70e1296f504" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Sep 09 02:57:34.223133 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:34.223092    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7463a0d2-3ecb-479f-b47b-b4b5e8190396" path="/var/lib/kubelet/pods/7463a0d2-3ecb-479f-b47b-b4b5e8190396/volumes"
Sep 09 02:57:34.444861 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:34.444824    2570 patch_prober.go:28] interesting pod/e2e-ipp-b-621ac9-openshift-default-549f68cf56-dl2vr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Sep 09 02:57:34.445067 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:34.444906    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-621ac9-openshift-default-549f68cf56-dl2vr" podUID="72a3ba64-14d5-4c42-b59a-b70e1296f504" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Sep 09 02:57:35.445350 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:35.445312    2570 patch_prober.go:28] interesting pod/e2e-ipp-b-621ac9-openshift-default-549f68cf56-dl2vr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Sep 09 02:57:35.445832 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:35.445383    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-621ac9-openshift-default-549f68cf56-dl2vr" podUID="72a3ba64-14d5-4c42-b59a-b70e1296f504" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Sep 09 02:57:36.444878 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:36.444838    2570 patch_prober.go:28] interesting pod/e2e-ipp-b-621ac9-openshift-default-549f68cf56-dl2vr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Sep 09 02:57:36.445206 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:36.445170    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-621ac9-openshift-default-549f68cf56-dl2vr" podUID="72a3ba64-14d5-4c42-b59a-b70e1296f504" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Sep 09 02:57:37.445150 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:37.445110    2570 patch_prober.go:28] interesting pod/e2e-ipp-b-621ac9-openshift-default-549f68cf56-dl2vr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Sep 09 02:57:37.445521 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:37.445182    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-621ac9-openshift-default-549f68cf56-dl2vr" podUID="72a3ba64-14d5-4c42-b59a-b70e1296f504" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Sep 09 02:57:38.449509 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:38.449479    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-621ac9-openshift-default-549f68cf56-dl2vr"
Sep 09 02:57:38.449938 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:38.449755    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-621ac9-openshift-default-549f68cf56-dl2vr"
Sep 09 02:57:38.450464 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:38.450445    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-621ac9-openshift-default-549f68cf56-dl2vr"
Sep 09 02:57:46.443268 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:46.443232    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-2085ee/ipp-route-2085ee-kserve-7b5c64ddc-7d8tj"]
Sep 09 02:57:46.443834 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:46.443774    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2e584dc-69ad-43db-9a53-e6be4f66640a" containerName="revoke-keys"
Sep 09 02:57:46.443834 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:46.443794    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2e584dc-69ad-43db-9a53-e6be4f66640a" containerName="revoke-keys"
Sep 09 02:57:46.443834 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:46.443812    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb43df80-abb3-4a89-870a-dc6587abf0fa" containerName="istio-proxy"
Sep 09 02:57:46.443834 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:46.443820    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb43df80-abb3-4a89-870a-dc6587abf0fa" containerName="istio-proxy"
Sep 09 02:57:46.443834 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:46.443829    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb43df80-abb3-4a89-870a-dc6587abf0fa" containerName="istio-proxy"
Sep 09 02:57:46.444117 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:46.443838    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb43df80-abb3-4a89-870a-dc6587abf0fa" containerName="istio-proxy"
Sep 09 02:57:46.444117 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:46.443853    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7463a0d2-3ecb-479f-b47b-b4b5e8190396" containerName="istio-proxy"
Sep 09 02:57:46.444117 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:46.443863    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="7463a0d2-3ecb-479f-b47b-b4b5e8190396" containerName="istio-proxy"
Sep 09 02:57:46.444117 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:46.443962    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb43df80-abb3-4a89-870a-dc6587abf0fa" containerName="istio-proxy"
Sep 09 02:57:46.444117 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:46.443992    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2e584dc-69ad-43db-9a53-e6be4f66640a" containerName="revoke-keys"
Sep 09 02:57:46.444117 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:46.444005    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb43df80-abb3-4a89-870a-dc6587abf0fa" containerName="istio-proxy"
Sep 09 02:57:46.444117 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:46.444016    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="7463a0d2-3ecb-479f-b47b-b4b5e8190396" containerName="istio-proxy"
Sep 09 02:57:46.444117 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:46.444023    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="7463a0d2-3ecb-479f-b47b-b4b5e8190396" containerName="istio-proxy"
Sep 09 02:57:46.448860 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:46.448826    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-2085ee/ipp-route-2085ee-kserve-7b5c64ddc-7d8tj"
Sep 09 02:57:46.452999 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:46.452964    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-2085ee\"/\"kube-root-ca.crt\""
Sep 09 02:57:46.453292 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:46.452965    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-2085ee\"/\"default-dockercfg-pf7jb\""
Sep 09 02:57:46.453292 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:46.453071    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-2085ee\"/\"ipp-route-2085ee-kserve-self-signed-certs\""
Sep 09 02:57:46.453543 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:46.453482    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-2085ee\"/\"openshift-service-ca.crt\""
Sep 09 02:57:46.455704 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:46.455679    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-2085ee/ipp-route-2085ee-kserve-7b5c64ddc-7d8tj"]
Sep 09 02:57:46.491672 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:46.491645    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77b51174-a926-4354-a640-df1878956023-tmp-dir\") pod \"ipp-route-2085ee-kserve-7b5c64ddc-7d8tj\" (UID: \"77b51174-a926-4354-a640-df1878956023\") " pod="ai-tenant-e2e-ipp-a-2085ee/ipp-route-2085ee-kserve-7b5c64ddc-7d8tj"
Sep 09 02:57:46.491843 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:46.491682    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77b51174-a926-4354-a640-df1878956023-dshm\") pod \"ipp-route-2085ee-kserve-7b5c64ddc-7d8tj\" (UID: \"77b51174-a926-4354-a640-df1878956023\") " pod="ai-tenant-e2e-ipp-a-2085ee/ipp-route-2085ee-kserve-7b5c64ddc-7d8tj"
Sep 09 02:57:46.491843 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:46.491785    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77b51174-a926-4354-a640-df1878956023-home\") pod \"ipp-route-2085ee-kserve-7b5c64ddc-7d8tj\" (UID: \"77b51174-a926-4354-a640-df1878956023\") " pod="ai-tenant-e2e-ipp-a-2085ee/ipp-route-2085ee-kserve-7b5c64ddc-7d8tj"
Sep 09 02:57:46.491843 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:46.491827    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77b51174-a926-4354-a640-df1878956023-tls-certs\") pod \"ipp-route-2085ee-kserve-7b5c64ddc-7d8tj\" (UID: \"77b51174-a926-4354-a640-df1878956023\") " pod="ai-tenant-e2e-ipp-a-2085ee/ipp-route-2085ee-kserve-7b5c64ddc-7d8tj"
Sep 09 02:57:46.491954 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:46.491861    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khtgd\" (UniqueName: \"kubernetes.io/projected/77b51174-a926-4354-a640-df1878956023-kube-api-access-khtgd\") pod \"ipp-route-2085ee-kserve-7b5c64ddc-7d8tj\" (UID: \"77b51174-a926-4354-a640-df1878956023\") " pod="ai-tenant-e2e-ipp-a-2085ee/ipp-route-2085ee-kserve-7b5c64ddc-7d8tj"
Sep 09 02:57:46.491954 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:46.491937    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77b51174-a926-4354-a640-df1878956023-model-cache\") pod \"ipp-route-2085ee-kserve-7b5c64ddc-7d8tj\" (UID: \"77b51174-a926-4354-a640-df1878956023\") " pod="ai-tenant-e2e-ipp-a-2085ee/ipp-route-2085ee-kserve-7b5c64ddc-7d8tj"
Sep 09 02:57:46.593000 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:46.592946    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77b51174-a926-4354-a640-df1878956023-tmp-dir\") pod \"ipp-route-2085ee-kserve-7b5c64ddc-7d8tj\" (UID: \"77b51174-a926-4354-a640-df1878956023\") " pod="ai-tenant-e2e-ipp-a-2085ee/ipp-route-2085ee-kserve-7b5c64ddc-7d8tj"
Sep 09 02:57:46.593000 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:46.592999    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77b51174-a926-4354-a640-df1878956023-dshm\") pod \"ipp-route-2085ee-kserve-7b5c64ddc-7d8tj\" (UID: \"77b51174-a926-4354-a640-df1878956023\") " pod="ai-tenant-e2e-ipp-a-2085ee/ipp-route-2085ee-kserve-7b5c64ddc-7d8tj"
Sep 09 02:57:46.593210 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:46.593036    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77b51174-a926-4354-a640-df1878956023-home\") pod \"ipp-route-2085ee-kserve-7b5c64ddc-7d8tj\" (UID: \"77b51174-a926-4354-a640-df1878956023\") " pod="ai-tenant-e2e-ipp-a-2085ee/ipp-route-2085ee-kserve-7b5c64ddc-7d8tj"
Sep 09 02:57:46.593210 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:46.593060    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77b51174-a926-4354-a640-df1878956023-tls-certs\") pod \"ipp-route-2085ee-kserve-7b5c64ddc-7d8tj\" (UID: \"77b51174-a926-4354-a640-df1878956023\") " pod="ai-tenant-e2e-ipp-a-2085ee/ipp-route-2085ee-kserve-7b5c64ddc-7d8tj"
Sep 09 02:57:46.593210 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:46.593080    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khtgd\" (UniqueName: \"kubernetes.io/projected/77b51174-a926-4354-a640-df1878956023-kube-api-access-khtgd\") pod \"ipp-route-2085ee-kserve-7b5c64ddc-7d8tj\" (UID: \"77b51174-a926-4354-a640-df1878956023\") " pod="ai-tenant-e2e-ipp-a-2085ee/ipp-route-2085ee-kserve-7b5c64ddc-7d8tj"
Sep 09 02:57:46.593210 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:46.593136    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77b51174-a926-4354-a640-df1878956023-model-cache\") pod \"ipp-route-2085ee-kserve-7b5c64ddc-7d8tj\" (UID: \"77b51174-a926-4354-a640-df1878956023\") " pod="ai-tenant-e2e-ipp-a-2085ee/ipp-route-2085ee-kserve-7b5c64ddc-7d8tj"
Sep 09 02:57:46.593400 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:46.593376    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77b51174-a926-4354-a640-df1878956023-tmp-dir\") pod \"ipp-route-2085ee-kserve-7b5c64ddc-7d8tj\" (UID: \"77b51174-a926-4354-a640-df1878956023\") " pod="ai-tenant-e2e-ipp-a-2085ee/ipp-route-2085ee-kserve-7b5c64ddc-7d8tj"
Sep 09 02:57:46.593454 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:46.593387    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77b51174-a926-4354-a640-df1878956023-home\") pod \"ipp-route-2085ee-kserve-7b5c64ddc-7d8tj\" (UID: \"77b51174-a926-4354-a640-df1878956023\") " pod="ai-tenant-e2e-ipp-a-2085ee/ipp-route-2085ee-kserve-7b5c64ddc-7d8tj"
Sep 09 02:57:46.593500 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:46.593456    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77b51174-a926-4354-a640-df1878956023-model-cache\") pod \"ipp-route-2085ee-kserve-7b5c64ddc-7d8tj\" (UID: \"77b51174-a926-4354-a640-df1878956023\") " pod="ai-tenant-e2e-ipp-a-2085ee/ipp-route-2085ee-kserve-7b5c64ddc-7d8tj"
Sep 09 02:57:46.595393 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:46.595371    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77b51174-a926-4354-a640-df1878956023-dshm\") pod \"ipp-route-2085ee-kserve-7b5c64ddc-7d8tj\" (UID: \"77b51174-a926-4354-a640-df1878956023\") " pod="ai-tenant-e2e-ipp-a-2085ee/ipp-route-2085ee-kserve-7b5c64ddc-7d8tj"
Sep 09 02:57:46.595576 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:46.595557    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77b51174-a926-4354-a640-df1878956023-tls-certs\") pod \"ipp-route-2085ee-kserve-7b5c64ddc-7d8tj\" (UID: \"77b51174-a926-4354-a640-df1878956023\") " pod="ai-tenant-e2e-ipp-a-2085ee/ipp-route-2085ee-kserve-7b5c64ddc-7d8tj"
Sep 09 02:57:46.617231 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:46.617197    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-khtgd\" (UniqueName: \"kubernetes.io/projected/77b51174-a926-4354-a640-df1878956023-kube-api-access-khtgd\") pod \"ipp-route-2085ee-kserve-7b5c64ddc-7d8tj\" (UID: \"77b51174-a926-4354-a640-df1878956023\") " pod="ai-tenant-e2e-ipp-a-2085ee/ipp-route-2085ee-kserve-7b5c64ddc-7d8tj"
Sep 09 02:57:46.760435 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:46.760345    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-2085ee/ipp-route-2085ee-kserve-7b5c64ddc-7d8tj"
Sep 09 02:57:47.032657 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:47.032631    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-2085ee/ipp-route-2085ee-kserve-7b5c64ddc-7d8tj"]
Sep 09 02:57:47.034437 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:57:47.034405    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod77b51174_a926_4354_a640_df1878956023.slice/crio-3db2dd706ea6f096906d61aad3b1b483b7b4378bec9cd46f475b9e74ab8ae436 WatchSource:0}: Error finding container 3db2dd706ea6f096906d61aad3b1b483b7b4378bec9cd46f475b9e74ab8ae436: Status 404 returned error can't find the container with id 3db2dd706ea6f096906d61aad3b1b483b7b4378bec9cd46f475b9e74ab8ae436
Sep 09 02:57:47.323471 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:47.323375    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-2085ee/ipp-route-2085ee-kserve-7b5c64ddc-7d8tj" event={"ID":"77b51174-a926-4354-a640-df1878956023","Type":"ContainerStarted","Data":"b9f139cca66d1bc7786cb3add7c11e01aa4aad06e90721857dac97bb10e3c2a8"}
Sep 09 02:57:47.323471 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:47.323427    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-2085ee/ipp-route-2085ee-kserve-7b5c64ddc-7d8tj" event={"ID":"77b51174-a926-4354-a640-df1878956023","Type":"ContainerStarted","Data":"3db2dd706ea6f096906d61aad3b1b483b7b4378bec9cd46f475b9e74ab8ae436"}
Sep 09 02:57:47.341626 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:47.341579    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-2085ee/ipp-route-2085ee-kserve-7b5c64ddc-7d8tj" podStartSLOduration=1.34156331 podStartE2EDuration="1.34156331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:57:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:57:47.339580981 +0000 UTC m=+1649.593884618" watchObservedRunningTime="2026-09-09 02:57:47.34156331 +0000 UTC m=+1649.595866945"
Sep 09 02:57:50.448162 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:50.448124    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-99qjf"]
Sep 09 02:57:56.761174 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:56.761127    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-2085ee/ipp-route-2085ee-kserve-7b5c64ddc-7d8tj"
Sep 09 02:57:56.761174 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:56.761187    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-2085ee/ipp-route-2085ee-kserve-7b5c64ddc-7d8tj"
Sep 09 02:57:56.773195 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:56.773167    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-2085ee/ipp-route-2085ee-kserve-7b5c64ddc-7d8tj"
Sep 09 02:57:57.383813 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:57:57.383782    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-2085ee/ipp-route-2085ee-kserve-7b5c64ddc-7d8tj"
Sep 09 02:58:03.942822 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:03.942784    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-99qjf"]
Sep 09 02:58:10.296163 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:10.296128    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-2b8247-openshift-default-76c4697bbb-tjlqv"]
Sep 09 02:58:10.296573 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:10.296559    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7463a0d2-3ecb-479f-b47b-b4b5e8190396" containerName="istio-proxy"
Sep 09 02:58:10.296631 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:10.296575    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="7463a0d2-3ecb-479f-b47b-b4b5e8190396" containerName="istio-proxy"
Sep 09 02:58:10.299789 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:10.299772    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-2b8247-openshift-default-76c4697bbb-tjlqv"
Sep 09 02:58:10.302750 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:10.302731    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-2b8247-openshift-default-dockercfg-csx4v\""
Sep 09 02:58:10.315322 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:10.315299    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-2b8247-openshift-default-76c4697bbb-tjlqv"]
Sep 09 02:58:10.416662 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:10.416626    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca2e58af-7028-4ba4-9b83-59485b5af103-istiod-ca-cert\") pod \"e2e-ipp-cleanup-2b8247-openshift-default-76c4697bbb-tjlqv\" (UID: \"ca2e58af-7028-4ba4-9b83-59485b5af103\") " pod="openshift-ingress/e2e-ipp-cleanup-2b8247-openshift-default-76c4697bbb-tjlqv"
Sep 09 02:58:10.416847 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:10.416747    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca2e58af-7028-4ba4-9b83-59485b5af103-credential-socket\") pod \"e2e-ipp-cleanup-2b8247-openshift-default-76c4697bbb-tjlqv\" (UID: \"ca2e58af-7028-4ba4-9b83-59485b5af103\") " pod="openshift-ingress/e2e-ipp-cleanup-2b8247-openshift-default-76c4697bbb-tjlqv"
Sep 09 02:58:10.416847 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:10.416790    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqvbz\" (UniqueName: \"kubernetes.io/projected/ca2e58af-7028-4ba4-9b83-59485b5af103-kube-api-access-tqvbz\") pod \"e2e-ipp-cleanup-2b8247-openshift-default-76c4697bbb-tjlqv\" (UID: \"ca2e58af-7028-4ba4-9b83-59485b5af103\") " pod="openshift-ingress/e2e-ipp-cleanup-2b8247-openshift-default-76c4697bbb-tjlqv"
Sep 09 02:58:10.416847 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:10.416817    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca2e58af-7028-4ba4-9b83-59485b5af103-istio-envoy\") pod \"e2e-ipp-cleanup-2b8247-openshift-default-76c4697bbb-tjlqv\" (UID: \"ca2e58af-7028-4ba4-9b83-59485b5af103\") " pod="openshift-ingress/e2e-ipp-cleanup-2b8247-openshift-default-76c4697bbb-tjlqv"
Sep 09 02:58:10.416959 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:10.416878    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca2e58af-7028-4ba4-9b83-59485b5af103-istio-data\") pod \"e2e-ipp-cleanup-2b8247-openshift-default-76c4697bbb-tjlqv\" (UID: \"ca2e58af-7028-4ba4-9b83-59485b5af103\") " pod="openshift-ingress/e2e-ipp-cleanup-2b8247-openshift-default-76c4697bbb-tjlqv"
Sep 09 02:58:10.416959 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:10.416912    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca2e58af-7028-4ba4-9b83-59485b5af103-workload-socket\") pod \"e2e-ipp-cleanup-2b8247-openshift-default-76c4697bbb-tjlqv\" (UID: \"ca2e58af-7028-4ba4-9b83-59485b5af103\") " pod="openshift-ingress/e2e-ipp-cleanup-2b8247-openshift-default-76c4697bbb-tjlqv"
Sep 09 02:58:10.416959 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:10.416949    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca2e58af-7028-4ba4-9b83-59485b5af103-workload-certs\") pod \"e2e-ipp-cleanup-2b8247-openshift-default-76c4697bbb-tjlqv\" (UID: \"ca2e58af-7028-4ba4-9b83-59485b5af103\") " pod="openshift-ingress/e2e-ipp-cleanup-2b8247-openshift-default-76c4697bbb-tjlqv"
Sep 09 02:58:10.417075 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:10.416970    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca2e58af-7028-4ba4-9b83-59485b5af103-istio-podinfo\") pod \"e2e-ipp-cleanup-2b8247-openshift-default-76c4697bbb-tjlqv\" (UID: \"ca2e58af-7028-4ba4-9b83-59485b5af103\") " pod="openshift-ingress/e2e-ipp-cleanup-2b8247-openshift-default-76c4697bbb-tjlqv"
Sep 09 02:58:10.417075 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:10.417018    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca2e58af-7028-4ba4-9b83-59485b5af103-istio-token\") pod \"e2e-ipp-cleanup-2b8247-openshift-default-76c4697bbb-tjlqv\" (UID: \"ca2e58af-7028-4ba4-9b83-59485b5af103\") " pod="openshift-ingress/e2e-ipp-cleanup-2b8247-openshift-default-76c4697bbb-tjlqv"
Sep 09 02:58:10.517847 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:10.517815    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca2e58af-7028-4ba4-9b83-59485b5af103-istiod-ca-cert\") pod \"e2e-ipp-cleanup-2b8247-openshift-default-76c4697bbb-tjlqv\" (UID: \"ca2e58af-7028-4ba4-9b83-59485b5af103\") " pod="openshift-ingress/e2e-ipp-cleanup-2b8247-openshift-default-76c4697bbb-tjlqv"
Sep 09 02:58:10.518048 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:10.517898    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca2e58af-7028-4ba4-9b83-59485b5af103-credential-socket\") pod \"e2e-ipp-cleanup-2b8247-openshift-default-76c4697bbb-tjlqv\" (UID: \"ca2e58af-7028-4ba4-9b83-59485b5af103\") " pod="openshift-ingress/e2e-ipp-cleanup-2b8247-openshift-default-76c4697bbb-tjlqv"
Sep 09 02:58:10.518048 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:10.517919    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqvbz\" (UniqueName: \"kubernetes.io/projected/ca2e58af-7028-4ba4-9b83-59485b5af103-kube-api-access-tqvbz\") pod \"e2e-ipp-cleanup-2b8247-openshift-default-76c4697bbb-tjlqv\" (UID: \"ca2e58af-7028-4ba4-9b83-59485b5af103\") " pod="openshift-ingress/e2e-ipp-cleanup-2b8247-openshift-default-76c4697bbb-tjlqv"
Sep 09 02:58:10.518048 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:10.517942    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca2e58af-7028-4ba4-9b83-59485b5af103-istio-envoy\") pod \"e2e-ipp-cleanup-2b8247-openshift-default-76c4697bbb-tjlqv\" (UID: \"ca2e58af-7028-4ba4-9b83-59485b5af103\") " pod="openshift-ingress/e2e-ipp-cleanup-2b8247-openshift-default-76c4697bbb-tjlqv"
Sep 09 02:58:10.518048 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:10.518005    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca2e58af-7028-4ba4-9b83-59485b5af103-istio-data\") pod \"e2e-ipp-cleanup-2b8247-openshift-default-76c4697bbb-tjlqv\" (UID: \"ca2e58af-7028-4ba4-9b83-59485b5af103\") " pod="openshift-ingress/e2e-ipp-cleanup-2b8247-openshift-default-76c4697bbb-tjlqv"
Sep 09 02:58:10.518048 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:10.518038    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca2e58af-7028-4ba4-9b83-59485b5af103-workload-socket\") pod \"e2e-ipp-cleanup-2b8247-openshift-default-76c4697bbb-tjlqv\" (UID: \"ca2e58af-7028-4ba4-9b83-59485b5af103\") " pod="openshift-ingress/e2e-ipp-cleanup-2b8247-openshift-default-76c4697bbb-tjlqv"
Sep 09 02:58:10.518322 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:10.518083    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca2e58af-7028-4ba4-9b83-59485b5af103-workload-certs\") pod \"e2e-ipp-cleanup-2b8247-openshift-default-76c4697bbb-tjlqv\" (UID: \"ca2e58af-7028-4ba4-9b83-59485b5af103\") " pod="openshift-ingress/e2e-ipp-cleanup-2b8247-openshift-default-76c4697bbb-tjlqv"
Sep 09 02:58:10.518322 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:10.518300    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca2e58af-7028-4ba4-9b83-59485b5af103-istio-podinfo\") pod \"e2e-ipp-cleanup-2b8247-openshift-default-76c4697bbb-tjlqv\" (UID: \"ca2e58af-7028-4ba4-9b83-59485b5af103\") " pod="openshift-ingress/e2e-ipp-cleanup-2b8247-openshift-default-76c4697bbb-tjlqv"
Sep 09 02:58:10.518440 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:10.518333    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca2e58af-7028-4ba4-9b83-59485b5af103-credential-socket\") pod \"e2e-ipp-cleanup-2b8247-openshift-default-76c4697bbb-tjlqv\" (UID: \"ca2e58af-7028-4ba4-9b83-59485b5af103\") " pod="openshift-ingress/e2e-ipp-cleanup-2b8247-openshift-default-76c4697bbb-tjlqv"
Sep 09 02:58:10.518440 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:10.518356    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca2e58af-7028-4ba4-9b83-59485b5af103-istio-token\") pod \"e2e-ipp-cleanup-2b8247-openshift-default-76c4697bbb-tjlqv\" (UID: \"ca2e58af-7028-4ba4-9b83-59485b5af103\") " pod="openshift-ingress/e2e-ipp-cleanup-2b8247-openshift-default-76c4697bbb-tjlqv"
Sep 09 02:58:10.518440 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:10.518427    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca2e58af-7028-4ba4-9b83-59485b5af103-workload-certs\") pod \"e2e-ipp-cleanup-2b8247-openshift-default-76c4697bbb-tjlqv\" (UID: \"ca2e58af-7028-4ba4-9b83-59485b5af103\") " pod="openshift-ingress/e2e-ipp-cleanup-2b8247-openshift-default-76c4697bbb-tjlqv"
Sep 09 02:58:10.518610 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:10.518493    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca2e58af-7028-4ba4-9b83-59485b5af103-istio-data\") pod \"e2e-ipp-cleanup-2b8247-openshift-default-76c4697bbb-tjlqv\" (UID: \"ca2e58af-7028-4ba4-9b83-59485b5af103\") " pod="openshift-ingress/e2e-ipp-cleanup-2b8247-openshift-default-76c4697bbb-tjlqv"
Sep 09 02:58:10.518610 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:10.518557    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca2e58af-7028-4ba4-9b83-59485b5af103-workload-socket\") pod \"e2e-ipp-cleanup-2b8247-openshift-default-76c4697bbb-tjlqv\" (UID: \"ca2e58af-7028-4ba4-9b83-59485b5af103\") " pod="openshift-ingress/e2e-ipp-cleanup-2b8247-openshift-default-76c4697bbb-tjlqv"
Sep 09 02:58:10.518716 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:10.518653    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca2e58af-7028-4ba4-9b83-59485b5af103-istiod-ca-cert\") pod \"e2e-ipp-cleanup-2b8247-openshift-default-76c4697bbb-tjlqv\" (UID: \"ca2e58af-7028-4ba4-9b83-59485b5af103\") " pod="openshift-ingress/e2e-ipp-cleanup-2b8247-openshift-default-76c4697bbb-tjlqv"
Sep 09 02:58:10.520343 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:10.520323    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca2e58af-7028-4ba4-9b83-59485b5af103-istio-envoy\") pod \"e2e-ipp-cleanup-2b8247-openshift-default-76c4697bbb-tjlqv\" (UID: \"ca2e58af-7028-4ba4-9b83-59485b5af103\") " pod="openshift-ingress/e2e-ipp-cleanup-2b8247-openshift-default-76c4697bbb-tjlqv"
Sep 09 02:58:10.520634 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:10.520616    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca2e58af-7028-4ba4-9b83-59485b5af103-istio-podinfo\") pod \"e2e-ipp-cleanup-2b8247-openshift-default-76c4697bbb-tjlqv\" (UID: \"ca2e58af-7028-4ba4-9b83-59485b5af103\") " pod="openshift-ingress/e2e-ipp-cleanup-2b8247-openshift-default-76c4697bbb-tjlqv"
Sep 09 02:58:10.526930 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:10.526906    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca2e58af-7028-4ba4-9b83-59485b5af103-istio-token\") pod \"e2e-ipp-cleanup-2b8247-openshift-default-76c4697bbb-tjlqv\" (UID: \"ca2e58af-7028-4ba4-9b83-59485b5af103\") " pod="openshift-ingress/e2e-ipp-cleanup-2b8247-openshift-default-76c4697bbb-tjlqv"
Sep 09 02:58:10.527272 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:10.527251    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqvbz\" (UniqueName: \"kubernetes.io/projected/ca2e58af-7028-4ba4-9b83-59485b5af103-kube-api-access-tqvbz\") pod \"e2e-ipp-cleanup-2b8247-openshift-default-76c4697bbb-tjlqv\" (UID: \"ca2e58af-7028-4ba4-9b83-59485b5af103\") " pod="openshift-ingress/e2e-ipp-cleanup-2b8247-openshift-default-76c4697bbb-tjlqv"
Sep 09 02:58:10.610426 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:10.610398    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-2b8247-openshift-default-76c4697bbb-tjlqv"
Sep 09 02:58:10.889378 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:10.889351    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-2b8247-openshift-default-76c4697bbb-tjlqv"]
Sep 09 02:58:10.891158 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:58:10.891130    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca2e58af_7028_4ba4_9b83_59485b5af103.slice/crio-8a474b7032eb96654c3bf50f118d62a79407983514fac550bc080110944f5748 WatchSource:0}: Error finding container 8a474b7032eb96654c3bf50f118d62a79407983514fac550bc080110944f5748: Status 404 returned error can't find the container with id 8a474b7032eb96654c3bf50f118d62a79407983514fac550bc080110944f5748
Sep 09 02:58:10.893419 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:10.893386    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 02:58:10.893516 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:10.893465    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 02:58:10.893516 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:10.893509    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 02:58:11.446550 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:11.446515    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-2b8247-openshift-default-76c4697bbb-tjlqv" event={"ID":"ca2e58af-7028-4ba4-9b83-59485b5af103","Type":"ContainerStarted","Data":"fa9ea1e66c7a618fdba5e108fb25c35ecff6f1a6225efb1947e2d584fdba947c"}
Sep 09 02:58:11.446916 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:11.446599    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-2b8247-openshift-default-76c4697bbb-tjlqv" event={"ID":"ca2e58af-7028-4ba4-9b83-59485b5af103","Type":"ContainerStarted","Data":"8a474b7032eb96654c3bf50f118d62a79407983514fac550bc080110944f5748"}
Sep 09 02:58:11.470916 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:11.470869    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-2b8247-openshift-default-76c4697bbb-tjlqv" podStartSLOduration=1.470854421 podStartE2EDuration="1.470854421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:58:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:58:11.468761588 +0000 UTC m=+1673.723065225" watchObservedRunningTime="2026-09-09 02:58:11.470854421 +0000 UTC m=+1673.725158057"
Sep 09 02:58:11.611497 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:11.611464    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-2b8247-openshift-default-76c4697bbb-tjlqv"
Sep 09 02:58:11.616431 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:11.616408    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-2b8247-openshift-default-76c4697bbb-tjlqv"
Sep 09 02:58:12.452203 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:12.452171    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-2b8247-openshift-default-76c4697bbb-tjlqv"
Sep 09 02:58:12.453240 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:12.453218    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-2b8247-openshift-default-76c4697bbb-tjlqv"
Sep 09 02:58:18.532683 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:18.532652    2570 scope.go:117] "RemoveContainer" containerID="f8533d367700abbbe51870e99aa0628352be9d5ccd943215c616f30dd05d07fb"
Sep 09 02:58:23.446533 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:23.446499    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-99qjf"]
Sep 09 02:58:34.572678 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:34.572643    2570 generic.go:356] "Generic (PLEG): container finished" podID="ca2e58af-7028-4ba4-9b83-59485b5af103" containerID="fa9ea1e66c7a618fdba5e108fb25c35ecff6f1a6225efb1947e2d584fdba947c" exitCode=0
Sep 09 02:58:34.573117 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:34.572718    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-2b8247-openshift-default-76c4697bbb-tjlqv" event={"ID":"ca2e58af-7028-4ba4-9b83-59485b5af103","Type":"ContainerDied","Data":"fa9ea1e66c7a618fdba5e108fb25c35ecff6f1a6225efb1947e2d584fdba947c"}
Sep 09 02:58:34.573244 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:34.573223    2570 scope.go:117] "RemoveContainer" containerID="fa9ea1e66c7a618fdba5e108fb25c35ecff6f1a6225efb1947e2d584fdba947c"
Sep 09 02:58:34.574398 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:34.574361    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 02:58:34.574517 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:34.574453    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 02:58:34.574517 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:34.574498    2570 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 02:58:35.580171 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:35.580140    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-2b8247-openshift-default-76c4697bbb-tjlqv" event={"ID":"ca2e58af-7028-4ba4-9b83-59485b5af103","Type":"ContainerStarted","Data":"90cbeb579da38ed88054597a43c581565e1ec2c6f15be38d034c7b632949099c"}
Sep 09 02:58:35.611053 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:35.611023    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-2b8247-openshift-default-76c4697bbb-tjlqv"
Sep 09 02:58:35.611270 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:35.611249    2570 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-2b8247-openshift-default-76c4697bbb-tjlqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Sep 09 02:58:35.611314 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:35.611295    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-2b8247-openshift-default-76c4697bbb-tjlqv" podUID="ca2e58af-7028-4ba4-9b83-59485b5af103" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Sep 09 02:58:36.610855 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:36.610819    2570 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-2b8247-openshift-default-76c4697bbb-tjlqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Sep 09 02:58:36.611248 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:36.610883    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-2b8247-openshift-default-76c4697bbb-tjlqv" podUID="ca2e58af-7028-4ba4-9b83-59485b5af103" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Sep 09 02:58:37.611630 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:37.611591    2570 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-2b8247-openshift-default-76c4697bbb-tjlqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Sep 09 02:58:37.612140 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:37.611652    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-2b8247-openshift-default-76c4697bbb-tjlqv" podUID="ca2e58af-7028-4ba4-9b83-59485b5af103" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Sep 09 02:58:38.610960 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:38.610925    2570 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-2b8247-openshift-default-76c4697bbb-tjlqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Sep 09 02:58:38.611155 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:38.611001    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-2b8247-openshift-default-76c4697bbb-tjlqv" podUID="ca2e58af-7028-4ba4-9b83-59485b5af103" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Sep 09 02:58:39.336365 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:39.336329    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2b8247-kjjgn"]
Sep 09 02:58:39.339905 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:39.339875    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2b8247-kjjgn"
Sep 09 02:58:39.342849 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:39.342821    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 02:58:39.343316 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:39.343094    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vbr52\""
Sep 09 02:58:39.348502 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:39.348473    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2b8247-kjjgn"]
Sep 09 02:58:39.400375 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:39.400337    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1679d327-aa41-47f9-9a8a-7a8d061563c1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-2b8247-kjjgn\" (UID: \"1679d327-aa41-47f9-9a8a-7a8d061563c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2b8247-kjjgn"
Sep 09 02:58:39.501875 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:39.501817    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1679d327-aa41-47f9-9a8a-7a8d061563c1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-2b8247-kjjgn\" (UID: \"1679d327-aa41-47f9-9a8a-7a8d061563c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2b8247-kjjgn"
Sep 09 02:58:39.502550 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:39.502530    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1679d327-aa41-47f9-9a8a-7a8d061563c1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-2b8247-kjjgn\" (UID: \"1679d327-aa41-47f9-9a8a-7a8d061563c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2b8247-kjjgn"
Sep 09 02:58:39.610849 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:39.610762    2570 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-2b8247-openshift-default-76c4697bbb-tjlqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Sep 09 02:58:39.610849 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:39.610832    2570 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-2b8247-openshift-default-76c4697bbb-tjlqv" podUID="ca2e58af-7028-4ba4-9b83-59485b5af103" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Sep 09 02:58:39.680409 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:39.680372    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2b8247-kjjgn"
Sep 09 02:58:39.952259 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:39.952167    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2b8247-kjjgn"]
Sep 09 02:58:39.965328 ip-10-0-136-188 kubenswrapper[2570]: W0909 02:58:39.965294    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1679d327_aa41_47f9_9a8a_7a8d061563c1.slice/crio-acd120025189dba8414f85423bcda239321b2c92df0ac012957f6d540258ad85 WatchSource:0}: Error finding container acd120025189dba8414f85423bcda239321b2c92df0ac012957f6d540258ad85: Status 404 returned error can't find the container with id acd120025189dba8414f85423bcda239321b2c92df0ac012957f6d540258ad85
Sep 09 02:58:40.607877 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:40.607764    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2b8247-kjjgn" event={"ID":"1679d327-aa41-47f9-9a8a-7a8d061563c1","Type":"ContainerStarted","Data":"9a0c073a851ba8d4302d9332fe7741dfcb400e35c99e619a4b68d59b369b659c"}
Sep 09 02:58:40.607877 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:40.607833    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2b8247-kjjgn" event={"ID":"1679d327-aa41-47f9-9a8a-7a8d061563c1","Type":"ContainerStarted","Data":"acd120025189dba8414f85423bcda239321b2c92df0ac012957f6d540258ad85"}
Sep 09 02:58:40.624539 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:40.624481    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2b8247-kjjgn" podStartSLOduration=1.624465583 podStartE2EDuration="1.624465583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 02:58:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 02:58:40.624263893 +0000 UTC m=+1702.878567540" watchObservedRunningTime="2026-09-09 02:58:40.624465583 +0000 UTC m=+1702.878769216"
Sep 09 02:58:40.714665 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:40.714630    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-2b8247-openshift-default-76c4697bbb-tjlqv"
Sep 09 02:58:40.715030 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:40.715009    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-2b8247-openshift-default-76c4697bbb-tjlqv"
Sep 09 02:58:40.715692 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:40.715677    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-2b8247-openshift-default-76c4697bbb-tjlqv"
Sep 09 02:58:41.614216 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:41.614181    2570 generic.go:356] "Generic (PLEG): container finished" podID="1679d327-aa41-47f9-9a8a-7a8d061563c1" containerID="9a0c073a851ba8d4302d9332fe7741dfcb400e35c99e619a4b68d59b369b659c" exitCode=0
Sep 09 02:58:41.614627 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:41.614223    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2b8247-kjjgn" event={"ID":"1679d327-aa41-47f9-9a8a-7a8d061563c1","Type":"ContainerDied","Data":"9a0c073a851ba8d4302d9332fe7741dfcb400e35c99e619a4b68d59b369b659c"}
Sep 09 02:58:42.751359 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:42.751336    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2b8247-kjjgn"
Sep 09 02:58:42.836916 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:42.836880    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1679d327-aa41-47f9-9a8a-7a8d061563c1-maas-api-service-ca\") pod \"1679d327-aa41-47f9-9a8a-7a8d061563c1\" (UID: \"1679d327-aa41-47f9-9a8a-7a8d061563c1\") "
Sep 09 02:58:42.837250 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:42.837229    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1679d327-aa41-47f9-9a8a-7a8d061563c1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1679d327-aa41-47f9-9a8a-7a8d061563c1" (UID: "1679d327-aa41-47f9-9a8a-7a8d061563c1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 02:58:42.938323 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:42.938249    2570 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1679d327-aa41-47f9-9a8a-7a8d061563c1-maas-api-service-ca\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:58:43.626476 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:43.626444    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2b8247-kjjgn"
Sep 09 02:58:43.626647 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:43.626477    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2b8247-kjjgn" event={"ID":"1679d327-aa41-47f9-9a8a-7a8d061563c1","Type":"ContainerDied","Data":"acd120025189dba8414f85423bcda239321b2c92df0ac012957f6d540258ad85"}
Sep 09 02:58:43.626647 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:43.626524    2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="acd120025189dba8414f85423bcda239321b2c92df0ac012957f6d540258ad85"
Sep 09 02:58:45.383003 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:45.382949    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2b8247-kjjgn"]
Sep 09 02:58:45.385530 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:45.385505    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2b8247-kjjgn"]
Sep 09 02:58:46.223014 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:46.222967    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1679d327-aa41-47f9-9a8a-7a8d061563c1" path="/var/lib/kubelet/pods/1679d327-aa41-47f9-9a8a-7a8d061563c1/volumes"
Sep 09 02:58:49.845585 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:49.845551    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-99qjf"]
Sep 09 02:58:52.017190 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:52.017149    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-2b8247-openshift-default-76c4697bbb-tjlqv"]
Sep 09 02:58:52.017656 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:52.017412    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-2b8247-openshift-default-76c4697bbb-tjlqv" podUID="ca2e58af-7028-4ba4-9b83-59485b5af103" containerName="istio-proxy" containerID="cri-o://90cbeb579da38ed88054597a43c581565e1ec2c6f15be38d034c7b632949099c" gracePeriod=30
Sep 09 02:58:55.718048 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:55.718005    2570 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-2b8247-openshift-default-76c4697bbb-tjlqv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused
Sep 09 02:58:55.718480 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:55.718074    2570 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-cleanup-2b8247-openshift-default-76c4697bbb-tjlqv" podUID="ca2e58af-7028-4ba4-9b83-59485b5af103" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 02:58:57.281658 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:57.281591    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-2b8247-openshift-default-76c4697bbb-tjlqv"
Sep 09 02:58:57.380530 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:57.380497    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca2e58af-7028-4ba4-9b83-59485b5af103-credential-socket\") pod \"ca2e58af-7028-4ba4-9b83-59485b5af103\" (UID: \"ca2e58af-7028-4ba4-9b83-59485b5af103\") "
Sep 09 02:58:57.380720 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:57.380551    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tqvbz\" (UniqueName: \"kubernetes.io/projected/ca2e58af-7028-4ba4-9b83-59485b5af103-kube-api-access-tqvbz\") pod \"ca2e58af-7028-4ba4-9b83-59485b5af103\" (UID: \"ca2e58af-7028-4ba4-9b83-59485b5af103\") "
Sep 09 02:58:57.380720 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:57.380578    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca2e58af-7028-4ba4-9b83-59485b5af103-istio-token\") pod \"ca2e58af-7028-4ba4-9b83-59485b5af103\" (UID: \"ca2e58af-7028-4ba4-9b83-59485b5af103\") "
Sep 09 02:58:57.380720 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:57.380618    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca2e58af-7028-4ba4-9b83-59485b5af103-workload-certs\") pod \"ca2e58af-7028-4ba4-9b83-59485b5af103\" (UID: \"ca2e58af-7028-4ba4-9b83-59485b5af103\") "
Sep 09 02:58:57.380720 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:57.380680    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca2e58af-7028-4ba4-9b83-59485b5af103-workload-socket\") pod \"ca2e58af-7028-4ba4-9b83-59485b5af103\" (UID: \"ca2e58af-7028-4ba4-9b83-59485b5af103\") "
Sep 09 02:58:57.380934 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:57.380726    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca2e58af-7028-4ba4-9b83-59485b5af103-istio-envoy\") pod \"ca2e58af-7028-4ba4-9b83-59485b5af103\" (UID: \"ca2e58af-7028-4ba4-9b83-59485b5af103\") "
Sep 09 02:58:57.380934 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:57.380770    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca2e58af-7028-4ba4-9b83-59485b5af103-istio-podinfo\") pod \"ca2e58af-7028-4ba4-9b83-59485b5af103\" (UID: \"ca2e58af-7028-4ba4-9b83-59485b5af103\") "
Sep 09 02:58:57.380934 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:57.380774    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca2e58af-7028-4ba4-9b83-59485b5af103-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ca2e58af-7028-4ba4-9b83-59485b5af103" (UID: "ca2e58af-7028-4ba4-9b83-59485b5af103"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 02:58:57.380934 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:57.380810    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca2e58af-7028-4ba4-9b83-59485b5af103-istiod-ca-cert\") pod \"ca2e58af-7028-4ba4-9b83-59485b5af103\" (UID: \"ca2e58af-7028-4ba4-9b83-59485b5af103\") "
Sep 09 02:58:57.381179 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:57.380935    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca2e58af-7028-4ba4-9b83-59485b5af103-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ca2e58af-7028-4ba4-9b83-59485b5af103" (UID: "ca2e58af-7028-4ba4-9b83-59485b5af103"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 02:58:57.381179 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:57.380955    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca2e58af-7028-4ba4-9b83-59485b5af103-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ca2e58af-7028-4ba4-9b83-59485b5af103" (UID: "ca2e58af-7028-4ba4-9b83-59485b5af103"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 02:58:57.381179 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:57.381091    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca2e58af-7028-4ba4-9b83-59485b5af103-istio-data\") pod \"ca2e58af-7028-4ba4-9b83-59485b5af103\" (UID: \"ca2e58af-7028-4ba4-9b83-59485b5af103\") "
Sep 09 02:58:57.381338 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:57.381275    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca2e58af-7028-4ba4-9b83-59485b5af103-istio-data" (OuterVolumeSpecName: "istio-data") pod "ca2e58af-7028-4ba4-9b83-59485b5af103" (UID: "ca2e58af-7028-4ba4-9b83-59485b5af103"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 02:58:57.381338 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:57.381309    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca2e58af-7028-4ba4-9b83-59485b5af103-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ca2e58af-7028-4ba4-9b83-59485b5af103" (UID: "ca2e58af-7028-4ba4-9b83-59485b5af103"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 02:58:57.381545 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:57.381526    2570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca2e58af-7028-4ba4-9b83-59485b5af103-workload-certs\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:58:57.381607 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:57.381552    2570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca2e58af-7028-4ba4-9b83-59485b5af103-workload-socket\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:58:57.381607 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:57.381567    2570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca2e58af-7028-4ba4-9b83-59485b5af103-istiod-ca-cert\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:58:57.381607 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:57.381582    2570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca2e58af-7028-4ba4-9b83-59485b5af103-istio-data\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:58:57.381607 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:57.381594    2570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca2e58af-7028-4ba4-9b83-59485b5af103-credential-socket\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:58:57.383078 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:57.383048    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca2e58af-7028-4ba4-9b83-59485b5af103-istio-token" (OuterVolumeSpecName: "istio-token") pod "ca2e58af-7028-4ba4-9b83-59485b5af103" (UID: "ca2e58af-7028-4ba4-9b83-59485b5af103"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 02:58:57.383175 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:57.383091    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca2e58af-7028-4ba4-9b83-59485b5af103-kube-api-access-tqvbz" (OuterVolumeSpecName: "kube-api-access-tqvbz") pod "ca2e58af-7028-4ba4-9b83-59485b5af103" (UID: "ca2e58af-7028-4ba4-9b83-59485b5af103"). InnerVolumeSpecName "kube-api-access-tqvbz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 02:58:57.383269 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:57.383249    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca2e58af-7028-4ba4-9b83-59485b5af103-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ca2e58af-7028-4ba4-9b83-59485b5af103" (UID: "ca2e58af-7028-4ba4-9b83-59485b5af103"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 02:58:57.383412 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:57.383389    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ca2e58af-7028-4ba4-9b83-59485b5af103-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ca2e58af-7028-4ba4-9b83-59485b5af103" (UID: "ca2e58af-7028-4ba4-9b83-59485b5af103"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 02:58:57.482941 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:57.482901    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqvbz\" (UniqueName: \"kubernetes.io/projected/ca2e58af-7028-4ba4-9b83-59485b5af103-kube-api-access-tqvbz\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:58:57.482941 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:57.482936    2570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca2e58af-7028-4ba4-9b83-59485b5af103-istio-token\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:58:57.482941 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:57.482946    2570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca2e58af-7028-4ba4-9b83-59485b5af103-istio-envoy\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:58:57.483202 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:57.482954    2570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca2e58af-7028-4ba4-9b83-59485b5af103-istio-podinfo\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:58:57.699335 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:57.699299    2570 generic.go:356] "Generic (PLEG): container finished" podID="ca2e58af-7028-4ba4-9b83-59485b5af103" containerID="90cbeb579da38ed88054597a43c581565e1ec2c6f15be38d034c7b632949099c" exitCode=0
Sep 09 02:58:57.699510 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:57.699348    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-2b8247-openshift-default-76c4697bbb-tjlqv"
Sep 09 02:58:57.699510 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:57.699355    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-2b8247-openshift-default-76c4697bbb-tjlqv" event={"ID":"ca2e58af-7028-4ba4-9b83-59485b5af103","Type":"ContainerDied","Data":"90cbeb579da38ed88054597a43c581565e1ec2c6f15be38d034c7b632949099c"}
Sep 09 02:58:57.699510 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:57.699441    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-2b8247-openshift-default-76c4697bbb-tjlqv" event={"ID":"ca2e58af-7028-4ba4-9b83-59485b5af103","Type":"ContainerDied","Data":"8a474b7032eb96654c3bf50f118d62a79407983514fac550bc080110944f5748"}
Sep 09 02:58:57.699510 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:57.699473    2570 scope.go:117] "RemoveContainer" containerID="90cbeb579da38ed88054597a43c581565e1ec2c6f15be38d034c7b632949099c"
Sep 09 02:58:57.708479 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:57.708461    2570 scope.go:117] "RemoveContainer" containerID="fa9ea1e66c7a618fdba5e108fb25c35ecff6f1a6225efb1947e2d584fdba947c"
Sep 09 02:58:57.716506 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:57.716488    2570 scope.go:117] "RemoveContainer" containerID="90cbeb579da38ed88054597a43c581565e1ec2c6f15be38d034c7b632949099c"
Sep 09 02:58:57.716776 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:58:57.716754    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90cbeb579da38ed88054597a43c581565e1ec2c6f15be38d034c7b632949099c\": container with ID starting with 90cbeb579da38ed88054597a43c581565e1ec2c6f15be38d034c7b632949099c not found: ID does not exist" containerID="90cbeb579da38ed88054597a43c581565e1ec2c6f15be38d034c7b632949099c"
Sep 09 02:58:57.716834 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:57.716784    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90cbeb579da38ed88054597a43c581565e1ec2c6f15be38d034c7b632949099c"} err="failed to get container status \"90cbeb579da38ed88054597a43c581565e1ec2c6f15be38d034c7b632949099c\": rpc error: code = NotFound desc = could not find container \"90cbeb579da38ed88054597a43c581565e1ec2c6f15be38d034c7b632949099c\": container with ID starting with 90cbeb579da38ed88054597a43c581565e1ec2c6f15be38d034c7b632949099c not found: ID does not exist"
Sep 09 02:58:57.716834 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:57.716800    2570 scope.go:117] "RemoveContainer" containerID="fa9ea1e66c7a618fdba5e108fb25c35ecff6f1a6225efb1947e2d584fdba947c"
Sep 09 02:58:57.717032 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:58:57.717014    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa9ea1e66c7a618fdba5e108fb25c35ecff6f1a6225efb1947e2d584fdba947c\": container with ID starting with fa9ea1e66c7a618fdba5e108fb25c35ecff6f1a6225efb1947e2d584fdba947c not found: ID does not exist" containerID="fa9ea1e66c7a618fdba5e108fb25c35ecff6f1a6225efb1947e2d584fdba947c"
Sep 09 02:58:57.717081 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:57.717040    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa9ea1e66c7a618fdba5e108fb25c35ecff6f1a6225efb1947e2d584fdba947c"} err="failed to get container status \"fa9ea1e66c7a618fdba5e108fb25c35ecff6f1a6225efb1947e2d584fdba947c\": rpc error: code = NotFound desc = could not find container \"fa9ea1e66c7a618fdba5e108fb25c35ecff6f1a6225efb1947e2d584fdba947c\": container with ID starting with fa9ea1e66c7a618fdba5e108fb25c35ecff6f1a6225efb1947e2d584fdba947c not found: ID does not exist"
Sep 09 02:58:57.721991 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:57.721955    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-2b8247-openshift-default-76c4697bbb-tjlqv"]
Sep 09 02:58:57.725791 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:57.725772    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-2b8247-openshift-default-76c4697bbb-tjlqv"]
Sep 09 02:58:58.224527 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:58:58.224493    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca2e58af-7028-4ba4-9b83-59485b5af103" path="/var/lib/kubelet/pods/ca2e58af-7028-4ba4-9b83-59485b5af103/volumes"
Sep 09 02:59:09.056954 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:59:09.056919    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-99qjf"]
Sep 09 02:59:17.249469 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:59:17.249436    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-2085ee/ipp-route-2085ee-kserve-7b5c64ddc-7d8tj"]
Sep 09 02:59:18.554882 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:59:18.554851    2570 scope.go:117] "RemoveContainer" containerID="1a90a50edd3e3ff4f78a08fc636783af85e4789fdcbd622ff8a8a77fa9b160a0"
Sep 09 02:59:21.544695 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:59:21.544654    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-99qjf"]
Sep 09 02:59:32.269380 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:59:32.269326    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-2085ee/ipp-route-2085ee-kserve-7b5c64ddc-7d8tj" podUID="77b51174-a926-4354-a640-df1878956023" containerName="main" containerID="cri-o://b9f139cca66d1bc7786cb3add7c11e01aa4aad06e90721857dac97bb10e3c2a8" gracePeriod=45
Sep 09 02:59:32.517094 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:59:32.517068    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-2085ee/ipp-route-2085ee-kserve-7b5c64ddc-7d8tj"
Sep 09 02:59:32.609415 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:59:32.609384    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77b51174-a926-4354-a640-df1878956023-dshm\") pod \"77b51174-a926-4354-a640-df1878956023\" (UID: \"77b51174-a926-4354-a640-df1878956023\") "
Sep 09 02:59:32.609415 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:59:32.609416    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77b51174-a926-4354-a640-df1878956023-model-cache\") pod \"77b51174-a926-4354-a640-df1878956023\" (UID: \"77b51174-a926-4354-a640-df1878956023\") "
Sep 09 02:59:32.609609 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:59:32.609456    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77b51174-a926-4354-a640-df1878956023-tmp-dir\") pod \"77b51174-a926-4354-a640-df1878956023\" (UID: \"77b51174-a926-4354-a640-df1878956023\") "
Sep 09 02:59:32.609609 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:59:32.609497    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77b51174-a926-4354-a640-df1878956023-home\") pod \"77b51174-a926-4354-a640-df1878956023\" (UID: \"77b51174-a926-4354-a640-df1878956023\") "
Sep 09 02:59:32.609609 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:59:32.609574    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77b51174-a926-4354-a640-df1878956023-tls-certs\") pod \"77b51174-a926-4354-a640-df1878956023\" (UID: \"77b51174-a926-4354-a640-df1878956023\") "
Sep 09 02:59:32.609739 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:59:32.609631    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-khtgd\" (UniqueName: \"kubernetes.io/projected/77b51174-a926-4354-a640-df1878956023-kube-api-access-khtgd\") pod \"77b51174-a926-4354-a640-df1878956023\" (UID: \"77b51174-a926-4354-a640-df1878956023\") "
Sep 09 02:59:32.609739 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:59:32.609687    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77b51174-a926-4354-a640-df1878956023-model-cache" (OuterVolumeSpecName: "model-cache") pod "77b51174-a926-4354-a640-df1878956023" (UID: "77b51174-a926-4354-a640-df1878956023"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 02:59:32.609845 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:59:32.609760    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77b51174-a926-4354-a640-df1878956023-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "77b51174-a926-4354-a640-df1878956023" (UID: "77b51174-a926-4354-a640-df1878956023"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 02:59:32.609912 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:59:32.609843    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77b51174-a926-4354-a640-df1878956023-home" (OuterVolumeSpecName: "home") pod "77b51174-a926-4354-a640-df1878956023" (UID: "77b51174-a926-4354-a640-df1878956023"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 02:59:32.610073 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:59:32.610055    2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77b51174-a926-4354-a640-df1878956023-model-cache\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:59:32.610124 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:59:32.610079    2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77b51174-a926-4354-a640-df1878956023-tmp-dir\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:59:32.610124 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:59:32.610092    2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77b51174-a926-4354-a640-df1878956023-home\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:59:32.611719 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:59:32.611691    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77b51174-a926-4354-a640-df1878956023-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "77b51174-a926-4354-a640-df1878956023" (UID: "77b51174-a926-4354-a640-df1878956023"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 02:59:32.611823 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:59:32.611722    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77b51174-a926-4354-a640-df1878956023-dshm" (OuterVolumeSpecName: "dshm") pod "77b51174-a926-4354-a640-df1878956023" (UID: "77b51174-a926-4354-a640-df1878956023"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 02:59:32.611823 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:59:32.611751    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77b51174-a926-4354-a640-df1878956023-kube-api-access-khtgd" (OuterVolumeSpecName: "kube-api-access-khtgd") pod "77b51174-a926-4354-a640-df1878956023" (UID: "77b51174-a926-4354-a640-df1878956023"). InnerVolumeSpecName "kube-api-access-khtgd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 02:59:32.710736 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:59:32.710705    2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77b51174-a926-4354-a640-df1878956023-dshm\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:59:32.710736 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:59:32.710730    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77b51174-a926-4354-a640-df1878956023-tls-certs\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:59:32.710736 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:59:32.710742    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khtgd\" (UniqueName: \"kubernetes.io/projected/77b51174-a926-4354-a640-df1878956023-kube-api-access-khtgd\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 02:59:32.890795 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:59:32.890706    2570 generic.go:356] "Generic (PLEG): container finished" podID="77b51174-a926-4354-a640-df1878956023" containerID="b9f139cca66d1bc7786cb3add7c11e01aa4aad06e90721857dac97bb10e3c2a8" exitCode=0
Sep 09 02:59:32.890795 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:59:32.890745    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-2085ee/ipp-route-2085ee-kserve-7b5c64ddc-7d8tj"
Sep 09 02:59:32.890795 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:59:32.890785    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-2085ee/ipp-route-2085ee-kserve-7b5c64ddc-7d8tj" event={"ID":"77b51174-a926-4354-a640-df1878956023","Type":"ContainerDied","Data":"b9f139cca66d1bc7786cb3add7c11e01aa4aad06e90721857dac97bb10e3c2a8"}
Sep 09 02:59:32.891106 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:59:32.890825    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-2085ee/ipp-route-2085ee-kserve-7b5c64ddc-7d8tj" event={"ID":"77b51174-a926-4354-a640-df1878956023","Type":"ContainerDied","Data":"3db2dd706ea6f096906d61aad3b1b483b7b4378bec9cd46f475b9e74ab8ae436"}
Sep 09 02:59:32.891106 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:59:32.890842    2570 scope.go:117] "RemoveContainer" containerID="b9f139cca66d1bc7786cb3add7c11e01aa4aad06e90721857dac97bb10e3c2a8"
Sep 09 02:59:32.903004 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:59:32.902962    2570 scope.go:117] "RemoveContainer" containerID="b9f139cca66d1bc7786cb3add7c11e01aa4aad06e90721857dac97bb10e3c2a8"
Sep 09 02:59:32.903293 ip-10-0-136-188 kubenswrapper[2570]: E0909 02:59:32.903276    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9f139cca66d1bc7786cb3add7c11e01aa4aad06e90721857dac97bb10e3c2a8\": container with ID starting with b9f139cca66d1bc7786cb3add7c11e01aa4aad06e90721857dac97bb10e3c2a8 not found: ID does not exist" containerID="b9f139cca66d1bc7786cb3add7c11e01aa4aad06e90721857dac97bb10e3c2a8"
Sep 09 02:59:32.903346 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:59:32.903304    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9f139cca66d1bc7786cb3add7c11e01aa4aad06e90721857dac97bb10e3c2a8"} err="failed to get container status \"b9f139cca66d1bc7786cb3add7c11e01aa4aad06e90721857dac97bb10e3c2a8\": rpc error: code = NotFound desc = could not find container \"b9f139cca66d1bc7786cb3add7c11e01aa4aad06e90721857dac97bb10e3c2a8\": container with ID starting with b9f139cca66d1bc7786cb3add7c11e01aa4aad06e90721857dac97bb10e3c2a8 not found: ID does not exist"
Sep 09 02:59:32.912644 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:59:32.912621    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-2085ee/ipp-route-2085ee-kserve-7b5c64ddc-7d8tj"]
Sep 09 02:59:32.919290 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:59:32.919268    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-2085ee/ipp-route-2085ee-kserve-7b5c64ddc-7d8tj"]
Sep 09 02:59:34.228811 ip-10-0-136-188 kubenswrapper[2570]: I0909 02:59:34.228759    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77b51174-a926-4354-a640-df1878956023" path="/var/lib/kubelet/pods/77b51174-a926-4354-a640-df1878956023/volumes"
Sep 09 03:00:00.144567 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:00.144531    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-56dbc0-29815380-brrrn"]
Sep 09 03:00:00.145011 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:00.144908    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca2e58af-7028-4ba4-9b83-59485b5af103" containerName="istio-proxy"
Sep 09 03:00:00.145011 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:00.144921    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca2e58af-7028-4ba4-9b83-59485b5af103" containerName="istio-proxy"
Sep 09 03:00:00.145011 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:00.144928    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca2e58af-7028-4ba4-9b83-59485b5af103" containerName="istio-proxy"
Sep 09 03:00:00.145011 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:00.144933    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca2e58af-7028-4ba4-9b83-59485b5af103" containerName="istio-proxy"
Sep 09 03:00:00.145011 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:00.144952    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1679d327-aa41-47f9-9a8a-7a8d061563c1" containerName="revoke-keys"
Sep 09 03:00:00.145011 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:00.144958    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="1679d327-aa41-47f9-9a8a-7a8d061563c1" containerName="revoke-keys"
Sep 09 03:00:00.145011 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:00.144965    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77b51174-a926-4354-a640-df1878956023" containerName="main"
Sep 09 03:00:00.145011 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:00.144971    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="77b51174-a926-4354-a640-df1878956023" containerName="main"
Sep 09 03:00:00.145271 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:00.145047    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="1679d327-aa41-47f9-9a8a-7a8d061563c1" containerName="revoke-keys"
Sep 09 03:00:00.145271 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:00.145057    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca2e58af-7028-4ba4-9b83-59485b5af103" containerName="istio-proxy"
Sep 09 03:00:00.145271 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:00.145063    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="77b51174-a926-4354-a640-df1878956023" containerName="main"
Sep 09 03:00:00.145271 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:00.145069    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca2e58af-7028-4ba4-9b83-59485b5af103" containerName="istio-proxy"
Sep 09 03:00:00.147905 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:00.147889    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-621ac9-29815380-vv7qq"]
Sep 09 03:00:00.148029 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:00.148009    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-56dbc0-29815380-brrrn"
Sep 09 03:00:00.151095 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:00.151071    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-621ac9-29815380-vv7qq"
Sep 09 03:00:00.152096 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:00.152069    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vbr52\""
Sep 09 03:00:00.160061 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:00.157196    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-621ac9-29815380-vv7qq"]
Sep 09 03:00:00.160731 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:00.160706    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-56dbc0-29815380-brrrn"]
Sep 09 03:00:00.166328 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:00.166305    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-56dbc0-29815380-brrrn"
Sep 09 03:00:00.177126 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:00.177108    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-621ac9-29815380-vv7qq"
Sep 09 03:00:00.181501 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:00.181452    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-2085ee-openshift-default-7fb75b87b8-nggqw"]
Sep 09 03:00:00.182235 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:00.181920    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-2085ee-openshift-default-7fb75b87b8-nggqw" podUID="8d70260f-d945-437c-9fe6-432561b37d70" containerName="istio-proxy" containerID="cri-o://17a49bac98918b39d38558e406879f2c301302438aa34df53d3977cf94cce3e0" gracePeriod=30
Sep 09 03:00:00.458307 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:00.458266    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-56dbc0-29815380-brrrn"]
Sep 09 03:00:00.460748 ip-10-0-136-188 kubenswrapper[2570]: W0909 03:00:00.460720    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb455032_74e3_4bd4_9211_78a7650d7c77.slice/crio-8dd16e890c6a90a53d6ea5f245c52dc6af55280a7d91e591926c2a36a6558ecb WatchSource:0}: Error finding container 8dd16e890c6a90a53d6ea5f245c52dc6af55280a7d91e591926c2a36a6558ecb: Status 404 returned error can't find the container with id 8dd16e890c6a90a53d6ea5f245c52dc6af55280a7d91e591926c2a36a6558ecb
Sep 09 03:00:00.470910 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:00.470884    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-621ac9-29815380-vv7qq"]
Sep 09 03:00:00.473938 ip-10-0-136-188 kubenswrapper[2570]: W0909 03:00:00.473916    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4490eb10_bc57_4534_b829_ccdb613adca6.slice/crio-f48a20de607529789175da509dd2e5b98a364d151dc59aedb4a36e9e971cf90f WatchSource:0}: Error finding container f48a20de607529789175da509dd2e5b98a364d151dc59aedb4a36e9e971cf90f: Status 404 returned error can't find the container with id f48a20de607529789175da509dd2e5b98a364d151dc59aedb4a36e9e971cf90f
Sep 09 03:00:01.030810 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:01.030713    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-56dbc0-29815380-brrrn" event={"ID":"bb455032-74e3-4bd4-9211-78a7650d7c77","Type":"ContainerStarted","Data":"6bef9fe9d93ec1697168109c5a2ffc532464f99460e87e53e02c79af7df11852"}
Sep 09 03:00:01.030810 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:01.030772    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-56dbc0-29815380-brrrn" event={"ID":"bb455032-74e3-4bd4-9211-78a7650d7c77","Type":"ContainerStarted","Data":"8dd16e890c6a90a53d6ea5f245c52dc6af55280a7d91e591926c2a36a6558ecb"}
Sep 09 03:00:01.032494 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:01.032467    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-621ac9-29815380-vv7qq" event={"ID":"4490eb10-bc57-4534-b829-ccdb613adca6","Type":"ContainerStarted","Data":"f13fa89f1c4f971a26c81022939f2e0253a01d44f7e5e8a2baabd20116c0fb88"}
Sep 09 03:00:01.032608 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:01.032507    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-621ac9-29815380-vv7qq" event={"ID":"4490eb10-bc57-4534-b829-ccdb613adca6","Type":"ContainerStarted","Data":"f48a20de607529789175da509dd2e5b98a364d151dc59aedb4a36e9e971cf90f"}
Sep 09 03:00:01.046348 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:01.046304    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-56dbc0-29815380-brrrn" podStartSLOduration=1.046285939 podStartE2EDuration="1.046285939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:00:01.045821802 +0000 UTC m=+1783.300125450" watchObservedRunningTime="2026-09-09 03:00:01.046285939 +0000 UTC m=+1783.300589574"
Sep 09 03:00:01.065345 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:01.065302    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-621ac9-29815380-vv7qq" podStartSLOduration=1.065287022 podStartE2EDuration="1.065287022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:00:01.06410583 +0000 UTC m=+1783.318409468" watchObservedRunningTime="2026-09-09 03:00:01.065287022 +0000 UTC m=+1783.319590659"
Sep 09 03:00:02.038454 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:02.038420    2570 generic.go:356] "Generic (PLEG): container finished" podID="bb455032-74e3-4bd4-9211-78a7650d7c77" containerID="6bef9fe9d93ec1697168109c5a2ffc532464f99460e87e53e02c79af7df11852" exitCode=0
Sep 09 03:00:02.038963 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:02.038504    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-56dbc0-29815380-brrrn" event={"ID":"bb455032-74e3-4bd4-9211-78a7650d7c77","Type":"ContainerDied","Data":"6bef9fe9d93ec1697168109c5a2ffc532464f99460e87e53e02c79af7df11852"}
Sep 09 03:00:02.040017 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:02.039997    2570 generic.go:356] "Generic (PLEG): container finished" podID="4490eb10-bc57-4534-b829-ccdb613adca6" containerID="f13fa89f1c4f971a26c81022939f2e0253a01d44f7e5e8a2baabd20116c0fb88" exitCode=0
Sep 09 03:00:02.040080 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:02.040072    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-621ac9-29815380-vv7qq" event={"ID":"4490eb10-bc57-4534-b829-ccdb613adca6","Type":"ContainerDied","Data":"f13fa89f1c4f971a26c81022939f2e0253a01d44f7e5e8a2baabd20116c0fb88"}
Sep 09 03:00:03.200877 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:03.200851    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-621ac9-29815380-vv7qq"
Sep 09 03:00:03.204063 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:03.204042    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-56dbc0-29815380-brrrn"
Sep 09 03:00:04.052276 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:04.052236    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-56dbc0-29815380-brrrn"
Sep 09 03:00:04.052471 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:04.052238    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-56dbc0-29815380-brrrn" event={"ID":"bb455032-74e3-4bd4-9211-78a7650d7c77","Type":"ContainerDied","Data":"8dd16e890c6a90a53d6ea5f245c52dc6af55280a7d91e591926c2a36a6558ecb"}
Sep 09 03:00:04.052471 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:04.052358    2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8dd16e890c6a90a53d6ea5f245c52dc6af55280a7d91e591926c2a36a6558ecb"
Sep 09 03:00:04.053567 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:04.053542    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-621ac9-29815380-vv7qq"
Sep 09 03:00:04.053688 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:04.053544    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-621ac9-29815380-vv7qq" event={"ID":"4490eb10-bc57-4534-b829-ccdb613adca6","Type":"ContainerDied","Data":"f48a20de607529789175da509dd2e5b98a364d151dc59aedb4a36e9e971cf90f"}
Sep 09 03:00:04.053688 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:04.053658    2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f48a20de607529789175da509dd2e5b98a364d151dc59aedb4a36e9e971cf90f"
Sep 09 03:00:05.431935 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:05.431907    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2085ee-openshift-default-7fb75b87b8-nggqw"
Sep 09 03:00:05.520469 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:05.520370    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d70260f-d945-437c-9fe6-432561b37d70-workload-socket\") pod \"8d70260f-d945-437c-9fe6-432561b37d70\" (UID: \"8d70260f-d945-437c-9fe6-432561b37d70\") "
Sep 09 03:00:05.520469 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:05.520415    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d70260f-d945-437c-9fe6-432561b37d70-istio-data\") pod \"8d70260f-d945-437c-9fe6-432561b37d70\" (UID: \"8d70260f-d945-437c-9fe6-432561b37d70\") "
Sep 09 03:00:05.520705 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:05.520479    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d70260f-d945-437c-9fe6-432561b37d70-istiod-ca-cert\") pod \"8d70260f-d945-437c-9fe6-432561b37d70\" (UID: \"8d70260f-d945-437c-9fe6-432561b37d70\") "
Sep 09 03:00:05.520705 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:05.520510    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d70260f-d945-437c-9fe6-432561b37d70-istio-envoy\") pod \"8d70260f-d945-437c-9fe6-432561b37d70\" (UID: \"8d70260f-d945-437c-9fe6-432561b37d70\") "
Sep 09 03:00:05.520705 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:05.520531    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d70260f-d945-437c-9fe6-432561b37d70-workload-certs\") pod \"8d70260f-d945-437c-9fe6-432561b37d70\" (UID: \"8d70260f-d945-437c-9fe6-432561b37d70\") "
Sep 09 03:00:05.520705 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:05.520556    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d70260f-d945-437c-9fe6-432561b37d70-istio-token\") pod \"8d70260f-d945-437c-9fe6-432561b37d70\" (UID: \"8d70260f-d945-437c-9fe6-432561b37d70\") "
Sep 09 03:00:05.520705 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:05.520587    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d70260f-d945-437c-9fe6-432561b37d70-credential-socket\") pod \"8d70260f-d945-437c-9fe6-432561b37d70\" (UID: \"8d70260f-d945-437c-9fe6-432561b37d70\") "
Sep 09 03:00:05.520705 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:05.520612    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tpjdm\" (UniqueName: \"kubernetes.io/projected/8d70260f-d945-437c-9fe6-432561b37d70-kube-api-access-tpjdm\") pod \"8d70260f-d945-437c-9fe6-432561b37d70\" (UID: \"8d70260f-d945-437c-9fe6-432561b37d70\") "
Sep 09 03:00:05.520705 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:05.520685    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d70260f-d945-437c-9fe6-432561b37d70-istio-podinfo\") pod \"8d70260f-d945-437c-9fe6-432561b37d70\" (UID: \"8d70260f-d945-437c-9fe6-432561b37d70\") "
Sep 09 03:00:05.521117 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:05.520695    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d70260f-d945-437c-9fe6-432561b37d70-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8d70260f-d945-437c-9fe6-432561b37d70" (UID: "8d70260f-d945-437c-9fe6-432561b37d70"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:00:05.521117 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:05.520799    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d70260f-d945-437c-9fe6-432561b37d70-istio-data" (OuterVolumeSpecName: "istio-data") pod "8d70260f-d945-437c-9fe6-432561b37d70" (UID: "8d70260f-d945-437c-9fe6-432561b37d70"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:00:05.521117 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:05.520811    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d70260f-d945-437c-9fe6-432561b37d70-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8d70260f-d945-437c-9fe6-432561b37d70" (UID: "8d70260f-d945-437c-9fe6-432561b37d70"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:00:05.521117 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:05.520932    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d70260f-d945-437c-9fe6-432561b37d70-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8d70260f-d945-437c-9fe6-432561b37d70" (UID: "8d70260f-d945-437c-9fe6-432561b37d70"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:00:05.521117 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:05.520944    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d70260f-d945-437c-9fe6-432561b37d70-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8d70260f-d945-437c-9fe6-432561b37d70" (UID: "8d70260f-d945-437c-9fe6-432561b37d70"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:00:05.521117 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:05.521094    2570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d70260f-d945-437c-9fe6-432561b37d70-workload-socket\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 03:00:05.521117 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:05.521117    2570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d70260f-d945-437c-9fe6-432561b37d70-istio-data\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 03:00:05.521381 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:05.521136    2570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d70260f-d945-437c-9fe6-432561b37d70-istiod-ca-cert\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 03:00:05.521381 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:05.521151    2570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d70260f-d945-437c-9fe6-432561b37d70-workload-certs\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 03:00:05.521381 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:05.521163    2570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d70260f-d945-437c-9fe6-432561b37d70-credential-socket\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 03:00:05.523003 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:05.522952    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d70260f-d945-437c-9fe6-432561b37d70-kube-api-access-tpjdm" (OuterVolumeSpecName: "kube-api-access-tpjdm") pod "8d70260f-d945-437c-9fe6-432561b37d70" (UID: "8d70260f-d945-437c-9fe6-432561b37d70"). InnerVolumeSpecName "kube-api-access-tpjdm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:00:05.523003 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:05.522901    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d70260f-d945-437c-9fe6-432561b37d70-istio-token" (OuterVolumeSpecName: "istio-token") pod "8d70260f-d945-437c-9fe6-432561b37d70" (UID: "8d70260f-d945-437c-9fe6-432561b37d70"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:00:05.523227 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:05.523211    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d70260f-d945-437c-9fe6-432561b37d70-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8d70260f-d945-437c-9fe6-432561b37d70" (UID: "8d70260f-d945-437c-9fe6-432561b37d70"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:00:05.523284 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:05.523261    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8d70260f-d945-437c-9fe6-432561b37d70-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8d70260f-d945-437c-9fe6-432561b37d70" (UID: "8d70260f-d945-437c-9fe6-432561b37d70"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 03:00:05.622279 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:05.622220    2570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d70260f-d945-437c-9fe6-432561b37d70-istio-envoy\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 03:00:05.622279 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:05.622268    2570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d70260f-d945-437c-9fe6-432561b37d70-istio-token\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 03:00:05.622279 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:05.622279    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpjdm\" (UniqueName: \"kubernetes.io/projected/8d70260f-d945-437c-9fe6-432561b37d70-kube-api-access-tpjdm\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 03:00:05.622279 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:05.622289    2570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d70260f-d945-437c-9fe6-432561b37d70-istio-podinfo\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 03:00:06.065908 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:06.065867    2570 generic.go:356] "Generic (PLEG): container finished" podID="8d70260f-d945-437c-9fe6-432561b37d70" containerID="17a49bac98918b39d38558e406879f2c301302438aa34df53d3977cf94cce3e0" exitCode=0
Sep 09 03:00:06.066095 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:06.065922    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2085ee-openshift-default-7fb75b87b8-nggqw"
Sep 09 03:00:06.066095 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:06.065960    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2085ee-openshift-default-7fb75b87b8-nggqw" event={"ID":"8d70260f-d945-437c-9fe6-432561b37d70","Type":"ContainerDied","Data":"17a49bac98918b39d38558e406879f2c301302438aa34df53d3977cf94cce3e0"}
Sep 09 03:00:06.066095 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:06.066045    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2085ee-openshift-default-7fb75b87b8-nggqw" event={"ID":"8d70260f-d945-437c-9fe6-432561b37d70","Type":"ContainerDied","Data":"e02d9a449ecdf1646a82926af13c253577952abb03f3f2ccfba9b2e7d268b09d"}
Sep 09 03:00:06.066095 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:06.066070    2570 scope.go:117] "RemoveContainer" containerID="17a49bac98918b39d38558e406879f2c301302438aa34df53d3977cf94cce3e0"
Sep 09 03:00:06.075778 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:06.075757    2570 scope.go:117] "RemoveContainer" containerID="96b1ad4a2f9bd24d32375e0c1b7c47cee11e44b638c9a4fa747c1291d08cda17"
Sep 09 03:00:06.083780 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:06.083762    2570 scope.go:117] "RemoveContainer" containerID="17a49bac98918b39d38558e406879f2c301302438aa34df53d3977cf94cce3e0"
Sep 09 03:00:06.084075 ip-10-0-136-188 kubenswrapper[2570]: E0909 03:00:06.084052    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17a49bac98918b39d38558e406879f2c301302438aa34df53d3977cf94cce3e0\": container with ID starting with 17a49bac98918b39d38558e406879f2c301302438aa34df53d3977cf94cce3e0 not found: ID does not exist" containerID="17a49bac98918b39d38558e406879f2c301302438aa34df53d3977cf94cce3e0"
Sep 09 03:00:06.084157 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:06.084088    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17a49bac98918b39d38558e406879f2c301302438aa34df53d3977cf94cce3e0"} err="failed to get container status \"17a49bac98918b39d38558e406879f2c301302438aa34df53d3977cf94cce3e0\": rpc error: code = NotFound desc = could not find container \"17a49bac98918b39d38558e406879f2c301302438aa34df53d3977cf94cce3e0\": container with ID starting with 17a49bac98918b39d38558e406879f2c301302438aa34df53d3977cf94cce3e0 not found: ID does not exist"
Sep 09 03:00:06.084157 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:06.084111    2570 scope.go:117] "RemoveContainer" containerID="96b1ad4a2f9bd24d32375e0c1b7c47cee11e44b638c9a4fa747c1291d08cda17"
Sep 09 03:00:06.084373 ip-10-0-136-188 kubenswrapper[2570]: E0909 03:00:06.084357    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96b1ad4a2f9bd24d32375e0c1b7c47cee11e44b638c9a4fa747c1291d08cda17\": container with ID starting with 96b1ad4a2f9bd24d32375e0c1b7c47cee11e44b638c9a4fa747c1291d08cda17 not found: ID does not exist" containerID="96b1ad4a2f9bd24d32375e0c1b7c47cee11e44b638c9a4fa747c1291d08cda17"
Sep 09 03:00:06.084413 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:06.084379    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96b1ad4a2f9bd24d32375e0c1b7c47cee11e44b638c9a4fa747c1291d08cda17"} err="failed to get container status \"96b1ad4a2f9bd24d32375e0c1b7c47cee11e44b638c9a4fa747c1291d08cda17\": rpc error: code = NotFound desc = could not find container \"96b1ad4a2f9bd24d32375e0c1b7c47cee11e44b638c9a4fa747c1291d08cda17\": container with ID starting with 96b1ad4a2f9bd24d32375e0c1b7c47cee11e44b638c9a4fa747c1291d08cda17 not found: ID does not exist"
Sep 09 03:00:06.089289 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:06.089265    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-2085ee-openshift-default-7fb75b87b8-nggqw"]
Sep 09 03:00:06.092556 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:06.092531    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-2085ee-openshift-default-7fb75b87b8-nggqw"]
Sep 09 03:00:06.223615 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:06.223573    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d70260f-d945-437c-9fe6-432561b37d70" path="/var/lib/kubelet/pods/8d70260f-d945-437c-9fe6-432561b37d70/volumes"
Sep 09 03:00:11.089987 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:11.089887    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-621ac9-29815380-vv7qq"]
Sep 09 03:00:11.095035 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:11.094952    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-621ac9-29815380-vv7qq"]
Sep 09 03:00:12.222488 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:12.222447    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4490eb10-bc57-4534-b829-ccdb613adca6" path="/var/lib/kubelet/pods/4490eb10-bc57-4534-b829-ccdb613adca6/volumes"
Sep 09 03:00:14.248488 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:14.248453    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-99qjf"]
Sep 09 03:00:17.965616 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:17.965578    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-621ac9-openshift-default-549f68cf56-dl2vr"]
Sep 09 03:00:17.966114 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:17.965814    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-621ac9-openshift-default-549f68cf56-dl2vr" podUID="72a3ba64-14d5-4c42-b59a-b70e1296f504" containerName="istio-proxy" containerID="cri-o://c99421f21bfd42fc7ead7bc143bc03314d67e7cd00e23731916e267050876cba" gracePeriod=30
Sep 09 03:00:23.157230 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:23.157137    2570 generic.go:356] "Generic (PLEG): container finished" podID="72a3ba64-14d5-4c42-b59a-b70e1296f504" containerID="c99421f21bfd42fc7ead7bc143bc03314d67e7cd00e23731916e267050876cba" exitCode=0
Sep 09 03:00:23.157640 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:23.157314    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-621ac9-openshift-default-549f68cf56-dl2vr" event={"ID":"72a3ba64-14d5-4c42-b59a-b70e1296f504","Type":"ContainerDied","Data":"c99421f21bfd42fc7ead7bc143bc03314d67e7cd00e23731916e267050876cba"}
Sep 09 03:00:23.157640 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:23.157405    2570 scope.go:117] "RemoveContainer" containerID="eae43e49b12736a2c2425cb562a6b984b6db4aa379939f253bdbeeff49780550"
Sep 09 03:00:23.218989 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:23.218955    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-621ac9-openshift-default-549f68cf56-dl2vr"
Sep 09 03:00:23.281519 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:23.281452    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jvglw\" (UniqueName: \"kubernetes.io/projected/72a3ba64-14d5-4c42-b59a-b70e1296f504-kube-api-access-jvglw\") pod \"72a3ba64-14d5-4c42-b59a-b70e1296f504\" (UID: \"72a3ba64-14d5-4c42-b59a-b70e1296f504\") "
Sep 09 03:00:23.281519 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:23.281511    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72a3ba64-14d5-4c42-b59a-b70e1296f504-workload-certs\") pod \"72a3ba64-14d5-4c42-b59a-b70e1296f504\" (UID: \"72a3ba64-14d5-4c42-b59a-b70e1296f504\") "
Sep 09 03:00:23.281691 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:23.281542    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72a3ba64-14d5-4c42-b59a-b70e1296f504-workload-socket\") pod \"72a3ba64-14d5-4c42-b59a-b70e1296f504\" (UID: \"72a3ba64-14d5-4c42-b59a-b70e1296f504\") "
Sep 09 03:00:23.281691 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:23.281637    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72a3ba64-14d5-4c42-b59a-b70e1296f504-istio-token\") pod \"72a3ba64-14d5-4c42-b59a-b70e1296f504\" (UID: \"72a3ba64-14d5-4c42-b59a-b70e1296f504\") "
Sep 09 03:00:23.281691 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:23.281684    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72a3ba64-14d5-4c42-b59a-b70e1296f504-istio-data\") pod \"72a3ba64-14d5-4c42-b59a-b70e1296f504\" (UID: \"72a3ba64-14d5-4c42-b59a-b70e1296f504\") "
Sep 09 03:00:23.281853 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:23.281713    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72a3ba64-14d5-4c42-b59a-b70e1296f504-istiod-ca-cert\") pod \"72a3ba64-14d5-4c42-b59a-b70e1296f504\" (UID: \"72a3ba64-14d5-4c42-b59a-b70e1296f504\") "
Sep 09 03:00:23.281853 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:23.281764    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72a3ba64-14d5-4c42-b59a-b70e1296f504-istio-podinfo\") pod \"72a3ba64-14d5-4c42-b59a-b70e1296f504\" (UID: \"72a3ba64-14d5-4c42-b59a-b70e1296f504\") "
Sep 09 03:00:23.281853 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:23.281792    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72a3ba64-14d5-4c42-b59a-b70e1296f504-istio-envoy\") pod \"72a3ba64-14d5-4c42-b59a-b70e1296f504\" (UID: \"72a3ba64-14d5-4c42-b59a-b70e1296f504\") "
Sep 09 03:00:23.281853 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:23.281819    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72a3ba64-14d5-4c42-b59a-b70e1296f504-credential-socket\") pod \"72a3ba64-14d5-4c42-b59a-b70e1296f504\" (UID: \"72a3ba64-14d5-4c42-b59a-b70e1296f504\") "
Sep 09 03:00:23.281853 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:23.281822    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72a3ba64-14d5-4c42-b59a-b70e1296f504-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "72a3ba64-14d5-4c42-b59a-b70e1296f504" (UID: "72a3ba64-14d5-4c42-b59a-b70e1296f504"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:00:23.282135 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:23.281839    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72a3ba64-14d5-4c42-b59a-b70e1296f504-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "72a3ba64-14d5-4c42-b59a-b70e1296f504" (UID: "72a3ba64-14d5-4c42-b59a-b70e1296f504"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:00:23.282135 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:23.281916    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72a3ba64-14d5-4c42-b59a-b70e1296f504-istio-data" (OuterVolumeSpecName: "istio-data") pod "72a3ba64-14d5-4c42-b59a-b70e1296f504" (UID: "72a3ba64-14d5-4c42-b59a-b70e1296f504"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:00:23.282135 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:23.282073    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72a3ba64-14d5-4c42-b59a-b70e1296f504-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "72a3ba64-14d5-4c42-b59a-b70e1296f504" (UID: "72a3ba64-14d5-4c42-b59a-b70e1296f504"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:00:23.282254 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:23.282180    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72a3ba64-14d5-4c42-b59a-b70e1296f504-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "72a3ba64-14d5-4c42-b59a-b70e1296f504" (UID: "72a3ba64-14d5-4c42-b59a-b70e1296f504"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:00:23.282291 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:23.282257    2570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72a3ba64-14d5-4c42-b59a-b70e1296f504-credential-socket\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 03:00:23.282291 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:23.282275    2570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72a3ba64-14d5-4c42-b59a-b70e1296f504-workload-certs\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 03:00:23.282364 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:23.282289    2570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72a3ba64-14d5-4c42-b59a-b70e1296f504-workload-socket\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 03:00:23.282364 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:23.282305    2570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72a3ba64-14d5-4c42-b59a-b70e1296f504-istio-data\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 03:00:23.282364 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:23.282318    2570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72a3ba64-14d5-4c42-b59a-b70e1296f504-istiod-ca-cert\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 03:00:23.283970 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:23.283935    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72a3ba64-14d5-4c42-b59a-b70e1296f504-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "72a3ba64-14d5-4c42-b59a-b70e1296f504" (UID: "72a3ba64-14d5-4c42-b59a-b70e1296f504"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:00:23.283970 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:23.283954    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72a3ba64-14d5-4c42-b59a-b70e1296f504-kube-api-access-jvglw" (OuterVolumeSpecName: "kube-api-access-jvglw") pod "72a3ba64-14d5-4c42-b59a-b70e1296f504" (UID: "72a3ba64-14d5-4c42-b59a-b70e1296f504"). InnerVolumeSpecName "kube-api-access-jvglw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:00:23.284173 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:23.284024    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72a3ba64-14d5-4c42-b59a-b70e1296f504-istio-token" (OuterVolumeSpecName: "istio-token") pod "72a3ba64-14d5-4c42-b59a-b70e1296f504" (UID: "72a3ba64-14d5-4c42-b59a-b70e1296f504"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:00:23.284173 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:23.284044    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/72a3ba64-14d5-4c42-b59a-b70e1296f504-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "72a3ba64-14d5-4c42-b59a-b70e1296f504" (UID: "72a3ba64-14d5-4c42-b59a-b70e1296f504"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 03:00:23.383396 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:23.383363    2570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72a3ba64-14d5-4c42-b59a-b70e1296f504-istio-token\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 03:00:23.383396 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:23.383389    2570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72a3ba64-14d5-4c42-b59a-b70e1296f504-istio-podinfo\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 03:00:23.383396 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:23.383399    2570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72a3ba64-14d5-4c42-b59a-b70e1296f504-istio-envoy\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 03:00:23.383605 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:23.383408    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvglw\" (UniqueName: \"kubernetes.io/projected/72a3ba64-14d5-4c42-b59a-b70e1296f504-kube-api-access-jvglw\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 03:00:24.164252 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:24.164222    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-621ac9-openshift-default-549f68cf56-dl2vr"
Sep 09 03:00:24.164659 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:24.164253    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-621ac9-openshift-default-549f68cf56-dl2vr" event={"ID":"72a3ba64-14d5-4c42-b59a-b70e1296f504","Type":"ContainerDied","Data":"6a2d402d7196dd0a05893e60f84d2d3719c378426f8d860c2d7d24931e52436a"}
Sep 09 03:00:24.164659 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:24.164310    2570 scope.go:117] "RemoveContainer" containerID="c99421f21bfd42fc7ead7bc143bc03314d67e7cd00e23731916e267050876cba"
Sep 09 03:00:24.192314 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:24.192272    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-621ac9-openshift-default-549f68cf56-dl2vr"]
Sep 09 03:00:24.193830 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:24.193806    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-621ac9-openshift-default-549f68cf56-dl2vr"]
Sep 09 03:00:24.222359 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:24.222330    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72a3ba64-14d5-4c42-b59a-b70e1296f504" path="/var/lib/kubelet/pods/72a3ba64-14d5-4c42-b59a-b70e1296f504/volumes"
Sep 09 03:00:29.039025 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:29.038995    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-56dbc0-29815380-brrrn"]
Sep 09 03:00:29.040830 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:29.040804    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-56dbc0-29815380-brrrn"]
Sep 09 03:00:30.223601 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:30.223333    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb455032-74e3-4bd4-9211-78a7650d7c77" path="/var/lib/kubelet/pods/bb455032-74e3-4bd4-9211-78a7650d7c77/volumes"
Sep 09 03:00:32.052525 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:32.052485    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-99qjf"]
Sep 09 03:00:49.449297 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:49.449260    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-99qjf"]
Sep 09 03:00:53.531199 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:53.531160    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-c173ad-openshift-default-55b8b5b556-hbcds"]
Sep 09 03:00:53.531656 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:53.531380    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-c173ad-openshift-default-55b8b5b556-hbcds" podUID="efa812e6-5ca9-45a7-9249-7d1cb2de135a" containerName="istio-proxy" containerID="cri-o://cc983b1a0496fcd61e0a89c916ea32bc27970aa2e60c44160150c873f1cee4b8" gracePeriod=30
Sep 09 03:00:58.778064 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:58.778036    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-c173ad-openshift-default-55b8b5b556-hbcds"
Sep 09 03:00:58.919178 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:58.919145    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efa812e6-5ca9-45a7-9249-7d1cb2de135a-istiod-ca-cert\") pod \"efa812e6-5ca9-45a7-9249-7d1cb2de135a\" (UID: \"efa812e6-5ca9-45a7-9249-7d1cb2de135a\") "
Sep 09 03:00:58.919352 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:58.919190    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efa812e6-5ca9-45a7-9249-7d1cb2de135a-istio-podinfo\") pod \"efa812e6-5ca9-45a7-9249-7d1cb2de135a\" (UID: \"efa812e6-5ca9-45a7-9249-7d1cb2de135a\") "
Sep 09 03:00:58.919352 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:58.919225    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efa812e6-5ca9-45a7-9249-7d1cb2de135a-workload-socket\") pod \"efa812e6-5ca9-45a7-9249-7d1cb2de135a\" (UID: \"efa812e6-5ca9-45a7-9249-7d1cb2de135a\") "
Sep 09 03:00:58.919352 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:58.919240    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efa812e6-5ca9-45a7-9249-7d1cb2de135a-istio-token\") pod \"efa812e6-5ca9-45a7-9249-7d1cb2de135a\" (UID: \"efa812e6-5ca9-45a7-9249-7d1cb2de135a\") "
Sep 09 03:00:58.919352 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:58.919265    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efa812e6-5ca9-45a7-9249-7d1cb2de135a-workload-certs\") pod \"efa812e6-5ca9-45a7-9249-7d1cb2de135a\" (UID: \"efa812e6-5ca9-45a7-9249-7d1cb2de135a\") "
Sep 09 03:00:58.919352 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:58.919282    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7jgjr\" (UniqueName: \"kubernetes.io/projected/efa812e6-5ca9-45a7-9249-7d1cb2de135a-kube-api-access-7jgjr\") pod \"efa812e6-5ca9-45a7-9249-7d1cb2de135a\" (UID: \"efa812e6-5ca9-45a7-9249-7d1cb2de135a\") "
Sep 09 03:00:58.919352 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:58.919345    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efa812e6-5ca9-45a7-9249-7d1cb2de135a-istio-data\") pod \"efa812e6-5ca9-45a7-9249-7d1cb2de135a\" (UID: \"efa812e6-5ca9-45a7-9249-7d1cb2de135a\") "
Sep 09 03:00:58.919675 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:58.919397    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efa812e6-5ca9-45a7-9249-7d1cb2de135a-istio-envoy\") pod \"efa812e6-5ca9-45a7-9249-7d1cb2de135a\" (UID: \"efa812e6-5ca9-45a7-9249-7d1cb2de135a\") "
Sep 09 03:00:58.919675 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:58.919425    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efa812e6-5ca9-45a7-9249-7d1cb2de135a-credential-socket\") pod \"efa812e6-5ca9-45a7-9249-7d1cb2de135a\" (UID: \"efa812e6-5ca9-45a7-9249-7d1cb2de135a\") "
Sep 09 03:00:58.919675 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:58.919540    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efa812e6-5ca9-45a7-9249-7d1cb2de135a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "efa812e6-5ca9-45a7-9249-7d1cb2de135a" (UID: "efa812e6-5ca9-45a7-9249-7d1cb2de135a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:00:58.919675 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:58.919658    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efa812e6-5ca9-45a7-9249-7d1cb2de135a-istio-data" (OuterVolumeSpecName: "istio-data") pod "efa812e6-5ca9-45a7-9249-7d1cb2de135a" (UID: "efa812e6-5ca9-45a7-9249-7d1cb2de135a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:00:58.919675 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:58.919668    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efa812e6-5ca9-45a7-9249-7d1cb2de135a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "efa812e6-5ca9-45a7-9249-7d1cb2de135a" (UID: "efa812e6-5ca9-45a7-9249-7d1cb2de135a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 03:00:58.919894 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:58.919756    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efa812e6-5ca9-45a7-9249-7d1cb2de135a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "efa812e6-5ca9-45a7-9249-7d1cb2de135a" (UID: "efa812e6-5ca9-45a7-9249-7d1cb2de135a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:00:58.919894 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:58.919850    2570 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efa812e6-5ca9-45a7-9249-7d1cb2de135a-workload-socket\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 03:00:58.919894 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:58.919890    2570 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efa812e6-5ca9-45a7-9249-7d1cb2de135a-istio-data\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 03:00:58.920034 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:58.919904    2570 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efa812e6-5ca9-45a7-9249-7d1cb2de135a-credential-socket\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 03:00:58.920034 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:58.919916    2570 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efa812e6-5ca9-45a7-9249-7d1cb2de135a-istiod-ca-cert\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 03:00:58.920034 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:58.919919    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efa812e6-5ca9-45a7-9249-7d1cb2de135a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "efa812e6-5ca9-45a7-9249-7d1cb2de135a" (UID: "efa812e6-5ca9-45a7-9249-7d1cb2de135a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:00:58.921609 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:58.921577    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/efa812e6-5ca9-45a7-9249-7d1cb2de135a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "efa812e6-5ca9-45a7-9249-7d1cb2de135a" (UID: "efa812e6-5ca9-45a7-9249-7d1cb2de135a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 03:00:58.921710 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:58.921588    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efa812e6-5ca9-45a7-9249-7d1cb2de135a-istio-token" (OuterVolumeSpecName: "istio-token") pod "efa812e6-5ca9-45a7-9249-7d1cb2de135a" (UID: "efa812e6-5ca9-45a7-9249-7d1cb2de135a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:00:58.922117 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:58.922088    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efa812e6-5ca9-45a7-9249-7d1cb2de135a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "efa812e6-5ca9-45a7-9249-7d1cb2de135a" (UID: "efa812e6-5ca9-45a7-9249-7d1cb2de135a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:00:58.922117 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:58.922099    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efa812e6-5ca9-45a7-9249-7d1cb2de135a-kube-api-access-7jgjr" (OuterVolumeSpecName: "kube-api-access-7jgjr") pod "efa812e6-5ca9-45a7-9249-7d1cb2de135a" (UID: "efa812e6-5ca9-45a7-9249-7d1cb2de135a"). InnerVolumeSpecName "kube-api-access-7jgjr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:00:59.020945 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:59.020919    2570 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efa812e6-5ca9-45a7-9249-7d1cb2de135a-istio-podinfo\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 03:00:59.020945 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:59.020942    2570 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efa812e6-5ca9-45a7-9249-7d1cb2de135a-istio-token\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 03:00:59.020945 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:59.020952    2570 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efa812e6-5ca9-45a7-9249-7d1cb2de135a-workload-certs\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 03:00:59.021151 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:59.020960    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jgjr\" (UniqueName: \"kubernetes.io/projected/efa812e6-5ca9-45a7-9249-7d1cb2de135a-kube-api-access-7jgjr\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 03:00:59.021151 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:59.020969    2570 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efa812e6-5ca9-45a7-9249-7d1cb2de135a-istio-envoy\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 03:00:59.335635 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:59.335603    2570 generic.go:356] "Generic (PLEG): container finished" podID="efa812e6-5ca9-45a7-9249-7d1cb2de135a" containerID="cc983b1a0496fcd61e0a89c916ea32bc27970aa2e60c44160150c873f1cee4b8" exitCode=0
Sep 09 03:00:59.335795 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:59.335652    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-c173ad-openshift-default-55b8b5b556-hbcds"
Sep 09 03:00:59.335795 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:59.335660    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-c173ad-openshift-default-55b8b5b556-hbcds" event={"ID":"efa812e6-5ca9-45a7-9249-7d1cb2de135a","Type":"ContainerDied","Data":"cc983b1a0496fcd61e0a89c916ea32bc27970aa2e60c44160150c873f1cee4b8"}
Sep 09 03:00:59.335795 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:59.335703    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-c173ad-openshift-default-55b8b5b556-hbcds" event={"ID":"efa812e6-5ca9-45a7-9249-7d1cb2de135a","Type":"ContainerDied","Data":"66c2ce14b8c269cee3c9551010d192272a8c61970b4da4c4ac3613936dd2e204"}
Sep 09 03:00:59.335795 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:59.335723    2570 scope.go:117] "RemoveContainer" containerID="cc983b1a0496fcd61e0a89c916ea32bc27970aa2e60c44160150c873f1cee4b8"
Sep 09 03:00:59.345141 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:59.345124    2570 scope.go:117] "RemoveContainer" containerID="cbe941bdd98570226bc8dd220760a5541aeb1b6f0bf82071cabdeed4db1ef090"
Sep 09 03:00:59.352927 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:59.352912    2570 scope.go:117] "RemoveContainer" containerID="cc983b1a0496fcd61e0a89c916ea32bc27970aa2e60c44160150c873f1cee4b8"
Sep 09 03:00:59.353175 ip-10-0-136-188 kubenswrapper[2570]: E0909 03:00:59.353156    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc983b1a0496fcd61e0a89c916ea32bc27970aa2e60c44160150c873f1cee4b8\": container with ID starting with cc983b1a0496fcd61e0a89c916ea32bc27970aa2e60c44160150c873f1cee4b8 not found: ID does not exist" containerID="cc983b1a0496fcd61e0a89c916ea32bc27970aa2e60c44160150c873f1cee4b8"
Sep 09 03:00:59.353234 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:59.353182    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc983b1a0496fcd61e0a89c916ea32bc27970aa2e60c44160150c873f1cee4b8"} err="failed to get container status \"cc983b1a0496fcd61e0a89c916ea32bc27970aa2e60c44160150c873f1cee4b8\": rpc error: code = NotFound desc = could not find container \"cc983b1a0496fcd61e0a89c916ea32bc27970aa2e60c44160150c873f1cee4b8\": container with ID starting with cc983b1a0496fcd61e0a89c916ea32bc27970aa2e60c44160150c873f1cee4b8 not found: ID does not exist"
Sep 09 03:00:59.353234 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:59.353197    2570 scope.go:117] "RemoveContainer" containerID="cbe941bdd98570226bc8dd220760a5541aeb1b6f0bf82071cabdeed4db1ef090"
Sep 09 03:00:59.353397 ip-10-0-136-188 kubenswrapper[2570]: E0909 03:00:59.353383    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbe941bdd98570226bc8dd220760a5541aeb1b6f0bf82071cabdeed4db1ef090\": container with ID starting with cbe941bdd98570226bc8dd220760a5541aeb1b6f0bf82071cabdeed4db1ef090 not found: ID does not exist" containerID="cbe941bdd98570226bc8dd220760a5541aeb1b6f0bf82071cabdeed4db1ef090"
Sep 09 03:00:59.353440 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:59.353401    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbe941bdd98570226bc8dd220760a5541aeb1b6f0bf82071cabdeed4db1ef090"} err="failed to get container status \"cbe941bdd98570226bc8dd220760a5541aeb1b6f0bf82071cabdeed4db1ef090\": rpc error: code = NotFound desc = could not find container \"cbe941bdd98570226bc8dd220760a5541aeb1b6f0bf82071cabdeed4db1ef090\": container with ID starting with cbe941bdd98570226bc8dd220760a5541aeb1b6f0bf82071cabdeed4db1ef090 not found: ID does not exist"
Sep 09 03:00:59.361204 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:59.361184    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-c173ad-openshift-default-55b8b5b556-hbcds"]
Sep 09 03:00:59.364853 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:00:59.364833    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-c173ad-openshift-default-55b8b5b556-hbcds"]
Sep 09 03:01:00.224629 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:01:00.224595    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="efa812e6-5ca9-45a7-9249-7d1cb2de135a" path="/var/lib/kubelet/pods/efa812e6-5ca9-45a7-9249-7d1cb2de135a/volumes"
Sep 09 03:01:01.945275 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:01:01.945238    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-99qjf"]
Sep 09 03:02:18.608449 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:02:18.608416    2570 scope.go:117] "RemoveContainer" containerID="11d2e797335d0d584000678d078e2a864758213ea9730859e41e849f3e5d60c2"
Sep 09 03:02:23.440729 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:02:23.440688    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-99qjf"]
Sep 09 03:02:47.242127 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:02:47.242088    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-99qjf"]
Sep 09 03:02:55.139543 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:02:55.139503    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-99qjf"]
Sep 09 03:03:03.340902 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:03:03.340862    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-99qjf"]
Sep 09 03:03:18.629514 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:03:18.629434    2570 scope.go:117] "RemoveContainer" containerID="0b5473fcd1e047e4d01dd02bae3d1e66b43cc51fe2edb030c21727f9463f6a69"
Sep 09 03:03:18.637436 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:03:18.637414    2570 scope.go:117] "RemoveContainer" containerID="4ba54d49846d53cc891454474cee7dd8983d27972e878bfbb2d2db7f645f3032"
Sep 09 03:03:20.946135 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:03:20.946097    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-99qjf"]
Sep 09 03:03:26.948048 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:03:26.948005    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-99qjf"]
Sep 09 03:04:58.913524 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:04:58.913489    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-99qjf"]
Sep 09 03:05:04.605811 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:05:04.605768    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-99qjf"]
Sep 09 03:05:13.019696 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:05:13.019656    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-99qjf"]
Sep 09 03:05:18.664601 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:05:18.664568    2570 scope.go:117] "RemoveContainer" containerID="9a0c073a851ba8d4302d9332fe7741dfcb400e35c99e619a4b68d59b369b659c"
Sep 09 03:05:20.243021 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:05:20.242959    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-40e3fdc1-kserve-6dd46d7c95-g8fp2"]
Sep 09 03:05:20.243495 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:05:20.243473    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efa812e6-5ca9-45a7-9249-7d1cb2de135a" containerName="istio-proxy"
Sep 09 03:05:20.243573 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:05:20.243499    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="efa812e6-5ca9-45a7-9249-7d1cb2de135a" containerName="istio-proxy"
Sep 09 03:05:20.243573 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:05:20.243511    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72a3ba64-14d5-4c42-b59a-b70e1296f504" containerName="istio-proxy"
Sep 09 03:05:20.243573 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:05:20.243519    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="72a3ba64-14d5-4c42-b59a-b70e1296f504" containerName="istio-proxy"
Sep 09 03:05:20.243573 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:05:20.243533    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb455032-74e3-4bd4-9211-78a7650d7c77" containerName="cleanup"
Sep 09 03:05:20.243573 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:05:20.243541    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb455032-74e3-4bd4-9211-78a7650d7c77" containerName="cleanup"
Sep 09 03:05:20.243573 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:05:20.243559    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d70260f-d945-437c-9fe6-432561b37d70" containerName="istio-proxy"
Sep 09 03:05:20.243573 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:05:20.243567    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d70260f-d945-437c-9fe6-432561b37d70" containerName="istio-proxy"
Sep 09 03:05:20.243918 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:05:20.243595    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4490eb10-bc57-4534-b829-ccdb613adca6" containerName="cleanup"
Sep 09 03:05:20.243918 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:05:20.243603    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="4490eb10-bc57-4534-b829-ccdb613adca6" containerName="cleanup"
Sep 09 03:05:20.243918 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:05:20.243690    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="efa812e6-5ca9-45a7-9249-7d1cb2de135a" containerName="istio-proxy"
Sep 09 03:05:20.243918 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:05:20.243704    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="efa812e6-5ca9-45a7-9249-7d1cb2de135a" containerName="istio-proxy"
Sep 09 03:05:20.243918 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:05:20.243714    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d70260f-d945-437c-9fe6-432561b37d70" containerName="istio-proxy"
Sep 09 03:05:20.243918 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:05:20.243724    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="72a3ba64-14d5-4c42-b59a-b70e1296f504" containerName="istio-proxy"
Sep 09 03:05:20.243918 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:05:20.243734    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="4490eb10-bc57-4534-b829-ccdb613adca6" containerName="cleanup"
Sep 09 03:05:20.243918 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:05:20.243742    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb455032-74e3-4bd4-9211-78a7650d7c77" containerName="cleanup"
Sep 09 03:05:20.248263 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:05:20.248241    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-40e3fdc1-kserve-6dd46d7c95-g8fp2"
Sep 09 03:05:20.251241 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:05:20.251219    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-40e3fdc1-kserve-self-signed-certs\""
Sep 09 03:05:20.256359 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:05:20.256337    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-40e3fdc1-kserve-6dd46d7c95-g8fp2"]
Sep 09 03:05:20.307414 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:05:20.307379    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d4b5268-1ca1-4635-88a2-261fc17553f2-dshm\") pod \"e2e-same-id-b-40e3fdc1-kserve-6dd46d7c95-g8fp2\" (UID: \"4d4b5268-1ca1-4635-88a2-261fc17553f2\") " pod="llm/e2e-same-id-b-40e3fdc1-kserve-6dd46d7c95-g8fp2"
Sep 09 03:05:20.307618 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:05:20.307438    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d4b5268-1ca1-4635-88a2-261fc17553f2-home\") pod \"e2e-same-id-b-40e3fdc1-kserve-6dd46d7c95-g8fp2\" (UID: \"4d4b5268-1ca1-4635-88a2-261fc17553f2\") " pod="llm/e2e-same-id-b-40e3fdc1-kserve-6dd46d7c95-g8fp2"
Sep 09 03:05:20.307618 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:05:20.307520    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d4b5268-1ca1-4635-88a2-261fc17553f2-model-cache\") pod \"e2e-same-id-b-40e3fdc1-kserve-6dd46d7c95-g8fp2\" (UID: \"4d4b5268-1ca1-4635-88a2-261fc17553f2\") " pod="llm/e2e-same-id-b-40e3fdc1-kserve-6dd46d7c95-g8fp2"
Sep 09 03:05:20.307618 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:05:20.307566    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d4b5268-1ca1-4635-88a2-261fc17553f2-tls-certs\") pod \"e2e-same-id-b-40e3fdc1-kserve-6dd46d7c95-g8fp2\" (UID: \"4d4b5268-1ca1-4635-88a2-261fc17553f2\") " pod="llm/e2e-same-id-b-40e3fdc1-kserve-6dd46d7c95-g8fp2"
Sep 09 03:05:20.307769 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:05:20.307618    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d4b5268-1ca1-4635-88a2-261fc17553f2-tmp-dir\") pod \"e2e-same-id-b-40e3fdc1-kserve-6dd46d7c95-g8fp2\" (UID: \"4d4b5268-1ca1-4635-88a2-261fc17553f2\") " pod="llm/e2e-same-id-b-40e3fdc1-kserve-6dd46d7c95-g8fp2"
Sep 09 03:05:20.307769 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:05:20.307686    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpx26\" (UniqueName: \"kubernetes.io/projected/4d4b5268-1ca1-4635-88a2-261fc17553f2-kube-api-access-qpx26\") pod \"e2e-same-id-b-40e3fdc1-kserve-6dd46d7c95-g8fp2\" (UID: \"4d4b5268-1ca1-4635-88a2-261fc17553f2\") " pod="llm/e2e-same-id-b-40e3fdc1-kserve-6dd46d7c95-g8fp2"
Sep 09 03:05:20.409117 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:05:20.409075    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d4b5268-1ca1-4635-88a2-261fc17553f2-dshm\") pod \"e2e-same-id-b-40e3fdc1-kserve-6dd46d7c95-g8fp2\" (UID: \"4d4b5268-1ca1-4635-88a2-261fc17553f2\") " pod="llm/e2e-same-id-b-40e3fdc1-kserve-6dd46d7c95-g8fp2"
Sep 09 03:05:20.409329 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:05:20.409160    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d4b5268-1ca1-4635-88a2-261fc17553f2-home\") pod \"e2e-same-id-b-40e3fdc1-kserve-6dd46d7c95-g8fp2\" (UID: \"4d4b5268-1ca1-4635-88a2-261fc17553f2\") " pod="llm/e2e-same-id-b-40e3fdc1-kserve-6dd46d7c95-g8fp2"
Sep 09 03:05:20.409329 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:05:20.409191    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d4b5268-1ca1-4635-88a2-261fc17553f2-model-cache\") pod \"e2e-same-id-b-40e3fdc1-kserve-6dd46d7c95-g8fp2\" (UID: \"4d4b5268-1ca1-4635-88a2-261fc17553f2\") " pod="llm/e2e-same-id-b-40e3fdc1-kserve-6dd46d7c95-g8fp2"
Sep 09 03:05:20.409329 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:05:20.409217    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d4b5268-1ca1-4635-88a2-261fc17553f2-tls-certs\") pod \"e2e-same-id-b-40e3fdc1-kserve-6dd46d7c95-g8fp2\" (UID: \"4d4b5268-1ca1-4635-88a2-261fc17553f2\") " pod="llm/e2e-same-id-b-40e3fdc1-kserve-6dd46d7c95-g8fp2"
Sep 09 03:05:20.409329 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:05:20.409247    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d4b5268-1ca1-4635-88a2-261fc17553f2-tmp-dir\") pod \"e2e-same-id-b-40e3fdc1-kserve-6dd46d7c95-g8fp2\" (UID: \"4d4b5268-1ca1-4635-88a2-261fc17553f2\") " pod="llm/e2e-same-id-b-40e3fdc1-kserve-6dd46d7c95-g8fp2"
Sep 09 03:05:20.409329 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:05:20.409293    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpx26\" (UniqueName: \"kubernetes.io/projected/4d4b5268-1ca1-4635-88a2-261fc17553f2-kube-api-access-qpx26\") pod \"e2e-same-id-b-40e3fdc1-kserve-6dd46d7c95-g8fp2\" (UID: \"4d4b5268-1ca1-4635-88a2-261fc17553f2\") " pod="llm/e2e-same-id-b-40e3fdc1-kserve-6dd46d7c95-g8fp2"
Sep 09 03:05:20.409655 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:05:20.409619    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d4b5268-1ca1-4635-88a2-261fc17553f2-home\") pod \"e2e-same-id-b-40e3fdc1-kserve-6dd46d7c95-g8fp2\" (UID: \"4d4b5268-1ca1-4635-88a2-261fc17553f2\") " pod="llm/e2e-same-id-b-40e3fdc1-kserve-6dd46d7c95-g8fp2"
Sep 09 03:05:20.409723 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:05:20.409699    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d4b5268-1ca1-4635-88a2-261fc17553f2-tmp-dir\") pod \"e2e-same-id-b-40e3fdc1-kserve-6dd46d7c95-g8fp2\" (UID: \"4d4b5268-1ca1-4635-88a2-261fc17553f2\") " pod="llm/e2e-same-id-b-40e3fdc1-kserve-6dd46d7c95-g8fp2"
Sep 09 03:05:20.409926 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:05:20.409905    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d4b5268-1ca1-4635-88a2-261fc17553f2-model-cache\") pod \"e2e-same-id-b-40e3fdc1-kserve-6dd46d7c95-g8fp2\" (UID: \"4d4b5268-1ca1-4635-88a2-261fc17553f2\") " pod="llm/e2e-same-id-b-40e3fdc1-kserve-6dd46d7c95-g8fp2"
Sep 09 03:05:20.411544 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:05:20.411518    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d4b5268-1ca1-4635-88a2-261fc17553f2-dshm\") pod \"e2e-same-id-b-40e3fdc1-kserve-6dd46d7c95-g8fp2\" (UID: \"4d4b5268-1ca1-4635-88a2-261fc17553f2\") " pod="llm/e2e-same-id-b-40e3fdc1-kserve-6dd46d7c95-g8fp2"
Sep 09 03:05:20.412100 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:05:20.412080    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d4b5268-1ca1-4635-88a2-261fc17553f2-tls-certs\") pod \"e2e-same-id-b-40e3fdc1-kserve-6dd46d7c95-g8fp2\" (UID: \"4d4b5268-1ca1-4635-88a2-261fc17553f2\") " pod="llm/e2e-same-id-b-40e3fdc1-kserve-6dd46d7c95-g8fp2"
Sep 09 03:05:20.417966 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:05:20.417944    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpx26\" (UniqueName: \"kubernetes.io/projected/4d4b5268-1ca1-4635-88a2-261fc17553f2-kube-api-access-qpx26\") pod \"e2e-same-id-b-40e3fdc1-kserve-6dd46d7c95-g8fp2\" (UID: \"4d4b5268-1ca1-4635-88a2-261fc17553f2\") " pod="llm/e2e-same-id-b-40e3fdc1-kserve-6dd46d7c95-g8fp2"
Sep 09 03:05:20.558445 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:05:20.558353    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-40e3fdc1-kserve-6dd46d7c95-g8fp2"
Sep 09 03:05:20.830638 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:05:20.830610    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-40e3fdc1-kserve-6dd46d7c95-g8fp2"]
Sep 09 03:05:20.832948 ip-10-0-136-188 kubenswrapper[2570]: W0909 03:05:20.832915    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4d4b5268_1ca1_4635_88a2_261fc17553f2.slice/crio-f8e4e08c176f7aaef35e78a5b64fed4d673678973074b2d512aa917db0b5edf8 WatchSource:0}: Error finding container f8e4e08c176f7aaef35e78a5b64fed4d673678973074b2d512aa917db0b5edf8: Status 404 returned error can't find the container with id f8e4e08c176f7aaef35e78a5b64fed4d673678973074b2d512aa917db0b5edf8
Sep 09 03:05:21.581774 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:05:21.581736    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-40e3fdc1-kserve-6dd46d7c95-g8fp2" event={"ID":"4d4b5268-1ca1-4635-88a2-261fc17553f2","Type":"ContainerStarted","Data":"df20f12cc35c81a2ea244537e3ab6461d0f092e4742889e8169694121f488492"}
Sep 09 03:05:21.581774 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:05:21.581776    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-40e3fdc1-kserve-6dd46d7c95-g8fp2" event={"ID":"4d4b5268-1ca1-4635-88a2-261fc17553f2","Type":"ContainerStarted","Data":"f8e4e08c176f7aaef35e78a5b64fed4d673678973074b2d512aa917db0b5edf8"}
Sep 09 03:05:21.602385 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:05:21.602328    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-40e3fdc1-kserve-6dd46d7c95-g8fp2" podStartSLOduration=1.6023127879999999 podStartE2EDuration="1.602312788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:05:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:05:21.599292475 +0000 UTC m=+2103.853596105" watchObservedRunningTime="2026-09-09 03:05:21.602312788 +0000 UTC m=+2103.856616423"
Sep 09 03:05:23.012639 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:05:23.012602    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-99qjf"]
Sep 09 03:05:30.559394 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:05:30.559357    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-40e3fdc1-kserve-6dd46d7c95-g8fp2"
Sep 09 03:05:30.560039 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:05:30.559415    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-40e3fdc1-kserve-6dd46d7c95-g8fp2"
Sep 09 03:05:30.571556 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:05:30.571531    2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-40e3fdc1-kserve-6dd46d7c95-g8fp2"
Sep 09 03:05:30.636242 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:05:30.636209    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-40e3fdc1-kserve-6dd46d7c95-g8fp2"
Sep 09 03:05:44.121476 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:05:44.121433    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-40e3fdc1-kserve-6dd46d7c95-g8fp2"]
Sep 09 03:05:46.407715 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:05:46.407680    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-99qjf"]
Sep 09 03:05:54.622479 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:05:54.622445    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-99qjf"]
Sep 09 03:05:59.140962 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:05:59.140925    2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-40e3fdc1-kserve-6dd46d7c95-g8fp2" podUID="4d4b5268-1ca1-4635-88a2-261fc17553f2" containerName="main" containerID="cri-o://df20f12cc35c81a2ea244537e3ab6461d0f092e4742889e8169694121f488492" gracePeriod=45
Sep 09 03:05:59.379530 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:05:59.379506    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-40e3fdc1-kserve-6dd46d7c95-g8fp2"
Sep 09 03:05:59.452773 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:05:59.452706    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d4b5268-1ca1-4635-88a2-261fc17553f2-dshm\") pod \"4d4b5268-1ca1-4635-88a2-261fc17553f2\" (UID: \"4d4b5268-1ca1-4635-88a2-261fc17553f2\") "
Sep 09 03:05:59.452773 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:05:59.452764    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qpx26\" (UniqueName: \"kubernetes.io/projected/4d4b5268-1ca1-4635-88a2-261fc17553f2-kube-api-access-qpx26\") pod \"4d4b5268-1ca1-4635-88a2-261fc17553f2\" (UID: \"4d4b5268-1ca1-4635-88a2-261fc17553f2\") "
Sep 09 03:05:59.452944 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:05:59.452786    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d4b5268-1ca1-4635-88a2-261fc17553f2-tls-certs\") pod \"4d4b5268-1ca1-4635-88a2-261fc17553f2\" (UID: \"4d4b5268-1ca1-4635-88a2-261fc17553f2\") "
Sep 09 03:05:59.452944 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:05:59.452805    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d4b5268-1ca1-4635-88a2-261fc17553f2-tmp-dir\") pod \"4d4b5268-1ca1-4635-88a2-261fc17553f2\" (UID: \"4d4b5268-1ca1-4635-88a2-261fc17553f2\") "
Sep 09 03:05:59.452944 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:05:59.452852    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d4b5268-1ca1-4635-88a2-261fc17553f2-model-cache\") pod \"4d4b5268-1ca1-4635-88a2-261fc17553f2\" (UID: \"4d4b5268-1ca1-4635-88a2-261fc17553f2\") "
Sep 09 03:05:59.452944 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:05:59.452887    2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d4b5268-1ca1-4635-88a2-261fc17553f2-home\") pod \"4d4b5268-1ca1-4635-88a2-261fc17553f2\" (UID: \"4d4b5268-1ca1-4635-88a2-261fc17553f2\") "
Sep 09 03:05:59.453193 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:05:59.453150    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d4b5268-1ca1-4635-88a2-261fc17553f2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4d4b5268-1ca1-4635-88a2-261fc17553f2" (UID: "4d4b5268-1ca1-4635-88a2-261fc17553f2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:05:59.453193 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:05:59.453171    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d4b5268-1ca1-4635-88a2-261fc17553f2-model-cache" (OuterVolumeSpecName: "model-cache") pod "4d4b5268-1ca1-4635-88a2-261fc17553f2" (UID: "4d4b5268-1ca1-4635-88a2-261fc17553f2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:05:59.453281 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:05:59.453189    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d4b5268-1ca1-4635-88a2-261fc17553f2-home" (OuterVolumeSpecName: "home") pod "4d4b5268-1ca1-4635-88a2-261fc17553f2" (UID: "4d4b5268-1ca1-4635-88a2-261fc17553f2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:05:59.453281 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:05:59.453254    2570 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d4b5268-1ca1-4635-88a2-261fc17553f2-tmp-dir\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 03:05:59.454886 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:05:59.454862    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d4b5268-1ca1-4635-88a2-261fc17553f2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4d4b5268-1ca1-4635-88a2-261fc17553f2" (UID: "4d4b5268-1ca1-4635-88a2-261fc17553f2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 03:05:59.455330 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:05:59.455312    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d4b5268-1ca1-4635-88a2-261fc17553f2-dshm" (OuterVolumeSpecName: "dshm") pod "4d4b5268-1ca1-4635-88a2-261fc17553f2" (UID: "4d4b5268-1ca1-4635-88a2-261fc17553f2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 03:05:59.455381 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:05:59.455335    2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d4b5268-1ca1-4635-88a2-261fc17553f2-kube-api-access-qpx26" (OuterVolumeSpecName: "kube-api-access-qpx26") pod "4d4b5268-1ca1-4635-88a2-261fc17553f2" (UID: "4d4b5268-1ca1-4635-88a2-261fc17553f2"). InnerVolumeSpecName "kube-api-access-qpx26". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 03:05:59.553914 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:05:59.553885    2570 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d4b5268-1ca1-4635-88a2-261fc17553f2-dshm\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 03:05:59.553914 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:05:59.553910    2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpx26\" (UniqueName: \"kubernetes.io/projected/4d4b5268-1ca1-4635-88a2-261fc17553f2-kube-api-access-qpx26\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 03:05:59.554094 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:05:59.553921    2570 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d4b5268-1ca1-4635-88a2-261fc17553f2-tls-certs\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 03:05:59.554094 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:05:59.553931    2570 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d4b5268-1ca1-4635-88a2-261fc17553f2-model-cache\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 03:05:59.554094 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:05:59.553938    2570 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d4b5268-1ca1-4635-88a2-261fc17553f2-home\") on node \"ip-10-0-136-188.ec2.internal\" DevicePath \"\""
Sep 09 03:05:59.759824 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:05:59.759748    2570 generic.go:356] "Generic (PLEG): container finished" podID="4d4b5268-1ca1-4635-88a2-261fc17553f2" containerID="df20f12cc35c81a2ea244537e3ab6461d0f092e4742889e8169694121f488492" exitCode=0
Sep 09 03:05:59.759824 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:05:59.759786    2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-40e3fdc1-kserve-6dd46d7c95-g8fp2"
Sep 09 03:05:59.760011 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:05:59.759832    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-40e3fdc1-kserve-6dd46d7c95-g8fp2" event={"ID":"4d4b5268-1ca1-4635-88a2-261fc17553f2","Type":"ContainerDied","Data":"df20f12cc35c81a2ea244537e3ab6461d0f092e4742889e8169694121f488492"}
Sep 09 03:05:59.760011 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:05:59.759873    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-40e3fdc1-kserve-6dd46d7c95-g8fp2" event={"ID":"4d4b5268-1ca1-4635-88a2-261fc17553f2","Type":"ContainerDied","Data":"f8e4e08c176f7aaef35e78a5b64fed4d673678973074b2d512aa917db0b5edf8"}
Sep 09 03:05:59.760011 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:05:59.759894    2570 scope.go:117] "RemoveContainer" containerID="df20f12cc35c81a2ea244537e3ab6461d0f092e4742889e8169694121f488492"
Sep 09 03:05:59.769630 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:05:59.769613    2570 scope.go:117] "RemoveContainer" containerID="df20f12cc35c81a2ea244537e3ab6461d0f092e4742889e8169694121f488492"
Sep 09 03:05:59.769873 ip-10-0-136-188 kubenswrapper[2570]: E0909 03:05:59.769851    2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df20f12cc35c81a2ea244537e3ab6461d0f092e4742889e8169694121f488492\": container with ID starting with df20f12cc35c81a2ea244537e3ab6461d0f092e4742889e8169694121f488492 not found: ID does not exist" containerID="df20f12cc35c81a2ea244537e3ab6461d0f092e4742889e8169694121f488492"
Sep 09 03:05:59.769936 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:05:59.769879    2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df20f12cc35c81a2ea244537e3ab6461d0f092e4742889e8169694121f488492"} err="failed to get container status \"df20f12cc35c81a2ea244537e3ab6461d0f092e4742889e8169694121f488492\": rpc error: code = NotFound desc = could not find container \"df20f12cc35c81a2ea244537e3ab6461d0f092e4742889e8169694121f488492\": container with ID starting with df20f12cc35c81a2ea244537e3ab6461d0f092e4742889e8169694121f488492 not found: ID does not exist"
Sep 09 03:05:59.782511 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:05:59.782489    2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-40e3fdc1-kserve-6dd46d7c95-g8fp2"]
Sep 09 03:05:59.786216 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:05:59.786196    2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-40e3fdc1-kserve-6dd46d7c95-g8fp2"]
Sep 09 03:06:00.222647 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:06:00.222613    2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d4b5268-1ca1-4635-88a2-261fc17553f2" path="/var/lib/kubelet/pods/4d4b5268-1ca1-4635-88a2-261fc17553f2/volumes"
Sep 09 03:06:02.611759 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:06:02.611717    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-99qjf"]
Sep 09 03:06:09.712556 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:06:09.712524    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-99qjf"]
Sep 09 03:06:16.512893 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:06:16.512811    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-99qjf"]
Sep 09 03:06:18.684745 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:06:18.684708    2570 scope.go:117] "RemoveContainer" containerID="6bef9fe9d93ec1697168109c5a2ffc532464f99460e87e53e02c79af7df11852"
Sep 09 03:06:18.694467 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:06:18.694442    2570 scope.go:117] "RemoveContainer" containerID="f13fa89f1c4f971a26c81022939f2e0253a01d44f7e5e8a2baabd20116c0fb88"
Sep 09 03:06:22.407672 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:06:22.407636    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-99qjf"]
Sep 09 03:06:29.221277 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:06:29.221243    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-99qjf"]
Sep 09 03:06:32.151285 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:06:32.151248    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-59dbcfbf54-bsb94_30658216-f113-4e7a-b255-233300d755a9/authorino/0.log"
Sep 09 03:06:38.008350 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:06:38.008322    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-7z594_5ab4a0e6-a9fd-40a4-b403-964a44380c43/server/0.log"
Sep 09 03:06:38.146549 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:06:38.146517    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-mx5jk_256e03f1-1f0b-491a-8381-1280c89caa9f/manager/0.log"
Sep 09 03:06:38.512879 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:06:38.512851    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6dfc788bdd-djpwc_e24b4e11-a947-453d-9760-d7e27016e656/copy-manifests/0.log"
Sep 09 03:06:38.520654 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:06:38.520621    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6dfc788bdd-djpwc_e24b4e11-a947-453d-9760-d7e27016e656/manager/0.log"
Sep 09 03:06:40.145299 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:06:40.145271    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-59dbcfbf54-bsb94_30658216-f113-4e7a-b255-233300d755a9/authorino/0.log"
Sep 09 03:06:40.275313 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:06:40.275282    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-j8zkb_a46e6a0e-0beb-4dc2-be59-d82b625dbed6/manager/0.log"
Sep 09 03:06:40.522880 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:06:40.522792    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-r2j5r_c90ca26e-f6b3-4e41-9458-5a64bb1db802/kuadrant-console-plugin/0.log"
Sep 09 03:06:40.789569 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:06:40.789480    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-99qjf_e4119393-5203-4016-b923-7950e6111467/limitador/0.log"
Sep 09 03:06:41.310663 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:06:41.310625    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-gxnf5_af6f2234-9c64-45a6-9129-c708d33ad253/istio-proxy/0.log"
Sep 09 03:06:41.773278 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:06:41.773247    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-lfhw7_797e5120-d8b6-4405-8beb-6d48f3391f65/istio-proxy/1.log"
Sep 09 03:06:42.140863 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:06:42.140830    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-69c66d4b4-pl2k4_d5dad8e0-d4e8-4325-a274-a134fc6959a6/router/0.log"
Sep 09 03:06:42.731042 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:06:42.731013    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-snsjr_c287727b-944e-4401-8697-896eee8a9e71/main/0.log"
Sep 09 03:06:42.986851 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:06:42.986776    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fhsd6k_d58ee39c-1a09-4a8b-b02c-6d38a7b84255/main/0.log"
Sep 09 03:06:43.235495 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:06:43.235470    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-9bxfs_c6aab1e4-77bc-415b-bb29-d2885f6edb0d/main/0.log"
Sep 09 03:06:48.983909 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:06:48.983876    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-qh97t/must-gather-5jq2v"]
Sep 09 03:06:48.984301 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:06:48.984257    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efa812e6-5ca9-45a7-9249-7d1cb2de135a" containerName="istio-proxy"
Sep 09 03:06:48.984301 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:06:48.984270    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="efa812e6-5ca9-45a7-9249-7d1cb2de135a" containerName="istio-proxy"
Sep 09 03:06:48.984301 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:06:48.984279    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72a3ba64-14d5-4c42-b59a-b70e1296f504" containerName="istio-proxy"
Sep 09 03:06:48.984301 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:06:48.984285    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="72a3ba64-14d5-4c42-b59a-b70e1296f504" containerName="istio-proxy"
Sep 09 03:06:48.984301 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:06:48.984296    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d70260f-d945-437c-9fe6-432561b37d70" containerName="istio-proxy"
Sep 09 03:06:48.984301 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:06:48.984302    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d70260f-d945-437c-9fe6-432561b37d70" containerName="istio-proxy"
Sep 09 03:06:48.984496 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:06:48.984314    2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d4b5268-1ca1-4635-88a2-261fc17553f2" containerName="main"
Sep 09 03:06:48.984496 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:06:48.984320    2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d4b5268-1ca1-4635-88a2-261fc17553f2" containerName="main"
Sep 09 03:06:48.984496 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:06:48.984371    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="72a3ba64-14d5-4c42-b59a-b70e1296f504" containerName="istio-proxy"
Sep 09 03:06:48.984496 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:06:48.984380    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d4b5268-1ca1-4635-88a2-261fc17553f2" containerName="main"
Sep 09 03:06:48.984496 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:06:48.984388    2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d70260f-d945-437c-9fe6-432561b37d70" containerName="istio-proxy"
Sep 09 03:06:48.987518 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:06:48.987498    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qh97t/must-gather-5jq2v"
Sep 09 03:06:48.990903 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:06:48.990881    2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-qh97t\"/\"default-dockercfg-kzxwx\""
Sep 09 03:06:48.990903 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:06:48.990896    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qh97t\"/\"openshift-service-ca.crt\""
Sep 09 03:06:48.991099 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:06:48.990882    2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qh97t\"/\"kube-root-ca.crt\""
Sep 09 03:06:48.999750 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:06:48.999731    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qh97t/must-gather-5jq2v"]
Sep 09 03:06:49.082602 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:06:49.082578    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n655g\" (UniqueName: \"kubernetes.io/projected/8351ed0f-1915-4af0-aaa1-50d36924919b-kube-api-access-n655g\") pod \"must-gather-5jq2v\" (UID: \"8351ed0f-1915-4af0-aaa1-50d36924919b\") " pod="openshift-must-gather-qh97t/must-gather-5jq2v"
Sep 09 03:06:49.082730 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:06:49.082636    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/8351ed0f-1915-4af0-aaa1-50d36924919b-must-gather-output\") pod \"must-gather-5jq2v\" (UID: \"8351ed0f-1915-4af0-aaa1-50d36924919b\") " pod="openshift-must-gather-qh97t/must-gather-5jq2v"
Sep 09 03:06:49.183416 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:06:49.183391    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/8351ed0f-1915-4af0-aaa1-50d36924919b-must-gather-output\") pod \"must-gather-5jq2v\" (UID: \"8351ed0f-1915-4af0-aaa1-50d36924919b\") " pod="openshift-must-gather-qh97t/must-gather-5jq2v"
Sep 09 03:06:49.183510 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:06:49.183482    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n655g\" (UniqueName: \"kubernetes.io/projected/8351ed0f-1915-4af0-aaa1-50d36924919b-kube-api-access-n655g\") pod \"must-gather-5jq2v\" (UID: \"8351ed0f-1915-4af0-aaa1-50d36924919b\") " pod="openshift-must-gather-qh97t/must-gather-5jq2v"
Sep 09 03:06:49.183729 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:06:49.183712    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/8351ed0f-1915-4af0-aaa1-50d36924919b-must-gather-output\") pod \"must-gather-5jq2v\" (UID: \"8351ed0f-1915-4af0-aaa1-50d36924919b\") " pod="openshift-must-gather-qh97t/must-gather-5jq2v"
Sep 09 03:06:49.191997 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:06:49.191964    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n655g\" (UniqueName: \"kubernetes.io/projected/8351ed0f-1915-4af0-aaa1-50d36924919b-kube-api-access-n655g\") pod \"must-gather-5jq2v\" (UID: \"8351ed0f-1915-4af0-aaa1-50d36924919b\") " pod="openshift-must-gather-qh97t/must-gather-5jq2v"
Sep 09 03:06:49.296402 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:06:49.296326    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qh97t/must-gather-5jq2v"
Sep 09 03:06:49.567447 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:06:49.567419    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qh97t/must-gather-5jq2v"]
Sep 09 03:06:49.569453 ip-10-0-136-188 kubenswrapper[2570]: W0909 03:06:49.569408    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8351ed0f_1915_4af0_aaa1_50d36924919b.slice/crio-617a4434377fb856c72b137c177ca71857bf9d6b2aabb46a37c244c1b877ace5 WatchSource:0}: Error finding container 617a4434377fb856c72b137c177ca71857bf9d6b2aabb46a37c244c1b877ace5: Status 404 returned error can't find the container with id 617a4434377fb856c72b137c177ca71857bf9d6b2aabb46a37c244c1b877ace5
Sep 09 03:06:49.571917 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:06:49.571900    2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 03:06:49.997651 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:06:49.997616    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qh97t/must-gather-5jq2v" event={"ID":"8351ed0f-1915-4af0-aaa1-50d36924919b","Type":"ContainerStarted","Data":"617a4434377fb856c72b137c177ca71857bf9d6b2aabb46a37c244c1b877ace5"}
Sep 09 03:06:51.006718 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:06:51.006679    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qh97t/must-gather-5jq2v" event={"ID":"8351ed0f-1915-4af0-aaa1-50d36924919b","Type":"ContainerStarted","Data":"fe36106d1ed7cd31c6e55161caaec56e666447697abe6c87f4d155f51c8b94a1"}
Sep 09 03:06:51.007154 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:06:51.006745    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qh97t/must-gather-5jq2v" event={"ID":"8351ed0f-1915-4af0-aaa1-50d36924919b","Type":"ContainerStarted","Data":"4349d7823604b79523b21662268668d9e9d5fbf985399a3bd916b24ac0429e6e"}
Sep 09 03:06:51.024249 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:06:51.024196    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-qh97t/must-gather-5jq2v" podStartSLOduration=2.211873463 podStartE2EDuration="3.024177113s" totalImagesPullingTime="812.30365ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 03:06:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:06:51.024073293 +0000 UTC m=+2193.278376931" watchObservedRunningTime="2026-09-09 03:06:51.024177113 +0000 UTC m=+2193.278480751"
Sep 09 03:06:51.997497 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:06:51.997466    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-952h4_37ef684a-ce44-43fc-a803-519ce7d4d225/global-pull-secret-syncer/0.log"
Sep 09 03:06:52.096035 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:06:52.096002    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-kxvb7_679d664f-a0b2-4ad9-94e4-014aef8f2e16/konnectivity-agent/0.log"
Sep 09 03:06:52.145756 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:06:52.145717    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-188.ec2.internal_871e64cd6b8e47e52f74676ffa7cfe0c/haproxy/0.log"
Sep 09 03:06:56.726664 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:06:56.726626    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-59dbcfbf54-bsb94_30658216-f113-4e7a-b255-233300d755a9/authorino/0.log"
Sep 09 03:06:56.768333 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:06:56.767779    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-j8zkb_a46e6a0e-0beb-4dc2-be59-d82b625dbed6/manager/0.log"
Sep 09 03:06:56.826666 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:06:56.826634    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-r2j5r_c90ca26e-f6b3-4e41-9458-5a64bb1db802/kuadrant-console-plugin/0.log"
Sep 09 03:06:56.910208 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:06:56.910174    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-99qjf_e4119393-5203-4016-b923-7950e6111467/limitador/0.log"
Sep 09 03:06:58.606938 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:06:58.606894    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cdcc96965-5g4th_c5fd6885-5b56-4f0c-b1dd-3cff5153682b/cluster-monitoring-operator/0.log"
Sep 09 03:06:58.748826 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:06:58.748767    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-786f55c85c-6dh59_60fe9c3e-cc9c-4832-b1fb-be95737a6596/metrics-server/0.log"
Sep 09 03:06:59.103605 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:06:59.103519    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-p2sz7_32562e97-9ba2-4110-83e6-6013cc7ab2c0/node-exporter/0.log"
Sep 09 03:06:59.132374 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:06:59.132350    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-p2sz7_32562e97-9ba2-4110-83e6-6013cc7ab2c0/kube-rbac-proxy/0.log"
Sep 09 03:06:59.179251 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:06:59.179215    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-p2sz7_32562e97-9ba2-4110-83e6-6013cc7ab2c0/init-textfile/0.log"
Sep 09 03:06:59.600585 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:06:59.600555    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-zxcbk_d19499c4-d8c2-4865-bf9b-51bcc6f46905/prometheus-operator/0.log"
Sep 09 03:06:59.621569 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:06:59.621506    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-zxcbk_d19499c4-d8c2-4865-bf9b-51bcc6f46905/kube-rbac-proxy/0.log"
Sep 09 03:07:01.921325 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:07:01.921287    2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-qh97t/perf-node-gather-daemonset-2tsnr"]
Sep 09 03:07:01.932475 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:07:01.932441    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qh97t/perf-node-gather-daemonset-2tsnr"
Sep 09 03:07:01.939331 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:07:01.939300    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qh97t/perf-node-gather-daemonset-2tsnr"]
Sep 09 03:07:02.043997 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:07:02.043942    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c4d29189-91a0-4808-85ad-8b8642385ea9-lib-modules\") pod \"perf-node-gather-daemonset-2tsnr\" (UID: \"c4d29189-91a0-4808-85ad-8b8642385ea9\") " pod="openshift-must-gather-qh97t/perf-node-gather-daemonset-2tsnr"
Sep 09 03:07:02.044205 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:07:02.044025    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c4d29189-91a0-4808-85ad-8b8642385ea9-sys\") pod \"perf-node-gather-daemonset-2tsnr\" (UID: \"c4d29189-91a0-4808-85ad-8b8642385ea9\") " pod="openshift-must-gather-qh97t/perf-node-gather-daemonset-2tsnr"
Sep 09 03:07:02.044205 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:07:02.044152    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2xs2\" (UniqueName: \"kubernetes.io/projected/c4d29189-91a0-4808-85ad-8b8642385ea9-kube-api-access-n2xs2\") pod \"perf-node-gather-daemonset-2tsnr\" (UID: \"c4d29189-91a0-4808-85ad-8b8642385ea9\") " pod="openshift-must-gather-qh97t/perf-node-gather-daemonset-2tsnr"
Sep 09 03:07:02.044205 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:07:02.044186    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c4d29189-91a0-4808-85ad-8b8642385ea9-podres\") pod \"perf-node-gather-daemonset-2tsnr\" (UID: \"c4d29189-91a0-4808-85ad-8b8642385ea9\") " pod="openshift-must-gather-qh97t/perf-node-gather-daemonset-2tsnr"
Sep 09 03:07:02.044383 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:07:02.044223    2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c4d29189-91a0-4808-85ad-8b8642385ea9-proc\") pod \"perf-node-gather-daemonset-2tsnr\" (UID: \"c4d29189-91a0-4808-85ad-8b8642385ea9\") " pod="openshift-must-gather-qh97t/perf-node-gather-daemonset-2tsnr"
Sep 09 03:07:02.145204 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:07:02.145173    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c4d29189-91a0-4808-85ad-8b8642385ea9-lib-modules\") pod \"perf-node-gather-daemonset-2tsnr\" (UID: \"c4d29189-91a0-4808-85ad-8b8642385ea9\") " pod="openshift-must-gather-qh97t/perf-node-gather-daemonset-2tsnr"
Sep 09 03:07:02.145396 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:07:02.145216    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c4d29189-91a0-4808-85ad-8b8642385ea9-sys\") pod \"perf-node-gather-daemonset-2tsnr\" (UID: \"c4d29189-91a0-4808-85ad-8b8642385ea9\") " pod="openshift-must-gather-qh97t/perf-node-gather-daemonset-2tsnr"
Sep 09 03:07:02.145396 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:07:02.145284    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2xs2\" (UniqueName: \"kubernetes.io/projected/c4d29189-91a0-4808-85ad-8b8642385ea9-kube-api-access-n2xs2\") pod \"perf-node-gather-daemonset-2tsnr\" (UID: \"c4d29189-91a0-4808-85ad-8b8642385ea9\") " pod="openshift-must-gather-qh97t/perf-node-gather-daemonset-2tsnr"
Sep 09 03:07:02.145396 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:07:02.145310    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c4d29189-91a0-4808-85ad-8b8642385ea9-podres\") pod \"perf-node-gather-daemonset-2tsnr\" (UID: \"c4d29189-91a0-4808-85ad-8b8642385ea9\") " pod="openshift-must-gather-qh97t/perf-node-gather-daemonset-2tsnr"
Sep 09 03:07:02.145396 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:07:02.145335    2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c4d29189-91a0-4808-85ad-8b8642385ea9-proc\") pod \"perf-node-gather-daemonset-2tsnr\" (UID: \"c4d29189-91a0-4808-85ad-8b8642385ea9\") " pod="openshift-must-gather-qh97t/perf-node-gather-daemonset-2tsnr"
Sep 09 03:07:02.145396 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:07:02.145378    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c4d29189-91a0-4808-85ad-8b8642385ea9-lib-modules\") pod \"perf-node-gather-daemonset-2tsnr\" (UID: \"c4d29189-91a0-4808-85ad-8b8642385ea9\") " pod="openshift-must-gather-qh97t/perf-node-gather-daemonset-2tsnr"
Sep 09 03:07:02.145649 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:07:02.145386    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c4d29189-91a0-4808-85ad-8b8642385ea9-sys\") pod \"perf-node-gather-daemonset-2tsnr\" (UID: \"c4d29189-91a0-4808-85ad-8b8642385ea9\") " pod="openshift-must-gather-qh97t/perf-node-gather-daemonset-2tsnr"
Sep 09 03:07:02.145649 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:07:02.145422    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c4d29189-91a0-4808-85ad-8b8642385ea9-proc\") pod \"perf-node-gather-daemonset-2tsnr\" (UID: \"c4d29189-91a0-4808-85ad-8b8642385ea9\") " pod="openshift-must-gather-qh97t/perf-node-gather-daemonset-2tsnr"
Sep 09 03:07:02.145649 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:07:02.145422    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c4d29189-91a0-4808-85ad-8b8642385ea9-podres\") pod \"perf-node-gather-daemonset-2tsnr\" (UID: \"c4d29189-91a0-4808-85ad-8b8642385ea9\") " pod="openshift-must-gather-qh97t/perf-node-gather-daemonset-2tsnr"
Sep 09 03:07:02.155956 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:07:02.155925    2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2xs2\" (UniqueName: \"kubernetes.io/projected/c4d29189-91a0-4808-85ad-8b8642385ea9-kube-api-access-n2xs2\") pod \"perf-node-gather-daemonset-2tsnr\" (UID: \"c4d29189-91a0-4808-85ad-8b8642385ea9\") " pod="openshift-must-gather-qh97t/perf-node-gather-daemonset-2tsnr"
Sep 09 03:07:02.247255 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:07:02.247167    2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qh97t/perf-node-gather-daemonset-2tsnr"
Sep 09 03:07:02.527804 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:07:02.527714    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-54fd64c946-w5jf9_bc4a6464-355e-4063-9cd8-359fe8db6cfd/console/0.log"
Sep 09 03:07:02.546119 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:07:02.546089    2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qh97t/perf-node-gather-daemonset-2tsnr"]
Sep 09 03:07:02.549333 ip-10-0-136-188 kubenswrapper[2570]: W0909 03:07:02.549295    2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podc4d29189_91a0_4808_85ad_8b8642385ea9.slice/crio-8c2f5d6239442c32a9c6443fe22875a104808e494965e714dc7a10f8868829e5 WatchSource:0}: Error finding container 8c2f5d6239442c32a9c6443fe22875a104808e494965e714dc7a10f8868829e5: Status 404 returned error can't find the container with id 8c2f5d6239442c32a9c6443fe22875a104808e494965e714dc7a10f8868829e5
Sep 09 03:07:02.599392 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:07:02.599362    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-85d98c998f-vslgh_94b74aa9-c050-458e-af76-a2c72555c71b/download-server/0.log"
Sep 09 03:07:03.098408 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:07:03.098347    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qh97t/perf-node-gather-daemonset-2tsnr" event={"ID":"c4d29189-91a0-4808-85ad-8b8642385ea9","Type":"ContainerStarted","Data":"8d3fa29c7b0c98bf711dcf804092b917f614163fe5d750779fe05207e0b6de7f"}
Sep 09 03:07:03.098881 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:07:03.098603    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-qh97t/perf-node-gather-daemonset-2tsnr"
Sep 09 03:07:03.098881 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:07:03.098662    2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qh97t/perf-node-gather-daemonset-2tsnr" event={"ID":"c4d29189-91a0-4808-85ad-8b8642385ea9","Type":"ContainerStarted","Data":"8c2f5d6239442c32a9c6443fe22875a104808e494965e714dc7a10f8868829e5"}
Sep 09 03:07:03.122497 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:07:03.122413    2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-qh97t/perf-node-gather-daemonset-2tsnr" podStartSLOduration=2.122390444 podStartE2EDuration="2.122390444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 03:07:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 03:07:03.119509242 +0000 UTC m=+2205.373812870" watchObservedRunningTime="2026-09-09 03:07:03.122390444 +0000 UTC m=+2205.376694079"
Sep 09 03:07:04.681608 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:07:04.681580    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ttpxf_47cb49af-ac6a-4f18-ad58-5b40818d5827/dns/0.log"
Sep 09 03:07:04.713429 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:07:04.713399    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ttpxf_47cb49af-ac6a-4f18-ad58-5b40818d5827/kube-rbac-proxy/0.log"
Sep 09 03:07:04.755260 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:07:04.755231    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-d7t89_90413a10-3c11-4127-b7c8-6e8ac09b1fc9/dns-node-resolver/0.log"
Sep 09 03:07:05.440625 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:07:05.440591    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-fb8975b97-h6747_87f74e41-b9f9-4e9b-81fa-6d1599a0234b/registry/0.log"
Sep 09 03:07:05.573086 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:07:05.573052    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zg4dm_721fa3c1-a7d2-4245-a7bd-76eb7facfc36/node-ca/0.log"
Sep 09 03:07:06.581155 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:07:06.581090    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-gxnf5_af6f2234-9c64-45a6-9129-c708d33ad253/istio-proxy/0.log"
Sep 09 03:07:06.885477 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:07:06.885445    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-lfhw7_797e5120-d8b6-4405-8beb-6d48f3391f65/istio-proxy/0.log"
Sep 09 03:07:06.896001 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:07:06.895949    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-lfhw7_797e5120-d8b6-4405-8beb-6d48f3391f65/istio-proxy/1.log"
Sep 09 03:07:06.998364 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:07:06.998330    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-69c66d4b4-pl2k4_d5dad8e0-d4e8-4325-a274-a134fc6959a6/router/0.log"
Sep 09 03:07:07.864439 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:07:07.864406    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-wjvs5_ec031e2f-88e3-4608-a1b7-de11b4d537a0/serve-healthcheck-canary/0.log"
Sep 09 03:07:08.324562 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:07:08.324484    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-85dqd_a00f5e8a-987c-4c31-a8a9-b6a9d713d064/insights-operator/1.log"
Sep 09 03:07:08.324755 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:07:08.324680    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-85dqd_a00f5e8a-987c-4c31-a8a9-b6a9d713d064/insights-operator/0.log"
Sep 09 03:07:08.344632 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:07:08.344609    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jqjd8_175f5821-f3e7-44b4-8f45-945ab415b1c3/kube-rbac-proxy/0.log"
Sep 09 03:07:08.364261 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:07:08.364239    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jqjd8_175f5821-f3e7-44b4-8f45-945ab415b1c3/exporter/0.log"
Sep 09 03:07:08.384467 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:07:08.384443    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jqjd8_175f5821-f3e7-44b4-8f45-945ab415b1c3/extractor/0.log"
Sep 09 03:07:09.114170 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:07:09.114127    2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-qh97t/perf-node-gather-daemonset-2tsnr"
Sep 09 03:07:10.944197 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:07:10.944143    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-7z594_5ab4a0e6-a9fd-40a4-b403-964a44380c43/server/0.log"
Sep 09 03:07:11.049732 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:07:11.049699    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-mx5jk_256e03f1-1f0b-491a-8381-1280c89caa9f/manager/0.log"
Sep 09 03:07:11.219998 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:07:11.219951    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6dfc788bdd-djpwc_e24b4e11-a947-453d-9760-d7e27016e656/manager/0.log"
Sep 09 03:07:11.242500 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:07:11.242470    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6dfc788bdd-djpwc_e24b4e11-a947-453d-9760-d7e27016e656/copy-manifests/0.log"
Sep 09 03:07:12.406880 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:07:12.406848    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-gxnf5_af6f2234-9c64-45a6-9129-c708d33ad253/istio-proxy/0.log"
Sep 09 03:07:12.561874 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:07:12.561840    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6c4bb7846b-xfr4t_32d09487-e544-4730-89eb-815e2beda070/manager/0.log"
Sep 09 03:07:12.678750 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:07:12.678559    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-lfhw7_797e5120-d8b6-4405-8beb-6d48f3391f65/istio-proxy/0.log"
Sep 09 03:07:12.690503 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:07:12.690460    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-lfhw7_797e5120-d8b6-4405-8beb-6d48f3391f65/istio-proxy/1.log"
Sep 09 03:07:12.785008 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:07:12.784984    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-69c66d4b4-pl2k4_d5dad8e0-d4e8-4325-a274-a134fc6959a6/router/0.log"
Sep 09 03:07:14.215576 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:07:14.215541    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-gxnf5_af6f2234-9c64-45a6-9129-c708d33ad253/istio-proxy/0.log"
Sep 09 03:07:14.506863 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:07:14.505895    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-lfhw7_797e5120-d8b6-4405-8beb-6d48f3391f65/istio-proxy/0.log"
Sep 09 03:07:14.516909 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:07:14.516880    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-lfhw7_797e5120-d8b6-4405-8beb-6d48f3391f65/istio-proxy/1.log"
Sep 09 03:07:14.605830 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:07:14.605803    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-69c66d4b4-pl2k4_d5dad8e0-d4e8-4325-a274-a134fc6959a6/router/0.log"
Sep 09 03:07:19.284175 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:07:19.284097    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zw227_d33ba4b9-3e00-405a-bc2f-9144dbfa5c8f/kube-multus-additional-cni-plugins/0.log"
Sep 09 03:07:19.304853 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:07:19.304820    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zw227_d33ba4b9-3e00-405a-bc2f-9144dbfa5c8f/egress-router-binary-copy/0.log"
Sep 09 03:07:19.325362 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:07:19.325333    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zw227_d33ba4b9-3e00-405a-bc2f-9144dbfa5c8f/cni-plugins/0.log"
Sep 09 03:07:19.345483 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:07:19.345452    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zw227_d33ba4b9-3e00-405a-bc2f-9144dbfa5c8f/bond-cni-plugin/0.log"
Sep 09 03:07:19.367643 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:07:19.367591    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zw227_d33ba4b9-3e00-405a-bc2f-9144dbfa5c8f/routeoverride-cni/0.log"
Sep 09 03:07:19.386597 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:07:19.386554    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zw227_d33ba4b9-3e00-405a-bc2f-9144dbfa5c8f/whereabouts-cni-bincopy/0.log"
Sep 09 03:07:19.405571 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:07:19.405545    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zw227_d33ba4b9-3e00-405a-bc2f-9144dbfa5c8f/whereabouts-cni/0.log"
Sep 09 03:07:19.499514 ip-10-0-136-188 kubenswrapper[2570]: I0909 03:07:19.499484    2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-phjk6_ddccb44f-2b0d-4cf3-9419-31b5211ab2f5/kube-multus/0.log"