Sep 10 15:25:03.365097 ip-10-0-130-30 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 10 15:25:03.365244 ip-10-0-130-30 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 10 15:25:03.365335 ip-10-0-130-30 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 10 15:25:03.365815 ip-10-0-130-30 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 10 15:25:13.556078 ip-10-0-130-30 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 10 15:25:13.556098 ip-10-0-130-30 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 93dbebef977347c29d69eab177689ae3 --
Sep 10 15:27:04.130903 ip-10-0-130-30 systemd[1]: Starting Kubernetes Kubelet...
Sep 10 15:27:04.570120 ip-10-0-130-30 kubenswrapper[2559]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 15:27:04.570120 ip-10-0-130-30 kubenswrapper[2559]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 10 15:27:04.570120 ip-10-0-130-30 kubenswrapper[2559]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 15:27:04.570120 ip-10-0-130-30 kubenswrapper[2559]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 10 15:27:04.570120 ip-10-0-130-30 kubenswrapper[2559]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 15:27:04.572018 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.571868    2559 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 10 15:27:04.577258 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.577238    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 15:27:04.577258 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.577257    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 15:27:04.577258 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.577261    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 15:27:04.577375 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.577264    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 15:27:04.577375 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.577267    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 15:27:04.577375 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.577270    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 15:27:04.577375 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.577274    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 15:27:04.577375 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.577277    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 15:27:04.577375 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.577280    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 15:27:04.577375 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.577283    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 15:27:04.577375 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.577286    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 15:27:04.577375 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.577288    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 15:27:04.577375 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.577291    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 15:27:04.577375 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.577294    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 15:27:04.577375 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.577297    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 15:27:04.577375 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.577300    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 15:27:04.577375 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.577302    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 15:27:04.577375 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.577305    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 15:27:04.577375 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.577307    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 15:27:04.577375 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.577310    2559 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 15:27:04.577375 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.577314    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 15:27:04.577375 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.577318    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 15:27:04.577375 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.577330    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 15:27:04.577866 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.577333    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 15:27:04.577866 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.577335    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 15:27:04.577866 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.577338    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 15:27:04.577866 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.577340    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 15:27:04.577866 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.577343    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 15:27:04.577866 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.577346    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 15:27:04.577866 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.577349    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 15:27:04.577866 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.577351    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 15:27:04.577866 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.577354    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 15:27:04.577866 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.577357    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 15:27:04.577866 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.577360    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 15:27:04.577866 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.577363    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 15:27:04.577866 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.577366    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 15:27:04.577866 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.577370    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 15:27:04.577866 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.577373    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 15:27:04.577866 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.577375    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 15:27:04.577866 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.577378    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 15:27:04.577866 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.577381    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 15:27:04.577866 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.577384    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 15:27:04.577866 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.577387    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 15:27:04.578462 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.577389    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 15:27:04.578462 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.577392    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 15:27:04.578462 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.577394    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 15:27:04.578462 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.577397    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 15:27:04.578462 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.577399    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 15:27:04.578462 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.577402    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 15:27:04.578462 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.577405    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 15:27:04.578462 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.577408    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 15:27:04.578462 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.577411    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 15:27:04.578462 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.577413    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 15:27:04.578462 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.577416    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 15:27:04.578462 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.577418    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 15:27:04.578462 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.577422    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 15:27:04.578462 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.577425    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 15:27:04.578462 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.577427    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 15:27:04.578462 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.577430    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 15:27:04.578462 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.577433    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 15:27:04.578462 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.577436    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 15:27:04.578462 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.577438    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 15:27:04.578939 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.577442    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 15:27:04.578939 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.577447    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 15:27:04.578939 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.577450    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 15:27:04.578939 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.577453    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 15:27:04.578939 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.577456    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 15:27:04.578939 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.577459    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 15:27:04.578939 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.577462    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 15:27:04.578939 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.577465    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 15:27:04.578939 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.577468    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 15:27:04.578939 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.577471    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 15:27:04.578939 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.577473    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 15:27:04.578939 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.577476    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 15:27:04.578939 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.577479    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 15:27:04.578939 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.577482    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 15:27:04.578939 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.577485    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 15:27:04.578939 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.577487    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 15:27:04.578939 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.577490    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 15:27:04.578939 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.577492    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 15:27:04.578939 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.577495    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 15:27:04.579435 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.577499    2559 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 15:27:04.579435 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.577501    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 15:27:04.579435 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.577504    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 15:27:04.579435 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.577506    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 15:27:04.579435 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.577509    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 15:27:04.579435 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.577512    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 15:27:04.579435 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.577514    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 15:27:04.579435 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.577527    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 15:27:04.579435 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.577530    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 15:27:04.579435 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.577533    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 15:27:04.579435 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.577537    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 15:27:04.579435 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.577540    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 15:27:04.579435 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.577542    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 15:27:04.579435 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.577545    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 15:27:04.579435 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.577548    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 15:27:04.579435 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.577550    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 15:27:04.579435 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.577554    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 15:27:04.579435 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.577557    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 15:27:04.580699 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.580685    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 15:27:04.580699 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.580697    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 15:27:04.580699 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.580701    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 15:27:04.580790 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.580704    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 15:27:04.580790 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.580708    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 15:27:04.580790 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.580711    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 15:27:04.580790 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.580714    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 15:27:04.580790 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.580716    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 15:27:04.580790 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.580719    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 15:27:04.580790 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.580722    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 15:27:04.580790 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.580726    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 15:27:04.580790 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.580728    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 15:27:04.580790 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.580731    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 15:27:04.580790 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.580733    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 15:27:04.580790 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.580736    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 15:27:04.580790 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.580739    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 15:27:04.580790 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.580742    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 15:27:04.580790 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.580744    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 15:27:04.580790 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.580747    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 15:27:04.580790 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.580750    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 15:27:04.580790 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.580753    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 15:27:04.580790 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.580755    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 15:27:04.580790 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.580758    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 15:27:04.581281 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.580762    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 15:27:04.581281 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.580764    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 15:27:04.581281 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.580767    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 15:27:04.581281 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.580770    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 15:27:04.581281 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.580772    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 15:27:04.581281 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.580775    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 15:27:04.581281 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.580778    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 15:27:04.581281 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.580780    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 15:27:04.581281 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.580783    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 15:27:04.581281 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.580785    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 15:27:04.581281 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.580788    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 15:27:04.581281 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.580791    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 15:27:04.581281 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.580794    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 15:27:04.581281 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.580797    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 15:27:04.581281 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.580799    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 15:27:04.581281 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.580802    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 15:27:04.581281 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.580805    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 15:27:04.581281 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.580808    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 15:27:04.581281 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.580810    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 15:27:04.581761 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.580813    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 15:27:04.581761 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.580816    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 15:27:04.581761 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.580818    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 15:27:04.581761 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.580821    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 15:27:04.581761 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.580823    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 15:27:04.581761 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.580826    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 15:27:04.581761 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.580828    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 15:27:04.581761 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.580831    2559 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 15:27:04.581761 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.580833    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 15:27:04.581761 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.580836    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 15:27:04.581761 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.580840    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 15:27:04.581761 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.580842    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 15:27:04.581761 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.580845    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 15:27:04.581761 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.580850    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 15:27:04.581761 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.580854    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 15:27:04.581761 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.580857    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 15:27:04.581761 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.580859    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 15:27:04.581761 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.580863    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 15:27:04.581761 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.580865    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 15:27:04.581761 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.580868    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 15:27:04.582279 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.580871    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 15:27:04.582279 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.580875    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 15:27:04.582279 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.580879    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 15:27:04.582279 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.580882    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 15:27:04.582279 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.580885    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 15:27:04.582279 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.580888    2559 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 15:27:04.582279 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.580891    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 15:27:04.582279 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.580894    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 15:27:04.582279 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.580897    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 15:27:04.582279 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.580899    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 15:27:04.582279 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.580902    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 15:27:04.582279 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.580905    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 15:27:04.582279 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.580908    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 15:27:04.582279 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.580910    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 15:27:04.582279 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.580913    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 15:27:04.582279 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.580915    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 15:27:04.582279 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.580918    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 15:27:04.582279 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.580920    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 15:27:04.582279 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.580924    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 15:27:04.583040 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.580928    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 15:27:04.583040 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.580931    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 15:27:04.583040 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.580935    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 15:27:04.583040 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.580939    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 15:27:04.583040 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.580942    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 15:27:04.583040 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.580944    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 15:27:04.583040 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.580947    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 15:27:04.583040 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.580952    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 15:27:04.583040 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.580955    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 15:27:04.583040 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.580958    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 15:27:04.583040 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.580960    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 15:27:04.583040 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.580963    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 15:27:04.583040 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.580966    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 15:27:04.583040 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.580969    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 15:27:04.583040 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.580971    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 15:27:04.583040 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.580974    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 15:27:04.583040 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.580977    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 15:27:04.583040 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.580979    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 15:27:04.583040 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.581362    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 15:27:04.583709 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.581367    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 15:27:04.583709 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.581370    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 15:27:04.583709 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.581372    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 15:27:04.583709 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.581375    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 15:27:04.583709 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.581378    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 15:27:04.583709 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.581381    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 15:27:04.583709 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.581383    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 15:27:04.583709 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.581386    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 15:27:04.583709 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.581389    2559 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 15:27:04.583709 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.581391    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 15:27:04.583709 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.581394    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 15:27:04.583709 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.581397    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 15:27:04.583709 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.581400    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 15:27:04.583709 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.581403    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 15:27:04.583709 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.581406    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 15:27:04.583709 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.581408    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 15:27:04.583709 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.581411    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 15:27:04.583709 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.581413    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 15:27:04.583709 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.581416    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 15:27:04.583709 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.581419    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 15:27:04.584206 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.581422    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 15:27:04.584206 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.581425    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 15:27:04.584206 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.581428    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 15:27:04.584206 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.581430    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 15:27:04.584206 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.581433    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 15:27:04.584206 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.581437    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 15:27:04.584206 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.581441    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 15:27:04.584206 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.581444    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 15:27:04.584206 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.581446    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 15:27:04.584206 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.581449    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 15:27:04.584206 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.581452    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 15:27:04.584206 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.581455    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 15:27:04.584206 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.581458    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 15:27:04.584206 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.581461    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 15:27:04.584206 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.581464    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 15:27:04.584206 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.581466    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 15:27:04.584206 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.581469    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 15:27:04.584206 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.581472    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 15:27:04.584206 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.581475    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 15:27:04.584699 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.581477    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 15:27:04.584699 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.581480    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 15:27:04.584699 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.581483    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 15:27:04.584699 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.581486    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 15:27:04.584699 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.581488    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 15:27:04.584699 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.581491    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 15:27:04.584699 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.581494    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 15:27:04.584699 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.581496    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 15:27:04.584699 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.581499    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 15:27:04.584699 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.581501    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 15:27:04.584699 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.581504    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 15:27:04.584699 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.581506    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 15:27:04.584699 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.581509    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 15:27:04.584699 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.581512    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 15:27:04.584699 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.581515    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 15:27:04.584699 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.581518    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 15:27:04.584699 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.581521    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 15:27:04.584699 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.581524    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 15:27:04.584699 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.581526    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 15:27:04.584699 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.581529    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 15:27:04.585214 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.581531    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 15:27:04.585214 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.581534    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 15:27:04.585214 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.581536    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 15:27:04.585214 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.581539    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 15:27:04.585214 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.581541    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 15:27:04.585214 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.581544    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 15:27:04.585214 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.581546    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 15:27:04.585214 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.581549    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 15:27:04.585214 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.581551    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 15:27:04.585214 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.581554    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 15:27:04.585214 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.581556    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 15:27:04.585214 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.581559    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 15:27:04.585214 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.581562    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 15:27:04.585214 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.581565    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 15:27:04.585214 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.581567    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 15:27:04.585214 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.581570    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 15:27:04.585214 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.581572    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 15:27:04.585214 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.581588    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 15:27:04.585214 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.581594    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 15:27:04.585681 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.581599    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 15:27:04.585681 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.581604    2559 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 15:27:04.585681 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.581607    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 15:27:04.585681 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.581610    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 15:27:04.585681 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.581614    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 15:27:04.585681 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.581618    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 15:27:04.585681 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.581622    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 15:27:04.585681 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.581625    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 15:27:04.585681 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.581629    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 15:27:04.585681 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.581631    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 15:27:04.585681 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.581634    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 15:27:04.585681 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.581637    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 15:27:04.585681 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.581639    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 15:27:04.585681 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.581642    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 15:27:04.585681 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.581645    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 15:27:04.585681 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.581647    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 15:27:04.585681 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.581650    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 15:27:04.585681 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.581652    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 15:27:04.585681 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.581655    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 15:27:04.585681 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.581657    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 15:27:04.586196 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.581742    2559 flags.go:64] FLAG: --address="0.0.0.0"
Sep 10 15:27:04.586196 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.581778    2559 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 10 15:27:04.586196 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.581784    2559 flags.go:64] FLAG: --anonymous-auth="true"
Sep 10 15:27:04.586196 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.581789    2559 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 10 15:27:04.586196 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.581794    2559 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 10 15:27:04.586196 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.581797    2559 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 10 15:27:04.586196 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.581802    2559 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 10 15:27:04.586196 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.581807    2559 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 10 15:27:04.586196 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.581810    2559 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 10 15:27:04.586196 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.581813    2559 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 10 15:27:04.586196 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.581817    2559 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 10 15:27:04.586196 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.581820    2559 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 10 15:27:04.586196 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.581823    2559 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 10 15:27:04.586196 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.581826    2559 flags.go:64] FLAG: --cgroup-root=""
Sep 10 15:27:04.586196 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.581844    2559 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 10 15:27:04.586196 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.581848    2559 flags.go:64] FLAG: --client-ca-file=""
Sep 10 15:27:04.586196 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.581852    2559 flags.go:64] FLAG: --cloud-provider="external"
Sep 10 15:27:04.586196 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.581855    2559 flags.go:64] FLAG: --cluster-dns="[]"
Sep 10 15:27:04.586196 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.581861    2559 flags.go:64] FLAG: --cluster-domain=""
Sep 10 15:27:04.586196 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.581864    2559 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 10 15:27:04.586196 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.581868    2559 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 10 15:27:04.586196 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.581872    2559 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 10 15:27:04.586196 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.581875    2559 flags.go:64] FLAG: --container-log-max-files="5"
Sep 10 15:27:04.586196 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.581879    2559 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 10 15:27:04.586794 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.581882    2559 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 10 15:27:04.586794 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.581885    2559 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 10 15:27:04.586794 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.581889    2559 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 10 15:27:04.586794 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.581892    2559 flags.go:64] FLAG: --contention-profiling="false"
Sep 10 15:27:04.586794 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.581894    2559 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 10 15:27:04.586794 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.581897    2559 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 10 15:27:04.586794 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.581900    2559 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 10 15:27:04.586794 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.581903    2559 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 10 15:27:04.586794 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.581907    2559 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 10 15:27:04.586794 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.581910    2559 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 10 15:27:04.586794 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.581913    2559 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 10 15:27:04.586794 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.581916    2559 flags.go:64] FLAG: --enable-load-reader="false"
Sep 10 15:27:04.586794 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.581919    2559 flags.go:64] FLAG: --enable-server="true"
Sep 10 15:27:04.586794 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.581922    2559 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 10 15:27:04.586794 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.581927    2559 flags.go:64] FLAG: --event-burst="100"
Sep 10 15:27:04.586794 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.581930    2559 flags.go:64] FLAG: --event-qps="50"
Sep 10 15:27:04.586794 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.581933    2559 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 10 15:27:04.586794 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.581936    2559 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 10 15:27:04.586794 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.581939    2559 flags.go:64] FLAG: --eviction-hard=""
Sep 10 15:27:04.586794 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.581943    2559 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 10 15:27:04.586794 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.581946    2559 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 10 15:27:04.586794 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.581949    2559 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 10 15:27:04.586794 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.581952    2559 flags.go:64] FLAG: --eviction-soft=""
Sep 10 15:27:04.586794 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.581955    2559 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 10 15:27:04.586794 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.581958    2559 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 10 15:27:04.587417 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.581961    2559 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 10 15:27:04.587417 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.581965    2559 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 10 15:27:04.587417 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.581968    2559 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 10 15:27:04.587417 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.581971    2559 flags.go:64] FLAG: --fail-swap-on="true"
Sep 10 15:27:04.587417 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.581974    2559 flags.go:64] FLAG: --feature-gates=""
Sep 10 15:27:04.587417 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.581978    2559 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 10 15:27:04.587417 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.581982    2559 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 10 15:27:04.587417 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.581985    2559 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 10 15:27:04.587417 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.581988    2559 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 10 15:27:04.587417 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.581991    2559 flags.go:64] FLAG: --healthz-port="10248"
Sep 10 15:27:04.587417 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.581994    2559 flags.go:64] FLAG: --help="false"
Sep 10 15:27:04.587417 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.581997    2559 flags.go:64] FLAG: --hostname-override="ip-10-0-130-30.ec2.internal"
Sep 10 15:27:04.587417 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.582001    2559 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 10 15:27:04.587417 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.582004    2559 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 10 15:27:04.587417 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.582007    2559 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 10 15:27:04.587417 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.582011    2559 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 10 15:27:04.587417 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.582014    2559 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 10 15:27:04.587417 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.582017    2559 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 10 15:27:04.587417 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.582020    2559 flags.go:64] FLAG: --image-service-endpoint=""
Sep 10 15:27:04.587417 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.582023    2559 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 10 15:27:04.587417 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.582026    2559 flags.go:64] FLAG: --kube-api-burst="100"
Sep 10 15:27:04.587417 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.582034    2559 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 10 15:27:04.587417 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.582037    2559 flags.go:64] FLAG: --kube-api-qps="50"
Sep 10 15:27:04.587417 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.582040    2559 flags.go:64] FLAG: --kube-reserved=""
Sep 10 15:27:04.588023 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.582043    2559 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 10 15:27:04.588023 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.582045    2559 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 10 15:27:04.588023 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.582048    2559 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 10 15:27:04.588023 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.582051    2559 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 10 15:27:04.588023 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.582054    2559 flags.go:64] FLAG: --lock-file=""
Sep 10 15:27:04.588023 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.582057    2559 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 10 15:27:04.588023 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.582060    2559 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 10 15:27:04.588023 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.582063    2559 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 10 15:27:04.588023 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.582069    2559 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 10 15:27:04.588023 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.582072    2559 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 10 15:27:04.588023 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.582075    2559 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 10 15:27:04.588023 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.582078    2559 flags.go:64] FLAG: --logging-format="text"
Sep 10 15:27:04.588023 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.582081    2559 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 10 15:27:04.588023 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.582084    2559 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 10 15:27:04.588023 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.582088    2559 flags.go:64] FLAG: --manifest-url=""
Sep 10 15:27:04.588023 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.582091    2559 flags.go:64] FLAG: --manifest-url-header=""
Sep 10 15:27:04.588023 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.582100    2559 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 10 15:27:04.588023 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.582103    2559 flags.go:64] FLAG: --max-open-files="1000000"
Sep 10 15:27:04.588023 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.582107    2559 flags.go:64] FLAG: --max-pods="110"
Sep 10 15:27:04.588023 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.582110    2559 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 10 15:27:04.588023 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.582114    2559 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 10 15:27:04.588023 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.582116    2559 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 10 15:27:04.588023 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.582120    2559 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 10 15:27:04.588023 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.582123    2559 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 10 15:27:04.588023 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.582126    2559 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 10 15:27:04.588643 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.582128    2559 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 10 15:27:04.588643 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.582136    2559 flags.go:64] FLAG: --node-status-max-images="50"
Sep 10 15:27:04.588643 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.582139    2559 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 10 15:27:04.588643 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.582142    2559 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 10 15:27:04.588643 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.582146    2559 flags.go:64] FLAG: --pod-cidr=""
Sep 10 15:27:04.588643 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.582149    2559 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 10 15:27:04.588643 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.582236    2559 flags.go:64] FLAG: --pod-manifest-path=""
Sep 10 15:27:04.588643 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.582243    2559 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 10 15:27:04.588643 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.582248    2559 flags.go:64] FLAG: --pods-per-core="0"
Sep 10 15:27:04.588643 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.582252    2559 flags.go:64] FLAG: --port="10250"
Sep 10 15:27:04.588643 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.582256    2559 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 10 15:27:04.588643 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.582259    2559 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-03e7e805f9b100fc1"
Sep 10 15:27:04.588643 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.582262    2559 flags.go:64] FLAG: --qos-reserved=""
Sep 10 15:27:04.588643 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.582266    2559 flags.go:64] FLAG: --read-only-port="10255"
Sep 10 15:27:04.588643 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.582270    2559 flags.go:64] FLAG: --register-node="true"
Sep 10 15:27:04.588643 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.582277    2559 flags.go:64] FLAG: --register-with-taints=""
Sep 10 15:27:04.588643 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.582281    2559 flags.go:64] FLAG: --registry-burst="10"
Sep 10 15:27:04.588643 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.582285    2559 flags.go:64] FLAG: --registry-qps="5"
Sep 10 15:27:04.588643 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.582290    2559 flags.go:64] FLAG: --reserved-cpus=""
Sep 10 15:27:04.588643 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.582295    2559 flags.go:64] FLAG: --reserved-memory=""
Sep 10 15:27:04.588643 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.582302    2559 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 10 15:27:04.588643 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.582307    2559 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 10 15:27:04.588643 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.582391    2559 flags.go:64] FLAG: --rotate-certificates="false"
Sep 10 15:27:04.588643 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.582410    2559 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 10 15:27:04.588643 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.582417    2559 flags.go:64] FLAG: --runonce="false"
Sep 10 15:27:04.589283 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.582423    2559 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 10 15:27:04.589283 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.582429    2559 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 10 15:27:04.589283 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.582435    2559 flags.go:64] FLAG: --seccomp-default="false"
Sep 10 15:27:04.589283 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.582440    2559 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 10 15:27:04.589283 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.582445    2559 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 10 15:27:04.589283 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.582457    2559 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 10 15:27:04.589283 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.582463    2559 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 10 15:27:04.589283 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.582468    2559 flags.go:64] FLAG: --storage-driver-password="root"
Sep 10 15:27:04.589283 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.582473    2559 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 10 15:27:04.589283 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.582478    2559 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 10 15:27:04.589283 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.582483    2559 flags.go:64] FLAG: --storage-driver-user="root"
Sep 10 15:27:04.589283 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.582487    2559 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 10 15:27:04.589283 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.582493    2559 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 10 15:27:04.589283 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.582498    2559 flags.go:64] FLAG: --system-cgroups=""
Sep 10 15:27:04.589283 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.582513    2559 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 10 15:27:04.589283 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.582539    2559 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 10 15:27:04.589283 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.582544    2559 flags.go:64] FLAG: --tls-cert-file=""
Sep 10 15:27:04.589283 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.582549    2559 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 10 15:27:04.589283 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.582557    2559 flags.go:64] FLAG: --tls-min-version=""
Sep 10 15:27:04.589283 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.582563    2559 flags.go:64] FLAG: --tls-private-key-file=""
Sep 10 15:27:04.589283 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.582567    2559 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 10 15:27:04.589283 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.582595    2559 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 10 15:27:04.589283 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.582600    2559 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 10 15:27:04.589283 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.582605    2559 flags.go:64] FLAG: --v="2"
Sep 10 15:27:04.589283 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.582627    2559 flags.go:64] FLAG: --version="false"
Sep 10 15:27:04.589971 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.582634    2559 flags.go:64] FLAG: --vmodule=""
Sep 10 15:27:04.589971 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.582640    2559 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 10 15:27:04.589971 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.582646    2559 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 10 15:27:04.589971 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.583054    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 15:27:04.589971 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.583064    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 15:27:04.589971 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.583069    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 15:27:04.589971 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.583075    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 15:27:04.589971 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.583085    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 15:27:04.589971 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.583089    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 15:27:04.589971 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.583095    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 15:27:04.589971 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.583099    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 15:27:04.589971 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.583104    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 15:27:04.589971 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.583110    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 15:27:04.589971 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.583114    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 15:27:04.589971 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.583119    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 15:27:04.589971 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.583124    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 15:27:04.589971 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.583129    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 15:27:04.589971 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.583133    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 15:27:04.589971 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.583138    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 15:27:04.589971 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.583147    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 15:27:04.590505 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.583227    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 15:27:04.590505 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.583254    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 15:27:04.590505 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.583263    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 15:27:04.590505 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.583272    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 15:27:04.590505 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.583278    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 15:27:04.590505 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.583284    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 15:27:04.590505 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.583289    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 15:27:04.590505 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.583295    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 15:27:04.590505 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.583301    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 15:27:04.590505 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.583307    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 15:27:04.590505 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.583312    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 15:27:04.590505 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.583324    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 15:27:04.590505 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.583329    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 15:27:04.590505 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.583334    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 15:27:04.590505 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.583339    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 15:27:04.590505 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.583343    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 15:27:04.590505 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.583348    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 15:27:04.590505 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.583353    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 15:27:04.590505 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.583358    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 15:27:04.590988 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.583363    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 15:27:04.590988 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.583368    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 15:27:04.590988 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.583372    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 15:27:04.590988 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.583377    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 15:27:04.590988 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.583420    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 15:27:04.590988 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.583433    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 15:27:04.590988 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.583470    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 15:27:04.590988 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.583640    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 15:27:04.590988 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.583645    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 15:27:04.590988 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.583648    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 15:27:04.590988 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.583651    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 15:27:04.590988 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.583654    2559 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 15:27:04.590988 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.583658    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 15:27:04.590988 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.583661    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 15:27:04.590988 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.583664    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 15:27:04.590988 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.583667    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 15:27:04.590988 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.583670    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 15:27:04.590988 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.583672    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 15:27:04.590988 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.583675    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 15:27:04.590988 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.583681    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 15:27:04.591469 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.583683    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 15:27:04.591469 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.583686    2559 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 15:27:04.591469 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.583689    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 15:27:04.591469 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.583693    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 15:27:04.591469 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.583696    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 15:27:04.591469 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.583699    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 15:27:04.591469 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.583701    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 15:27:04.591469 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.583704    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 15:27:04.591469 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.583709    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 15:27:04.591469 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.583712    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 15:27:04.591469 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.583715    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 15:27:04.591469 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.583718    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 15:27:04.591469 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.583720    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 15:27:04.591469 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.583723    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 15:27:04.591469 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.583726    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 15:27:04.591469 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.583729    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 15:27:04.591469 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.583732    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 15:27:04.591469 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.583735    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 15:27:04.591469 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.583738    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 15:27:04.591952 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.583741    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 15:27:04.591952 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.583744    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 15:27:04.591952 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.583746    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 15:27:04.591952 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.583749    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 15:27:04.591952 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.583751    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 15:27:04.591952 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.583754    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 15:27:04.591952 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.583756    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 15:27:04.591952 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.583759    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 15:27:04.591952 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.583762    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 15:27:04.591952 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.583765    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 15:27:04.591952 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.583767    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 15:27:04.591952 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.583770    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 15:27:04.591952 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.583774    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 15:27:04.591952 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.583776    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 15:27:04.591952 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.583779    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 15:27:04.591952 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.583781    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 15:27:04.591952 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.583784    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 15:27:04.591952 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.583787    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 15:27:04.591952 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.583790    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 15:27:04.591952 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.583792    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 15:27:04.592466 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.583795    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 15:27:04.592466 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.583799    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 15:27:04.592466 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.583803    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 15:27:04.592466 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.583807    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 15:27:04.592466 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.584345    2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 15:27:04.592466 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.591593    2559 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 10 15:27:04.592466 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.591611    2559 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 10 15:27:04.592466 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.592458    2559 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 10 15:27:04.592466 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.592465    2559 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 10 15:27:04.592712 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.592524    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 15:27:04.592712 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.592530    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 15:27:04.592712 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.592533    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 15:27:04.592712 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.592536    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 15:27:04.592712 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.592539    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 15:27:04.592712 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.592542    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 15:27:04.592712 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.592545    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 15:27:04.592712 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.592548    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 15:27:04.592712 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.592551    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 15:27:04.592712 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.592554    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 15:27:04.592712 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.592557    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 15:27:04.592712 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.592560    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 15:27:04.592712 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.592563    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 15:27:04.592712 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.592565    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 15:27:04.592712 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.592568    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 15:27:04.592712 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.592571    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 15:27:04.592712 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.592585    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 15:27:04.592712 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.592588    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 15:27:04.592712 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.592590    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 15:27:04.593169 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.592593    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 15:27:04.593169 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.592596    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 15:27:04.593169 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.592599    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 15:27:04.593169 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.592602    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 15:27:04.593169 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.592605    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 15:27:04.593169 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.592607    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 15:27:04.593169 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.592610    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 15:27:04.593169 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.592613    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 15:27:04.593169 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.592615    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 15:27:04.593169 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.592618    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 15:27:04.593169 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.592621    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 15:27:04.593169 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.592623    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 15:27:04.593169 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.592626    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 15:27:04.593169 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.592629    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 15:27:04.593169 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.592632    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 15:27:04.593169 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.592635    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 15:27:04.593169 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.592638    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 15:27:04.593169 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.592641    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 15:27:04.593169 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.592644    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 15:27:04.593632 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.592647    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 15:27:04.593632 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.592649    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 15:27:04.593632 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.592652    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 15:27:04.593632 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.592655    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 15:27:04.593632 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.592658    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 15:27:04.593632 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.592660    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 15:27:04.593632 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.592663    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 15:27:04.593632 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.592665    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 15:27:04.593632 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.592668    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 15:27:04.593632 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.592671    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 15:27:04.593632 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.592673    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 15:27:04.593632 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.592676    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 15:27:04.593632 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.592678    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 15:27:04.593632 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.592681    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 15:27:04.593632 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.592683    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 15:27:04.593632 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.592686    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 15:27:04.593632 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.592688    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 15:27:04.593632 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.592691    2559 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 15:27:04.593632 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.592695    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 15:27:04.593632 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.592699    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 15:27:04.594122 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.592702    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 15:27:04.594122 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.592705    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 15:27:04.594122 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.592707    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 15:27:04.594122 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.592710    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 15:27:04.594122 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.592712    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 15:27:04.594122 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.592717    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 15:27:04.594122 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.592720    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 15:27:04.594122 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.592724    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 15:27:04.594122 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.592727    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 15:27:04.594122 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.592730    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 15:27:04.594122 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.592733    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 15:27:04.594122 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.592736    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 15:27:04.594122 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.592739    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 15:27:04.594122 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.592741    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 15:27:04.594122 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.592744    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 15:27:04.594122 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.592747    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 15:27:04.594122 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.592750    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 15:27:04.594122 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.592753    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 15:27:04.594122 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.592756    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 15:27:04.594569 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.592758    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 15:27:04.594569 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.592761    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 15:27:04.594569 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.592763    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 15:27:04.594569 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.592766    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 15:27:04.594569 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.592768    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 15:27:04.594569 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.592771    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 15:27:04.594569 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.592774    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 15:27:04.594569 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.592777    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 15:27:04.594569 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.592779    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 15:27:04.594569 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.592782    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 15:27:04.594569 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.592784    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 15:27:04.594569 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.592787    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 15:27:04.594569 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.592789    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 15:27:04.594569 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.592792    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 15:27:04.594569 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.592796    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 15:27:04.594569 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.592799    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 15:27:04.594569 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.592802    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 15:27:04.594569 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.592805    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 15:27:04.594569 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.592808    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 15:27:04.595114 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.592810    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 15:27:04.595114 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.592814    2559 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 15:27:04.595114 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.592817    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 15:27:04.595114 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.592822    2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 15:27:04.595114 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.592952    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 15:27:04.595114 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.592957    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 15:27:04.595114 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.592960    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 15:27:04.595114 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.592964    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 15:27:04.595114 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.592966    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 15:27:04.595114 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.592969    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 15:27:04.595114 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.592972    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 15:27:04.595114 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.592976    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 15:27:04.595114 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.592979    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 15:27:04.595114 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.592981    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 15:27:04.595114 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.592984    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 15:27:04.595114 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.592986    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 15:27:04.595513 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.592989    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 15:27:04.595513 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.592992    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 15:27:04.595513 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.592994    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 15:27:04.595513 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.592999    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 15:27:04.595513 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.593003    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 15:27:04.595513 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.593005    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 15:27:04.595513 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.593008    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 15:27:04.595513 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.593011    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 15:27:04.595513 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.593014    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 15:27:04.595513 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.593018    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 15:27:04.595513 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.593020    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 15:27:04.595513 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.593023    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 15:27:04.595513 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.593026    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 15:27:04.595513 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.593029    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 15:27:04.595513 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.593032    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 15:27:04.595513 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.593035    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 15:27:04.595513 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.593037    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 15:27:04.595513 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.593041    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 15:27:04.595513 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.593044    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 15:27:04.595979 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.593047    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 15:27:04.595979 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.593049    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 15:27:04.595979 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.593052    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 15:27:04.595979 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.593055    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 15:27:04.595979 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.593057    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 15:27:04.595979 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.593060    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 15:27:04.595979 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.593062    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 15:27:04.595979 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.593065    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 15:27:04.595979 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.593067    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 15:27:04.595979 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.593070    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 15:27:04.595979 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.593073    2559 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 15:27:04.595979 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.593076    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 15:27:04.595979 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.593078    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 15:27:04.595979 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.593081    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 15:27:04.595979 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.593084    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 15:27:04.595979 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.593086    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 15:27:04.595979 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.593089    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 15:27:04.595979 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.593091    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 15:27:04.595979 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.593094    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 15:27:04.595979 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.593096    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 15:27:04.596469 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.593099    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 15:27:04.596469 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.593101    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 15:27:04.596469 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.593104    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 15:27:04.596469 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.593106    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 15:27:04.596469 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.593109    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 15:27:04.596469 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.593111    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 15:27:04.596469 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.593114    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 15:27:04.596469 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.593116    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 15:27:04.596469 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.593120    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 15:27:04.596469 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.593124    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 15:27:04.596469 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.593127    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 15:27:04.596469 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.593130    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 15:27:04.596469 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.593133    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 15:27:04.596469 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.593136    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 15:27:04.596469 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.593139    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 15:27:04.596469 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.593142    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 15:27:04.596469 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.593145    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 15:27:04.596469 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.593148    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 15:27:04.596469 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.593151    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 15:27:04.596928 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.593154    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 15:27:04.596928 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.593157    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 15:27:04.596928 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.593160    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 15:27:04.596928 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.593164    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 15:27:04.596928 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.593167    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 15:27:04.596928 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.593170    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 15:27:04.596928 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.593173    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 15:27:04.596928 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.593176    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 15:27:04.596928 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.593179    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 15:27:04.596928 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.593181    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 15:27:04.596928 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.593184    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 15:27:04.596928 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.593187    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 15:27:04.596928 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.593190    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 15:27:04.596928 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.593193    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 15:27:04.596928 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.593195    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 15:27:04.596928 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.593198    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 15:27:04.596928 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.593200    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 15:27:04.596928 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.593202    2559 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 15:27:04.596928 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.593205    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 15:27:04.596928 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.593207    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 15:27:04.597405 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.593210    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 15:27:04.597405 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.593213    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 15:27:04.597405 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.593215    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 15:27:04.597405 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.593218    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 15:27:04.597405 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.593221    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 15:27:04.597405 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.593224    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 15:27:04.597405 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.593228    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 15:27:04.597405 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.593230    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 15:27:04.597405 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:04.593233    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 15:27:04.597405 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.593237    2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 15:27:04.597405 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.593356    2559 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 10 15:27:04.603920 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.603904    2559 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 10 15:27:04.605043 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.605018    2559 server.go:1019] "Starting client certificate rotation"
Sep 10 15:27:04.605148 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.605125    2559 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 10 15:27:04.605186 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.605164    2559 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 10 15:27:04.608000 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.607986    2559 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 10 15:27:04.609136 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.609108    2559 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 10 15:27:04.619970 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.619817    2559 log.go:25] "Validated CRI v1 runtime API"
Sep 10 15:27:04.624654 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.624639    2559 log.go:25] "Validated CRI v1 image API"
Sep 10 15:27:04.627355 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.627336    2559 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 10 15:27:04.631840 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.631818    2559 fs.go:135] Filesystem UUIDs: map[1a3c58be-b22d-4bb7-b316-8f0a6f637cb0:/dev/nvme0n1p3 3758dbae-9acd-46a6-bca0-0ba2bebd9483:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Sep 10 15:27:04.631919 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.631839    2559 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 10 15:27:04.637248 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.637228    2559 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 10 15:27:04.637529 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.637420    2559 manager.go:216] Machine: {Timestamp:2026-09-10 15:27:04.63591049 +0000 UTC m=+0.377875584 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3095553 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2fbfe0150b43050f304f36a0822231 SystemUUID:ec2fbfe0-150b-4305-0f30-4f36a0822231 BootID:93dbebef-9773-47c2-9d69-eab177689ae3 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:42:13:bd:ce:55 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:42:13:bd:ce:55 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:4a:71:e5:a6:6e:68 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 10 15:27:04.637529 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.637528    2559 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 10 15:27:04.637681 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.637668    2559 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 10 15:27:04.640759 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.640734    2559 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 10 15:27:04.640907 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.640764    2559 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-30.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 10 15:27:04.641586 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.641564    2559 topology_manager.go:138] "Creating topology manager with none policy"
Sep 10 15:27:04.641624 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.641590    2559 container_manager_linux.go:309] "Creating device plugin manager"
Sep 10 15:27:04.641624 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.641598    2559 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 10 15:27:04.641624 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.641612    2559 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 10 15:27:04.642924 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.642912    2559 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 10 15:27:04.647405 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.647393    2559 state_mem.go:36] "Initialized new in-memory state store"
Sep 10 15:27:04.647712 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.647702    2559 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 10 15:27:04.647745 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.647740    2559 kubelet.go:486] "Attempting to sync node with API server"
Sep 10 15:27:04.647772 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.647755    2559 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 10 15:27:04.647804 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.647780    2559 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 10 15:27:04.647804 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.647794    2559 kubelet.go:398] "Adding apiserver pod source"
Sep 10 15:27:04.647857 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.647810    2559 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 10 15:27:04.648259 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.648225    2559 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 10 15:27:04.648259 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.648250    2559 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 10 15:27:04.650871 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.650855    2559 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 10 15:27:04.653048 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.653035    2559 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 10 15:27:04.653100 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.653066    2559 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 10 15:27:04.653259 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.653249    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 10 15:27:04.653292 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.653269    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 10 15:27:04.653292 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.653282    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 10 15:27:04.653292 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.653292    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 10 15:27:04.653364 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.653302    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 10 15:27:04.653364 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.653309    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 10 15:27:04.653879 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.653869    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 10 15:27:04.653915 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.653879    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 10 15:27:04.653915 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.653895    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 10 15:27:04.653915 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.653902    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 10 15:27:04.653915 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.653910    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 10 15:27:04.654016 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.653917    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 10 15:27:04.656108 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.656099    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 10 15:27:04.656148 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.656110    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 10 15:27:04.660336 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.660322    2559 server.go:1268] "Started kubelet"
Sep 10 15:27:04.660557 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:04.660521    2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-30.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 10 15:27:04.660557 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.660478    2559 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 10 15:27:04.660701 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.660563    2559 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 10 15:27:04.660701 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:04.660562    2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 10 15:27:04.660701 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.660564    2559 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-30.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 15:27:04.660701 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.660603    2559 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 10 15:27:04.661421 ip-10-0-130-30 systemd[1]: Started Kubernetes Kubelet.
Sep 10 15:27:04.661803 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.661788    2559 server.go:310] "Adding debug handlers to kubelet server"
Sep 10 15:27:04.662508 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.662427    2559 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-rlk56"
Sep 10 15:27:04.664892 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.664876    2559 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 10 15:27:04.665667 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:04.665640    2559 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 10 15:27:04.668658 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.668641    2559 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 10 15:27:04.668736 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.668653    2559 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 10 15:27:04.671639 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:04.668082    2559 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-30.ec2.internal.18d3ff2817dc27f0  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-30.ec2.internal,UID:ip-10-0-130-30.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-30.ec2.internal,},FirstTimestamp:2026-09-10 15:27:04.660297712 +0000 UTC m=+0.402262806,LastTimestamp:2026-09-10 15:27:04.660297712 +0000 UTC m=+0.402262806,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-30.ec2.internal,}"
Sep 10 15:27:04.672054 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:04.672031    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-30.ec2.internal\" not found"
Sep 10 15:27:04.672535 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.672515    2559 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 10 15:27:04.672685 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.672550    2559 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 10 15:27:04.672685 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.672656    2559 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 10 15:27:04.672786 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.672732    2559 reconstruct.go:97] "Volume reconstruction finished"
Sep 10 15:27:04.672786 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.672739    2559 reconciler.go:29] "Reconciler: start to sync state"
Sep 10 15:27:04.673395 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.673162    2559 factory.go:55] Registering systemd factory
Sep 10 15:27:04.673395 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.673164    2559 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-rlk56"
Sep 10 15:27:04.673395 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.673178    2559 factory.go:223] Registration of the systemd container factory successfully
Sep 10 15:27:04.673821 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.673601    2559 factory.go:153] Registering CRI-O factory
Sep 10 15:27:04.673821 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.673619    2559 factory.go:223] Registration of the crio container factory successfully
Sep 10 15:27:04.673821 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.673674    2559 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 10 15:27:04.673821 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.673693    2559 factory.go:103] Registering Raw factory
Sep 10 15:27:04.673821 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.673707    2559 manager.go:1196] Started watching for new ooms in manager
Sep 10 15:27:04.674175 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.674158    2559 manager.go:349] Starting recovery of all containers
Sep 10 15:27:04.678372 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:04.678344    2559 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-30.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 10 15:27:04.678458 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:04.678393    2559 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 10 15:27:04.686871 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.686852    2559 manager.go:354] Recovery completed
Sep 10 15:27:04.690824 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.690810    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 15:27:04.692643 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.692626    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-30.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 15:27:04.692718 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.692656    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-30.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 15:27:04.692718 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.692673    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-30.ec2.internal" event="NodeHasSufficientPID"
Sep 10 15:27:04.693228 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.693216    2559 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 10 15:27:04.693228 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.693227    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 15:27:04.693329 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.693228    2559 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 10 15:27:04.693329 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.693282    2559 state_mem.go:36] "Initialized new in-memory state store"
Sep 10 15:27:04.693936 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.693921    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-30.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 15:27:04.694006 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.693947    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-30.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 15:27:04.694006 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.693960    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-30.ec2.internal" event="NodeHasSufficientPID"
Sep 10 15:27:04.695457 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.695444    2559 policy_none.go:49] "None policy: Start"
Sep 10 15:27:04.695519 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.695463    2559 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 10 15:27:04.695519 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.695477    2559 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 10 15:27:04.697472 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.697450    2559 policy_none.go:47] "Start"
Sep 10 15:27:04.736183 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.736165    2559 manager.go:337] "Starting Device Plugin manager"
Sep 10 15:27:04.755737 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:04.736208    2559 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 10 15:27:04.755737 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.736223    2559 server.go:87] "Starting device plugin registration server"
Sep 10 15:27:04.755737 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.736502    2559 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 10 15:27:04.755737 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.736514    2559 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 10 15:27:04.755737 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.736621    2559 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 10 15:27:04.755737 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.736706    2559 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 10 15:27:04.755737 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.736714    2559 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 10 15:27:04.755737 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:04.737306    2559 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 10 15:27:04.755737 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:04.737351    2559 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-30.ec2.internal\" not found"
Sep 10 15:27:04.776813 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.776781    2559 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 10 15:27:04.778062 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.778043    2559 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 10 15:27:04.778146 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.778106    2559 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 10 15:27:04.778146 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.778136    2559 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 10 15:27:04.778251 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:04.778177    2559 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 10 15:27:04.780746 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.780728    2559 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 15:27:04.837013 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.836915    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 15:27:04.837922 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.837901    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-30.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 15:27:04.838001 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.837937    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-30.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 15:27:04.838001 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.837948    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-30.ec2.internal" event="NodeHasSufficientPID"
Sep 10 15:27:04.838001 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.837990    2559 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-130-30.ec2.internal"
Sep 10 15:27:04.847246 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.847229    2559 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-130-30.ec2.internal"
Sep 10 15:27:04.847313 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:04.847256    2559 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-30.ec2.internal\": node \"ip-10-0-130-30.ec2.internal\" not found"
Sep 10 15:27:04.864523 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:04.864495    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-30.ec2.internal\" not found"
Sep 10 15:27:04.878852 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.878813    2559 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-30.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-30.ec2.internal"]
Sep 10 15:27:04.878936 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.878925    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 15:27:04.880385 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.880371    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-30.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 15:27:04.880457 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.880399    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-30.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 15:27:04.880457 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.880417    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-30.ec2.internal" event="NodeHasSufficientPID"
Sep 10 15:27:04.882449 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.882436    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 15:27:04.882604 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.882573    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-30.ec2.internal"
Sep 10 15:27:04.882647 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.882624    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 15:27:04.883203 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.883188    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-30.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 15:27:04.883282 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.883228    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-30.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 15:27:04.883282 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.883238    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-30.ec2.internal" event="NodeHasSufficientPID"
Sep 10 15:27:04.883282 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.883192    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-30.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 15:27:04.883282 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.883274    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-30.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 15:27:04.883406 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.883286    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-30.ec2.internal" event="NodeHasSufficientPID"
Sep 10 15:27:04.885274 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.885256    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-30.ec2.internal"
Sep 10 15:27:04.885381 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.885314    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 15:27:04.886039 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.886023    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-30.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 15:27:04.886130 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.886049    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-30.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 15:27:04.886130 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.886059    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-30.ec2.internal" event="NodeHasSufficientPID"
Sep 10 15:27:04.910296 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:04.910277    2559 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-30.ec2.internal\" not found" node="ip-10-0-130-30.ec2.internal"
Sep 10 15:27:04.914765 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:04.914739    2559 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-30.ec2.internal\" not found" node="ip-10-0-130-30.ec2.internal"
Sep 10 15:27:04.965029 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:04.964959    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-30.ec2.internal\" not found"
Sep 10 15:27:04.974429 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.974398    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bf09fa72a5150c7d471ca089e48ca49a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-30.ec2.internal\" (UID: \"bf09fa72a5150c7d471ca089e48ca49a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-30.ec2.internal"
Sep 10 15:27:04.974429 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.974429    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bf09fa72a5150c7d471ca089e48ca49a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-30.ec2.internal\" (UID: \"bf09fa72a5150c7d471ca089e48ca49a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-30.ec2.internal"
Sep 10 15:27:04.974629 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:04.974449    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5aa341fc62b3c7ff684a265510f7e3fe-config\") pod \"kube-apiserver-proxy-ip-10-0-130-30.ec2.internal\" (UID: \"5aa341fc62b3c7ff684a265510f7e3fe\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-30.ec2.internal"
Sep 10 15:27:05.066569 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:05.066529    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-30.ec2.internal\" not found"
Sep 10 15:27:05.075630 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:05.075602    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bf09fa72a5150c7d471ca089e48ca49a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-30.ec2.internal\" (UID: \"bf09fa72a5150c7d471ca089e48ca49a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-30.ec2.internal"
Sep 10 15:27:05.075735 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:05.075638    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bf09fa72a5150c7d471ca089e48ca49a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-30.ec2.internal\" (UID: \"bf09fa72a5150c7d471ca089e48ca49a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-30.ec2.internal"
Sep 10 15:27:05.075735 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:05.075657    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5aa341fc62b3c7ff684a265510f7e3fe-config\") pod \"kube-apiserver-proxy-ip-10-0-130-30.ec2.internal\" (UID: \"5aa341fc62b3c7ff684a265510f7e3fe\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-30.ec2.internal"
Sep 10 15:27:05.075735 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:05.075703    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5aa341fc62b3c7ff684a265510f7e3fe-config\") pod \"kube-apiserver-proxy-ip-10-0-130-30.ec2.internal\" (UID: \"5aa341fc62b3c7ff684a265510f7e3fe\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-30.ec2.internal"
Sep 10 15:27:05.075735 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:05.075721    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bf09fa72a5150c7d471ca089e48ca49a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-30.ec2.internal\" (UID: \"bf09fa72a5150c7d471ca089e48ca49a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-30.ec2.internal"
Sep 10 15:27:05.075886 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:05.075720    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bf09fa72a5150c7d471ca089e48ca49a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-30.ec2.internal\" (UID: \"bf09fa72a5150c7d471ca089e48ca49a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-30.ec2.internal"
Sep 10 15:27:05.167526 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:05.167495    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-30.ec2.internal\" not found"
Sep 10 15:27:05.214497 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:05.214456    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-30.ec2.internal"
Sep 10 15:27:05.217891 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:05.217872    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-30.ec2.internal"
Sep 10 15:27:05.267705 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:05.267672    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-30.ec2.internal\" not found"
Sep 10 15:27:05.371438 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:05.371406    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-30.ec2.internal\" not found"
Sep 10 15:27:05.473337 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:05.473238    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-30.ec2.internal\" not found"
Sep 10 15:27:05.573834 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:05.573796    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-30.ec2.internal\" not found"
Sep 10 15:27:05.604996 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:05.604959    2559 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 10 15:27:05.605160 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:05.605135    2559 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 10 15:27:05.674001 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:05.673974    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-30.ec2.internal\" not found"
Sep 10 15:27:05.676253 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:05.676223    2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-09 15:22:05 +0000 UTC" deadline="2028-06-20 13:46:29.517922064 +0000 UTC"
Sep 10 15:27:05.676312 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:05.676254    2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15574h19m23.841671735s"
Sep 10 15:27:05.684230 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:05.684206    2559 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 15:27:05.775013 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:05.774941    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-30.ec2.internal\" not found"
Sep 10 15:27:05.789818 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:05.789793    2559 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 15:27:05.811606 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:05.811438    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf09fa72a5150c7d471ca089e48ca49a.slice/crio-0a462633fda320196ae0057e2ffb410a34db098e78c4802eab7f36329c327c1a WatchSource:0}: Error finding container 0a462633fda320196ae0057e2ffb410a34db098e78c4802eab7f36329c327c1a: Status 404 returned error can't find the container with id 0a462633fda320196ae0057e2ffb410a34db098e78c4802eab7f36329c327c1a
Sep 10 15:27:05.811816 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:05.811788    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5aa341fc62b3c7ff684a265510f7e3fe.slice/crio-19fe4f504878e7d35eebc96f5627f270ba7a8d9ac6009f0db72589e451d49cbe WatchSource:0}: Error finding container 19fe4f504878e7d35eebc96f5627f270ba7a8d9ac6009f0db72589e451d49cbe: Status 404 returned error can't find the container with id 19fe4f504878e7d35eebc96f5627f270ba7a8d9ac6009f0db72589e451d49cbe
Sep 10 15:27:05.816172 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:05.816150    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 15:27:05.872355 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:05.872309    2559 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-30.ec2.internal"
Sep 10 15:27:05.885757 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:05.885733    2559 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 10 15:27:05.887078 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:05.887064    2559 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-30.ec2.internal"
Sep 10 15:27:05.897168 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:05.897150    2559 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 10 15:27:06.091536 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.091442    2559 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 15:27:06.649400 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.649230    2559 apiserver.go:52] "Watching apiserver"
Sep 10 15:27:06.660183 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.659966    2559 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 10 15:27:06.661236 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.661204    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-fkmfw","openshift-ovn-kubernetes/ovnkube-node-sdjq9","kube-system/global-pull-secret-syncer-7kktn","openshift-dns/node-resolver-md7dv","openshift-image-registry/node-ca-pzn2d","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-30.ec2.internal","openshift-multus/multus-additional-cni-plugins-6nx2l","openshift-multus/multus-vdtg4","kube-system/konnectivity-agent-wgbw2","kube-system/kube-apiserver-proxy-ip-10-0-130-30.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kcrzq","openshift-cluster-node-tuning-operator/tuned-zxgvr","openshift-multus/network-metrics-daemon-w24m7","openshift-network-diagnostics/network-check-target-qbx8c"]
Sep 10 15:27:06.664452 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.664007    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6nx2l"
Sep 10 15:27:06.666605 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.666365    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w24m7"
Sep 10 15:27:06.666605 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:06.666448    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w24m7" podUID="87bd161e-974d-49ee-861d-22763253529c"
Sep 10 15:27:06.666869 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.666831    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 10 15:27:06.666964 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.666841    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-pl4xd\""
Sep 10 15:27:06.666964 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.666942    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 10 15:27:06.667292 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.667272    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 10 15:27:06.667577 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.667559    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 10 15:27:06.667817 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.667799    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 10 15:27:06.675688 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.673778    2559 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 10 15:27:06.675688 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.673827    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-pzn2d"
Sep 10 15:27:06.675688 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.674783    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-md7dv"
Sep 10 15:27:06.675688 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.674853    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7kktn"
Sep 10 15:27:06.675688 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:06.674941    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7kktn" podUID="96b96370-410d-447c-8fb0-63f56ab52310"
Sep 10 15:27:06.678223 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.676639    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-6cbtf\""
Sep 10 15:27:06.678223 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.677027    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 10 15:27:06.678223 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.677056    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 10 15:27:06.678223 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.677124    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 10 15:27:06.678223 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.677585    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 10 15:27:06.678223 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.677884    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 10 15:27:06.678223 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.678141    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-q6dsw\""
Sep 10 15:27:06.680831 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.680809    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qbx8c"
Sep 10 15:27:06.680925 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:06.680908    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qbx8c" podUID="008da33d-29cc-4f79-b53a-d1a674b604e8"
Sep 10 15:27:06.682429 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.682122    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wgbw2"
Sep 10 15:27:06.684968 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.684627    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 10 15:27:06.684968 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.684770    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 10 15:27:06.684968 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.684812    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wx7jm\" (UniqueName: \"kubernetes.io/projected/008da33d-29cc-4f79-b53a-d1a674b604e8-kube-api-access-wx7jm\") pod \"network-check-target-qbx8c\" (UID: \"008da33d-29cc-4f79-b53a-d1a674b604e8\") " pod="openshift-network-diagnostics/network-check-target-qbx8c"
Sep 10 15:27:06.684968 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.684849    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dd421353-8e9c-4660-8224-cb11e8575e7a-system-cni-dir\") pod \"multus-additional-cni-plugins-6nx2l\" (UID: \"dd421353-8e9c-4660-8224-cb11e8575e7a\") " pod="openshift-multus/multus-additional-cni-plugins-6nx2l"
Sep 10 15:27:06.684968 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.684886    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dd421353-8e9c-4660-8224-cb11e8575e7a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6nx2l\" (UID: \"dd421353-8e9c-4660-8224-cb11e8575e7a\") " pod="openshift-multus/multus-additional-cni-plugins-6nx2l"
Sep 10 15:27:06.684968 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.684918    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbw4r\" (UniqueName: \"kubernetes.io/projected/dd421353-8e9c-4660-8224-cb11e8575e7a-kube-api-access-vbw4r\") pod \"multus-additional-cni-plugins-6nx2l\" (UID: \"dd421353-8e9c-4660-8224-cb11e8575e7a\") " pod="openshift-multus/multus-additional-cni-plugins-6nx2l"
Sep 10 15:27:06.684968 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.684948    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dd421353-8e9c-4660-8224-cb11e8575e7a-cnibin\") pod \"multus-additional-cni-plugins-6nx2l\" (UID: \"dd421353-8e9c-4660-8224-cb11e8575e7a\") " pod="openshift-multus/multus-additional-cni-plugins-6nx2l"
Sep 10 15:27:06.684968 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.684956    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-fkmfw"
Sep 10 15:27:06.685424 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.684975    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87bd161e-974d-49ee-861d-22763253529c-metrics-certs\") pod \"network-metrics-daemon-w24m7\" (UID: \"87bd161e-974d-49ee-861d-22763253529c\") " pod="openshift-multus/network-metrics-daemon-w24m7"
Sep 10 15:27:06.685424 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.685002    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/96b96370-410d-447c-8fb0-63f56ab52310-kubelet-config\") pod \"global-pull-secret-syncer-7kktn\" (UID: \"96b96370-410d-447c-8fb0-63f56ab52310\") " pod="kube-system/global-pull-secret-syncer-7kktn"
Sep 10 15:27:06.685424 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.684952    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-ww5jn\""
Sep 10 15:27:06.685424 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.685070    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/96b96370-410d-447c-8fb0-63f56ab52310-original-pull-secret\") pod \"global-pull-secret-syncer-7kktn\" (UID: \"96b96370-410d-447c-8fb0-63f56ab52310\") " pod="kube-system/global-pull-secret-syncer-7kktn"
Sep 10 15:27:06.685424 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.685134    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8jbt\" (UniqueName: \"kubernetes.io/projected/6db07abf-0c03-4565-a661-9df53fbe5957-kube-api-access-b8jbt\") pod \"node-ca-pzn2d\" (UID: \"6db07abf-0c03-4565-a661-9df53fbe5957\") " pod="openshift-image-registry/node-ca-pzn2d"
Sep 10 15:27:06.685424 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.685168    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/dd421353-8e9c-4660-8224-cb11e8575e7a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6nx2l\" (UID: \"dd421353-8e9c-4660-8224-cb11e8575e7a\") " pod="openshift-multus/multus-additional-cni-plugins-6nx2l"
Sep 10 15:27:06.685424 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.685209    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/dd421353-8e9c-4660-8224-cb11e8575e7a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6nx2l\" (UID: \"dd421353-8e9c-4660-8224-cb11e8575e7a\") " pod="openshift-multus/multus-additional-cni-plugins-6nx2l"
Sep 10 15:27:06.685424 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.685236    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c56c45fc-a485-48b6-a88a-355ba442927f-hosts-file\") pod \"node-resolver-md7dv\" (UID: \"c56c45fc-a485-48b6-a88a-355ba442927f\") " pod="openshift-dns/node-resolver-md7dv"
Sep 10 15:27:06.685424 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.685265    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c56c45fc-a485-48b6-a88a-355ba442927f-tmp-dir\") pod \"node-resolver-md7dv\" (UID: \"c56c45fc-a485-48b6-a88a-355ba442927f\") " pod="openshift-dns/node-resolver-md7dv"
Sep 10 15:27:06.685424 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.685292    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chmls\" (UniqueName: \"kubernetes.io/projected/c56c45fc-a485-48b6-a88a-355ba442927f-kube-api-access-chmls\") pod \"node-resolver-md7dv\" (UID: \"c56c45fc-a485-48b6-a88a-355ba442927f\") " pod="openshift-dns/node-resolver-md7dv"
Sep 10 15:27:06.685424 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.685317    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dd421353-8e9c-4660-8224-cb11e8575e7a-cni-binary-copy\") pod \"multus-additional-cni-plugins-6nx2l\" (UID: \"dd421353-8e9c-4660-8224-cb11e8575e7a\") " pod="openshift-multus/multus-additional-cni-plugins-6nx2l"
Sep 10 15:27:06.685424 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.685347    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/96b96370-410d-447c-8fb0-63f56ab52310-dbus\") pod \"global-pull-secret-syncer-7kktn\" (UID: \"96b96370-410d-447c-8fb0-63f56ab52310\") " pod="kube-system/global-pull-secret-syncer-7kktn"
Sep 10 15:27:06.685424 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.685369    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6db07abf-0c03-4565-a661-9df53fbe5957-host\") pod \"node-ca-pzn2d\" (UID: \"6db07abf-0c03-4565-a661-9df53fbe5957\") " pod="openshift-image-registry/node-ca-pzn2d"
Sep 10 15:27:06.685424 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.685393    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6db07abf-0c03-4565-a661-9df53fbe5957-serviceca\") pod \"node-ca-pzn2d\" (UID: \"6db07abf-0c03-4565-a661-9df53fbe5957\") " pod="openshift-image-registry/node-ca-pzn2d"
Sep 10 15:27:06.685424 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.685424    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdccr\" (UniqueName: \"kubernetes.io/projected/87bd161e-974d-49ee-861d-22763253529c-kube-api-access-fdccr\") pod \"network-metrics-daemon-w24m7\" (UID: \"87bd161e-974d-49ee-861d-22763253529c\") " pod="openshift-multus/network-metrics-daemon-w24m7"
Sep 10 15:27:06.688891 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.688822    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-sdjq9"
Sep 10 15:27:06.689688 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.689324    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 10 15:27:06.689688 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.689464    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 10 15:27:06.689864 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.689789    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-v978l\""
Sep 10 15:27:06.689922 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.689889    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 10 15:27:06.692034 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.691604    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vdtg4"
Sep 10 15:27:06.692453 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.692123    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 10 15:27:06.692453 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.692288    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 10 15:27:06.692674 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.692656    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 10 15:27:06.692764 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.692694    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-vx8kw\""
Sep 10 15:27:06.693165 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.693146    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kcrzq"
Sep 10 15:27:06.693842 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.693819    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 10 15:27:06.693937 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.693861    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 10 15:27:06.694280 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.693995    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 10 15:27:06.694280 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.694151    2559 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 10 15:27:06.695081 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.695056    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-hlkng\""
Sep 10 15:27:06.695371 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.695353    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 10 15:27:06.695975 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.695957    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zxgvr"
Sep 10 15:27:06.696523 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.696505    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 10 15:27:06.696773 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.696751    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-kj8n2\""
Sep 10 15:27:06.696773 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.696765    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 10 15:27:06.697008 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.696943    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 10 15:27:06.698361 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.698339    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 10 15:27:06.698726 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.698705    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-p2njt\""
Sep 10 15:27:06.699099 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.699079    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 10 15:27:06.713304 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.713275    2559 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-v2z9j"
Sep 10 15:27:06.724009 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.723987    2559 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-v2z9j"
Sep 10 15:27:06.774526 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.774496    2559 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 10 15:27:06.784261 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.784201    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-30.ec2.internal" event={"ID":"bf09fa72a5150c7d471ca089e48ca49a","Type":"ContainerStarted","Data":"0a462633fda320196ae0057e2ffb410a34db098e78c4802eab7f36329c327c1a"}
Sep 10 15:27:06.785679 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.785647    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dd421353-8e9c-4660-8224-cb11e8575e7a-cni-binary-copy\") pod \"multus-additional-cni-plugins-6nx2l\" (UID: \"dd421353-8e9c-4660-8224-cb11e8575e7a\") " pod="openshift-multus/multus-additional-cni-plugins-6nx2l"
Sep 10 15:27:06.785679 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.785669    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-30.ec2.internal" event={"ID":"5aa341fc62b3c7ff684a265510f7e3fe","Type":"ContainerStarted","Data":"19fe4f504878e7d35eebc96f5627f270ba7a8d9ac6009f0db72589e451d49cbe"}
Sep 10 15:27:06.785871 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.785692    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0cf24a6f-168a-4076-a08e-6b30b6f34234-socket-dir\") pod \"aws-ebs-csi-driver-node-kcrzq\" (UID: \"0cf24a6f-168a-4076-a08e-6b30b6f34234\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kcrzq"
Sep 10 15:27:06.785871 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.785718    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/153abd57-b263-4386-9087-f9138c09e4a0-etc-modprobe-d\") pod \"tuned-zxgvr\" (UID: \"153abd57-b263-4386-9087-f9138c09e4a0\") " pod="openshift-cluster-node-tuning-operator/tuned-zxgvr"
Sep 10 15:27:06.785871 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.785745    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/153abd57-b263-4386-9087-f9138c09e4a0-var-lib-kubelet\") pod \"tuned-zxgvr\" (UID: \"153abd57-b263-4386-9087-f9138c09e4a0\") " pod="openshift-cluster-node-tuning-operator/tuned-zxgvr"
Sep 10 15:27:06.785871 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.785851    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/96f18013-00ea-4663-87b5-4bca49f4295a-run-openvswitch\") pod \"ovnkube-node-sdjq9\" (UID: \"96f18013-00ea-4663-87b5-4bca49f4295a\") " pod="openshift-ovn-kubernetes/ovnkube-node-sdjq9"
Sep 10 15:27:06.786076 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.785881    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/96f18013-00ea-4663-87b5-4bca49f4295a-env-overrides\") pod \"ovnkube-node-sdjq9\" (UID: \"96f18013-00ea-4663-87b5-4bca49f4295a\") " pod="openshift-ovn-kubernetes/ovnkube-node-sdjq9"
Sep 10 15:27:06.786076 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.785917    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1a08685b-d954-46be-8456-410c35f09c93-host-var-lib-kubelet\") pod \"multus-vdtg4\" (UID: \"1a08685b-d954-46be-8456-410c35f09c93\") " pod="openshift-multus/multus-vdtg4"
Sep 10 15:27:06.786076 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.785945    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0cf24a6f-168a-4076-a08e-6b30b6f34234-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kcrzq\" (UID: \"0cf24a6f-168a-4076-a08e-6b30b6f34234\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kcrzq"
Sep 10 15:27:06.786076 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.785973    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87pf9\" (UniqueName: \"kubernetes.io/projected/0cf24a6f-168a-4076-a08e-6b30b6f34234-kube-api-access-87pf9\") pod \"aws-ebs-csi-driver-node-kcrzq\" (UID: \"0cf24a6f-168a-4076-a08e-6b30b6f34234\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kcrzq"
Sep 10 15:27:06.786076 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.786006    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/153abd57-b263-4386-9087-f9138c09e4a0-etc-sysctl-conf\") pod \"tuned-zxgvr\" (UID: \"153abd57-b263-4386-9087-f9138c09e4a0\") " pod="openshift-cluster-node-tuning-operator/tuned-zxgvr"
Sep 10 15:27:06.786076 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.786031    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1a08685b-d954-46be-8456-410c35f09c93-multus-conf-dir\") pod \"multus-vdtg4\" (UID: \"1a08685b-d954-46be-8456-410c35f09c93\") " pod="openshift-multus/multus-vdtg4"
Sep 10 15:27:06.786338 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.786083    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dd421353-8e9c-4660-8224-cb11e8575e7a-cnibin\") pod \"multus-additional-cni-plugins-6nx2l\" (UID: \"dd421353-8e9c-4660-8224-cb11e8575e7a\") " pod="openshift-multus/multus-additional-cni-plugins-6nx2l"
Sep 10 15:27:06.786338 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.786151    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dd421353-8e9c-4660-8224-cb11e8575e7a-cnibin\") pod \"multus-additional-cni-plugins-6nx2l\" (UID: \"dd421353-8e9c-4660-8224-cb11e8575e7a\") " pod="openshift-multus/multus-additional-cni-plugins-6nx2l"
Sep 10 15:27:06.786338 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.786166    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7a9f7d3b-8a21-40ed-ab47-067f97ed537a-konnectivity-ca\") pod \"konnectivity-agent-wgbw2\" (UID: \"7a9f7d3b-8a21-40ed-ab47-067f97ed537a\") " pod="kube-system/konnectivity-agent-wgbw2"
Sep 10 15:27:06.786338 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.786200    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/96f18013-00ea-4663-87b5-4bca49f4295a-etc-openvswitch\") pod \"ovnkube-node-sdjq9\" (UID: \"96f18013-00ea-4663-87b5-4bca49f4295a\") " pod="openshift-ovn-kubernetes/ovnkube-node-sdjq9"
Sep 10 15:27:06.786338 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.786231    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/153abd57-b263-4386-9087-f9138c09e4a0-etc-sysconfig\") pod \"tuned-zxgvr\" (UID: \"153abd57-b263-4386-9087-f9138c09e4a0\") " pod="openshift-cluster-node-tuning-operator/tuned-zxgvr"
Sep 10 15:27:06.786338 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.786258    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/96f18013-00ea-4663-87b5-4bca49f4295a-host-run-netns\") pod \"ovnkube-node-sdjq9\" (UID: \"96f18013-00ea-4663-87b5-4bca49f4295a\") " pod="openshift-ovn-kubernetes/ovnkube-node-sdjq9"
Sep 10 15:27:06.786338 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.786299    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/96f18013-00ea-4663-87b5-4bca49f4295a-node-log\") pod \"ovnkube-node-sdjq9\" (UID: \"96f18013-00ea-4663-87b5-4bca49f4295a\") " pod="openshift-ovn-kubernetes/ovnkube-node-sdjq9"
Sep 10 15:27:06.786338 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.786322    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/96f18013-00ea-4663-87b5-4bca49f4295a-ovnkube-config\") pod \"ovnkube-node-sdjq9\" (UID: \"96f18013-00ea-4663-87b5-4bca49f4295a\") " pod="openshift-ovn-kubernetes/ovnkube-node-sdjq9"
Sep 10 15:27:06.786690 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.786347    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/96f18013-00ea-4663-87b5-4bca49f4295a-ovn-node-metrics-cert\") pod \"ovnkube-node-sdjq9\" (UID: \"96f18013-00ea-4663-87b5-4bca49f4295a\") " pod="openshift-ovn-kubernetes/ovnkube-node-sdjq9"
Sep 10 15:27:06.786690 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.786373    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1a08685b-d954-46be-8456-410c35f09c93-host-var-lib-cni-multus\") pod \"multus-vdtg4\" (UID: \"1a08685b-d954-46be-8456-410c35f09c93\") " pod="openshift-multus/multus-vdtg4"
Sep 10 15:27:06.786690 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.786404    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6db07abf-0c03-4565-a661-9df53fbe5957-serviceca\") pod \"node-ca-pzn2d\" (UID: \"6db07abf-0c03-4565-a661-9df53fbe5957\") " pod="openshift-image-registry/node-ca-pzn2d"
Sep 10 15:27:06.786690 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.786450    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nh2lw\" (UniqueName: \"kubernetes.io/projected/153abd57-b263-4386-9087-f9138c09e4a0-kube-api-access-nh2lw\") pod \"tuned-zxgvr\" (UID: \"153abd57-b263-4386-9087-f9138c09e4a0\") " pod="openshift-cluster-node-tuning-operator/tuned-zxgvr"
Sep 10 15:27:06.786690 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.786475    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2fsc\" (UniqueName: \"kubernetes.io/projected/96f18013-00ea-4663-87b5-4bca49f4295a-kube-api-access-p2fsc\") pod \"ovnkube-node-sdjq9\" (UID: \"96f18013-00ea-4663-87b5-4bca49f4295a\") " pod="openshift-ovn-kubernetes/ovnkube-node-sdjq9"
Sep 10 15:27:06.786690 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.786501    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1a08685b-d954-46be-8456-410c35f09c93-multus-daemon-config\") pod \"multus-vdtg4\" (UID: \"1a08685b-d954-46be-8456-410c35f09c93\") " pod="openshift-multus/multus-vdtg4"
Sep 10 15:27:06.786690 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.786528    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wx7jm\" (UniqueName: \"kubernetes.io/projected/008da33d-29cc-4f79-b53a-d1a674b604e8-kube-api-access-wx7jm\") pod \"network-check-target-qbx8c\" (UID: \"008da33d-29cc-4f79-b53a-d1a674b604e8\") " pod="openshift-network-diagnostics/network-check-target-qbx8c"
Sep 10 15:27:06.786690 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.786556    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dd421353-8e9c-4660-8224-cb11e8575e7a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6nx2l\" (UID: \"dd421353-8e9c-4660-8224-cb11e8575e7a\") " pod="openshift-multus/multus-additional-cni-plugins-6nx2l"
Sep 10 15:27:06.787086 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.786753    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dd421353-8e9c-4660-8224-cb11e8575e7a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6nx2l\" (UID: \"dd421353-8e9c-4660-8224-cb11e8575e7a\") " pod="openshift-multus/multus-additional-cni-plugins-6nx2l"
Sep 10 15:27:06.787086 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.786791    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0cf24a6f-168a-4076-a08e-6b30b6f34234-device-dir\") pod \"aws-ebs-csi-driver-node-kcrzq\" (UID: \"0cf24a6f-168a-4076-a08e-6b30b6f34234\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kcrzq"
Sep 10 15:27:06.787086 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.786820    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/153abd57-b263-4386-9087-f9138c09e4a0-sys\") pod \"tuned-zxgvr\" (UID: \"153abd57-b263-4386-9087-f9138c09e4a0\") " pod="openshift-cluster-node-tuning-operator/tuned-zxgvr"
Sep 10 15:27:06.787086 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.786844    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/96f18013-00ea-4663-87b5-4bca49f4295a-host-slash\") pod \"ovnkube-node-sdjq9\" (UID: \"96f18013-00ea-4663-87b5-4bca49f4295a\") " pod="openshift-ovn-kubernetes/ovnkube-node-sdjq9"
Sep 10 15:27:06.787086 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.786874    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/96f18013-00ea-4663-87b5-4bca49f4295a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-sdjq9\" (UID: \"96f18013-00ea-4663-87b5-4bca49f4295a\") " pod="openshift-ovn-kubernetes/ovnkube-node-sdjq9"
Sep 10 15:27:06.787086 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.786899    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1a08685b-d954-46be-8456-410c35f09c93-cni-binary-copy\") pod \"multus-vdtg4\" (UID: \"1a08685b-d954-46be-8456-410c35f09c93\") " pod="openshift-multus/multus-vdtg4"
Sep 10 15:27:06.787086 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.786925    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1a08685b-d954-46be-8456-410c35f09c93-multus-socket-dir-parent\") pod \"multus-vdtg4\" (UID: \"1a08685b-d954-46be-8456-410c35f09c93\") " pod="openshift-multus/multus-vdtg4"
Sep 10 15:27:06.787086 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.786965    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87bd161e-974d-49ee-861d-22763253529c-metrics-certs\") pod \"network-metrics-daemon-w24m7\" (UID: \"87bd161e-974d-49ee-861d-22763253529c\") " pod="openshift-multus/network-metrics-daemon-w24m7"
Sep 10 15:27:06.787086 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.787006    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8jbt\" (UniqueName: \"kubernetes.io/projected/6db07abf-0c03-4565-a661-9df53fbe5957-kube-api-access-b8jbt\") pod \"node-ca-pzn2d\" (UID: \"6db07abf-0c03-4565-a661-9df53fbe5957\") " pod="openshift-image-registry/node-ca-pzn2d"
Sep 10 15:27:06.787086 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.787008    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6db07abf-0c03-4565-a661-9df53fbe5957-serviceca\") pod \"node-ca-pzn2d\" (UID: \"6db07abf-0c03-4565-a661-9df53fbe5957\") " pod="openshift-image-registry/node-ca-pzn2d"
Sep 10 15:27:06.787086 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.787046    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0cf24a6f-168a-4076-a08e-6b30b6f34234-etc-selinux\") pod \"aws-ebs-csi-driver-node-kcrzq\" (UID: \"0cf24a6f-168a-4076-a08e-6b30b6f34234\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kcrzq"
Sep 10 15:27:06.787086 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:06.787078    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 15:27:06.787590 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.787109    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dd421353-8e9c-4660-8224-cb11e8575e7a-cni-binary-copy\") pod \"multus-additional-cni-plugins-6nx2l\" (UID: \"dd421353-8e9c-4660-8224-cb11e8575e7a\") " pod="openshift-multus/multus-additional-cni-plugins-6nx2l"
Sep 10 15:27:06.787590 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.787183    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1a08685b-d954-46be-8456-410c35f09c93-host-var-lib-cni-bin\") pod \"multus-vdtg4\" (UID: \"1a08685b-d954-46be-8456-410c35f09c93\") " pod="openshift-multus/multus-vdtg4"
Sep 10 15:27:06.787590 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:06.787222    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87bd161e-974d-49ee-861d-22763253529c-metrics-certs podName:87bd161e-974d-49ee-861d-22763253529c nodeName:}" failed. No retries permitted until 2026-09-10 15:27:07.287188393 +0000 UTC m=+3.029153479 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/87bd161e-974d-49ee-861d-22763253529c-metrics-certs") pod "network-metrics-daemon-w24m7" (UID: "87bd161e-974d-49ee-861d-22763253529c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 15:27:06.787590 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.787271    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcnpj\" (UniqueName: \"kubernetes.io/projected/3f10d0b5-445b-4531-a2ff-ea0264c02ef5-kube-api-access-bcnpj\") pod \"iptables-alerter-fkmfw\" (UID: \"3f10d0b5-445b-4531-a2ff-ea0264c02ef5\") " pod="openshift-network-operator/iptables-alerter-fkmfw"
Sep 10 15:27:06.787590 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.787307    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chmls\" (UniqueName: \"kubernetes.io/projected/c56c45fc-a485-48b6-a88a-355ba442927f-kube-api-access-chmls\") pod \"node-resolver-md7dv\" (UID: \"c56c45fc-a485-48b6-a88a-355ba442927f\") " pod="openshift-dns/node-resolver-md7dv"
Sep 10 15:27:06.787590 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.787334    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0cf24a6f-168a-4076-a08e-6b30b6f34234-sys-fs\") pod \"aws-ebs-csi-driver-node-kcrzq\" (UID: \"0cf24a6f-168a-4076-a08e-6b30b6f34234\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kcrzq"
Sep 10 15:27:06.787590 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.787361    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/96f18013-00ea-4663-87b5-4bca49f4295a-run-systemd\") pod \"ovnkube-node-sdjq9\" (UID: \"96f18013-00ea-4663-87b5-4bca49f4295a\") " pod="openshift-ovn-kubernetes/ovnkube-node-sdjq9"
Sep 10 15:27:06.787590 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.787411    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/96f18013-00ea-4663-87b5-4bca49f4295a-run-ovn\") pod \"ovnkube-node-sdjq9\" (UID: \"96f18013-00ea-4663-87b5-4bca49f4295a\") " pod="openshift-ovn-kubernetes/ovnkube-node-sdjq9"
Sep 10 15:27:06.787590 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.787433    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/96f18013-00ea-4663-87b5-4bca49f4295a-host-run-ovn-kubernetes\") pod \"ovnkube-node-sdjq9\" (UID: \"96f18013-00ea-4663-87b5-4bca49f4295a\") " pod="openshift-ovn-kubernetes/ovnkube-node-sdjq9"
Sep 10 15:27:06.787590 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.787456    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1a08685b-d954-46be-8456-410c35f09c93-hostroot\") pod \"multus-vdtg4\" (UID: \"1a08685b-d954-46be-8456-410c35f09c93\") " pod="openshift-multus/multus-vdtg4"
Sep 10 15:27:06.787590 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.787505    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/96f18013-00ea-4663-87b5-4bca49f4295a-log-socket\") pod \"ovnkube-node-sdjq9\" (UID: \"96f18013-00ea-4663-87b5-4bca49f4295a\") " pod="openshift-ovn-kubernetes/ovnkube-node-sdjq9"
Sep 10 15:27:06.787590 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.787525    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6db07abf-0c03-4565-a661-9df53fbe5957-host\") pod \"node-ca-pzn2d\" (UID: \"6db07abf-0c03-4565-a661-9df53fbe5957\") " pod="openshift-image-registry/node-ca-pzn2d"
Sep 10 15:27:06.787590 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.787541    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/153abd57-b263-4386-9087-f9138c09e4a0-etc-sysctl-d\") pod \"tuned-zxgvr\" (UID: \"153abd57-b263-4386-9087-f9138c09e4a0\") " pod="openshift-cluster-node-tuning-operator/tuned-zxgvr"
Sep 10 15:27:06.787590 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.787566    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/96f18013-00ea-4663-87b5-4bca49f4295a-var-lib-openvswitch\") pod \"ovnkube-node-sdjq9\" (UID: \"96f18013-00ea-4663-87b5-4bca49f4295a\") " pod="openshift-ovn-kubernetes/ovnkube-node-sdjq9"
Sep 10 15:27:06.788294 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.787626    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6db07abf-0c03-4565-a661-9df53fbe5957-host\") pod \"node-ca-pzn2d\" (UID: \"6db07abf-0c03-4565-a661-9df53fbe5957\") " pod="openshift-image-registry/node-ca-pzn2d"
Sep 10 15:27:06.788294 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.787679    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fc4rw\" (UniqueName: \"kubernetes.io/projected/1a08685b-d954-46be-8456-410c35f09c93-kube-api-access-fc4rw\") pod \"multus-vdtg4\" (UID: \"1a08685b-d954-46be-8456-410c35f09c93\") " pod="openshift-multus/multus-vdtg4"
Sep 10 15:27:06.788294 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.787713    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbw4r\" (UniqueName: \"kubernetes.io/projected/dd421353-8e9c-4660-8224-cb11e8575e7a-kube-api-access-vbw4r\") pod \"multus-additional-cni-plugins-6nx2l\" (UID: \"dd421353-8e9c-4660-8224-cb11e8575e7a\") " pod="openshift-multus/multus-additional-cni-plugins-6nx2l"
Sep 10 15:27:06.788294 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.787744    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/153abd57-b263-4386-9087-f9138c09e4a0-host\") pod \"tuned-zxgvr\" (UID: \"153abd57-b263-4386-9087-f9138c09e4a0\") " pod="openshift-cluster-node-tuning-operator/tuned-zxgvr"
Sep 10 15:27:06.788294 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.787763    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/96f18013-00ea-4663-87b5-4bca49f4295a-host-kubelet\") pod \"ovnkube-node-sdjq9\" (UID: \"96f18013-00ea-4663-87b5-4bca49f4295a\") " pod="openshift-ovn-kubernetes/ovnkube-node-sdjq9"
Sep 10 15:27:06.788294 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.787778    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/96f18013-00ea-4663-87b5-4bca49f4295a-systemd-units\") pod \"ovnkube-node-sdjq9\" (UID: \"96f18013-00ea-4663-87b5-4bca49f4295a\") " pod="openshift-ovn-kubernetes/ovnkube-node-sdjq9"
Sep 10 15:27:06.788294 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.787793    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1a08685b-d954-46be-8456-410c35f09c93-system-cni-dir\") pod \"multus-vdtg4\" (UID: \"1a08685b-d954-46be-8456-410c35f09c93\") " pod="openshift-multus/multus-vdtg4"
Sep 10 15:27:06.788294 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.787811    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1a08685b-d954-46be-8456-410c35f09c93-multus-cni-dir\") pod \"multus-vdtg4\" (UID: \"1a08685b-d954-46be-8456-410c35f09c93\") " pod="openshift-multus/multus-vdtg4"
Sep 10 15:27:06.788294 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.787854    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/96b96370-410d-447c-8fb0-63f56ab52310-kubelet-config\") pod \"global-pull-secret-syncer-7kktn\" (UID: \"96b96370-410d-447c-8fb0-63f56ab52310\") " pod="kube-system/global-pull-secret-syncer-7kktn"
Sep 10 15:27:06.788294 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.787899    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/96b96370-410d-447c-8fb0-63f56ab52310-original-pull-secret\") pod \"global-pull-secret-syncer-7kktn\" (UID: \"96b96370-410d-447c-8fb0-63f56ab52310\") " pod="kube-system/global-pull-secret-syncer-7kktn"
Sep 10 15:27:06.788294 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.787928    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/96b96370-410d-447c-8fb0-63f56ab52310-kubelet-config\") pod \"global-pull-secret-syncer-7kktn\" (UID: \"96b96370-410d-447c-8fb0-63f56ab52310\") " pod="kube-system/global-pull-secret-syncer-7kktn"
Sep 10 15:27:06.788294 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.787929    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/153abd57-b263-4386-9087-f9138c09e4a0-etc-kubernetes\") pod \"tuned-zxgvr\" (UID: \"153abd57-b263-4386-9087-f9138c09e4a0\") " pod="openshift-cluster-node-tuning-operator/tuned-zxgvr"
Sep 10 15:27:06.788294 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.787964    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1a08685b-d954-46be-8456-410c35f09c93-host-run-k8s-cni-cncf-io\") pod \"multus-vdtg4\" (UID: \"1a08685b-d954-46be-8456-410c35f09c93\") " pod="openshift-multus/multus-vdtg4"
Sep 10 15:27:06.788294 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.787989    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3f10d0b5-445b-4531-a2ff-ea0264c02ef5-iptables-alerter-script\") pod \"iptables-alerter-fkmfw\" (UID: \"3f10d0b5-445b-4531-a2ff-ea0264c02ef5\") " pod="openshift-network-operator/iptables-alerter-fkmfw"
Sep 10 15:27:06.788294 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:06.788003    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 15:27:06.788294 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.788023    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/dd421353-8e9c-4660-8224-cb11e8575e7a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6nx2l\" (UID: \"dd421353-8e9c-4660-8224-cb11e8575e7a\") " pod="openshift-multus/multus-additional-cni-plugins-6nx2l"
Sep 10 15:27:06.789048 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:06.788046    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96b96370-410d-447c-8fb0-63f56ab52310-original-pull-secret podName:96b96370-410d-447c-8fb0-63f56ab52310 nodeName:}" failed. No retries permitted until 2026-09-10 15:27:07.288033076 +0000 UTC m=+3.029998164 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/96b96370-410d-447c-8fb0-63f56ab52310-original-pull-secret") pod "global-pull-secret-syncer-7kktn" (UID: "96b96370-410d-447c-8fb0-63f56ab52310") : object "kube-system"/"original-pull-secret" not registered
Sep 10 15:27:06.789048 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.788089    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c56c45fc-a485-48b6-a88a-355ba442927f-tmp-dir\") pod \"node-resolver-md7dv\" (UID: \"c56c45fc-a485-48b6-a88a-355ba442927f\") " pod="openshift-dns/node-resolver-md7dv"
Sep 10 15:27:06.789048 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.788115    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/153abd57-b263-4386-9087-f9138c09e4a0-tmp\") pod \"tuned-zxgvr\" (UID: \"153abd57-b263-4386-9087-f9138c09e4a0\") " pod="openshift-cluster-node-tuning-operator/tuned-zxgvr"
Sep 10 15:27:06.789048 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.788139    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7a9f7d3b-8a21-40ed-ab47-067f97ed537a-agent-certs\") pod \"konnectivity-agent-wgbw2\" (UID: \"7a9f7d3b-8a21-40ed-ab47-067f97ed537a\") " pod="kube-system/konnectivity-agent-wgbw2"
Sep 10 15:27:06.789048 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.788165    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/96f18013-00ea-4663-87b5-4bca49f4295a-host-cni-bin\") pod \"ovnkube-node-sdjq9\" (UID: \"96f18013-00ea-4663-87b5-4bca49f4295a\") " pod="openshift-ovn-kubernetes/ovnkube-node-sdjq9"
Sep 10 15:27:06.789048 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.788192    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/96b96370-410d-447c-8fb0-63f56ab52310-dbus\") pod \"global-pull-secret-syncer-7kktn\" (UID: \"96b96370-410d-447c-8fb0-63f56ab52310\") " pod="kube-system/global-pull-secret-syncer-7kktn"
Sep 10 15:27:06.789048 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.788215    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/153abd57-b263-4386-9087-f9138c09e4a0-run\") pod \"tuned-zxgvr\" (UID: \"153abd57-b263-4386-9087-f9138c09e4a0\") " pod="openshift-cluster-node-tuning-operator/tuned-zxgvr"
Sep 10 15:27:06.789048 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.788244    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdccr\" (UniqueName: \"kubernetes.io/projected/87bd161e-974d-49ee-861d-22763253529c-kube-api-access-fdccr\") pod \"network-metrics-daemon-w24m7\" (UID: \"87bd161e-974d-49ee-861d-22763253529c\") " pod="openshift-multus/network-metrics-daemon-w24m7"
Sep 10 15:27:06.789048 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.788272    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dd421353-8e9c-4660-8224-cb11e8575e7a-system-cni-dir\") pod \"multus-additional-cni-plugins-6nx2l\" (UID: \"dd421353-8e9c-4660-8224-cb11e8575e7a\") " pod="openshift-multus/multus-additional-cni-plugins-6nx2l"
Sep 10 15:27:06.789048 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.788321    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/153abd57-b263-4386-9087-f9138c09e4a0-etc-systemd\") pod \"tuned-zxgvr\" (UID: \"153abd57-b263-4386-9087-f9138c09e4a0\") " pod="openshift-cluster-node-tuning-operator/tuned-zxgvr"
Sep 10 15:27:06.789048 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.788384    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/96b96370-410d-447c-8fb0-63f56ab52310-dbus\") pod \"global-pull-secret-syncer-7kktn\" (UID: \"96b96370-410d-447c-8fb0-63f56ab52310\") " pod="kube-system/global-pull-secret-syncer-7kktn"
Sep 10 15:27:06.789048 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.788418    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c56c45fc-a485-48b6-a88a-355ba442927f-tmp-dir\") pod \"node-resolver-md7dv\" (UID: \"c56c45fc-a485-48b6-a88a-355ba442927f\") " pod="openshift-dns/node-resolver-md7dv"
Sep 10 15:27:06.789048 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.788389    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/153abd57-b263-4386-9087-f9138c09e4a0-lib-modules\") pod \"tuned-zxgvr\" (UID: \"153abd57-b263-4386-9087-f9138c09e4a0\") " pod="openshift-cluster-node-tuning-operator/tuned-zxgvr"
Sep 10 15:27:06.789048 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.788464    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/96f18013-00ea-4663-87b5-4bca49f4295a-host-cni-netd\") pod \"ovnkube-node-sdjq9\" (UID: \"96f18013-00ea-4663-87b5-4bca49f4295a\") " pod="openshift-ovn-kubernetes/ovnkube-node-sdjq9"
Sep 10 15:27:06.789048 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.788495    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dd421353-8e9c-4660-8224-cb11e8575e7a-system-cni-dir\") pod \"multus-additional-cni-plugins-6nx2l\" (UID: \"dd421353-8e9c-4660-8224-cb11e8575e7a\") " pod="openshift-multus/multus-additional-cni-plugins-6nx2l"
Sep 10 15:27:06.789048 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.788494    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/96f18013-00ea-4663-87b5-4bca49f4295a-ovnkube-script-lib\") pod \"ovnkube-node-sdjq9\" (UID: \"96f18013-00ea-4663-87b5-4bca49f4295a\") " pod="openshift-ovn-kubernetes/ovnkube-node-sdjq9"
Sep 10 15:27:06.789706 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.788534    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1a08685b-d954-46be-8456-410c35f09c93-cnibin\") pod \"multus-vdtg4\" (UID: \"1a08685b-d954-46be-8456-410c35f09c93\") " pod="openshift-multus/multus-vdtg4"
Sep 10 15:27:06.789706 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.788548    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/dd421353-8e9c-4660-8224-cb11e8575e7a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6nx2l\" (UID: \"dd421353-8e9c-4660-8224-cb11e8575e7a\") " pod="openshift-multus/multus-additional-cni-plugins-6nx2l"
Sep 10 15:27:06.789706 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.788550    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1a08685b-d954-46be-8456-410c35f09c93-host-run-netns\") pod \"multus-vdtg4\" (UID: \"1a08685b-d954-46be-8456-410c35f09c93\") " pod="openshift-multus/multus-vdtg4"
Sep 10 15:27:06.789706 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.788627    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0cf24a6f-168a-4076-a08e-6b30b6f34234-registration-dir\") pod \"aws-ebs-csi-driver-node-kcrzq\" (UID: \"0cf24a6f-168a-4076-a08e-6b30b6f34234\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kcrzq"
Sep 10 15:27:06.789706 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.788673    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/153abd57-b263-4386-9087-f9138c09e4a0-etc-tuned\") pod \"tuned-zxgvr\" (UID: \"153abd57-b263-4386-9087-f9138c09e4a0\") " pod="openshift-cluster-node-tuning-operator/tuned-zxgvr"
Sep 10 15:27:06.789706 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.788702    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3f10d0b5-445b-4531-a2ff-ea0264c02ef5-host-slash\") pod \"iptables-alerter-fkmfw\" (UID: \"3f10d0b5-445b-4531-a2ff-ea0264c02ef5\") " pod="openshift-network-operator/iptables-alerter-fkmfw"
Sep 10 15:27:06.789706 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.788731    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/dd421353-8e9c-4660-8224-cb11e8575e7a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6nx2l\" (UID: \"dd421353-8e9c-4660-8224-cb11e8575e7a\") " pod="openshift-multus/multus-additional-cni-plugins-6nx2l"
Sep 10 15:27:06.789706 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.788757    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c56c45fc-a485-48b6-a88a-355ba442927f-hosts-file\") pod \"node-resolver-md7dv\" (UID: \"c56c45fc-a485-48b6-a88a-355ba442927f\") " pod="openshift-dns/node-resolver-md7dv"
Sep 10 15:27:06.789706 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.788785    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1a08685b-d954-46be-8456-410c35f09c93-host-run-multus-certs\") pod \"multus-vdtg4\" (UID: \"1a08685b-d954-46be-8456-410c35f09c93\") " pod="openshift-multus/multus-vdtg4"
Sep 10 15:27:06.789706 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.788825    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1a08685b-d954-46be-8456-410c35f09c93-etc-kubernetes\") pod \"multus-vdtg4\" (UID: \"1a08685b-d954-46be-8456-410c35f09c93\") " pod="openshift-multus/multus-vdtg4"
Sep 10 15:27:06.789706 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.788860    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c56c45fc-a485-48b6-a88a-355ba442927f-hosts-file\") pod \"node-resolver-md7dv\" (UID: \"c56c45fc-a485-48b6-a88a-355ba442927f\") " pod="openshift-dns/node-resolver-md7dv"
Sep 10 15:27:06.789706 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.789147    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/dd421353-8e9c-4660-8224-cb11e8575e7a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6nx2l\" (UID: \"dd421353-8e9c-4660-8224-cb11e8575e7a\") " pod="openshift-multus/multus-additional-cni-plugins-6nx2l"
Sep 10 15:27:06.794390 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:06.794363    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 15:27:06.794390 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:06.794394    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 15:27:06.794587 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:06.794408    2559 projected.go:196] Error preparing data for projected volume kube-api-access-wx7jm for pod openshift-network-diagnostics/network-check-target-qbx8c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 15:27:06.794587 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:06.794473    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/008da33d-29cc-4f79-b53a-d1a674b604e8-kube-api-access-wx7jm podName:008da33d-29cc-4f79-b53a-d1a674b604e8 nodeName:}" failed. No retries permitted until 2026-09-10 15:27:07.294455328 +0000 UTC m=+3.036420433 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-wx7jm" (UniqueName: "kubernetes.io/projected/008da33d-29cc-4f79-b53a-d1a674b604e8-kube-api-access-wx7jm") pod "network-check-target-qbx8c" (UID: "008da33d-29cc-4f79-b53a-d1a674b604e8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 15:27:06.794970 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.794927    2559 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 10 15:27:06.801348 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.801319    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-chmls\" (UniqueName: \"kubernetes.io/projected/c56c45fc-a485-48b6-a88a-355ba442927f-kube-api-access-chmls\") pod \"node-resolver-md7dv\" (UID: \"c56c45fc-a485-48b6-a88a-355ba442927f\") " pod="openshift-dns/node-resolver-md7dv"
Sep 10 15:27:06.801495 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.801325    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbw4r\" (UniqueName: \"kubernetes.io/projected/dd421353-8e9c-4660-8224-cb11e8575e7a-kube-api-access-vbw4r\") pod \"multus-additional-cni-plugins-6nx2l\" (UID: \"dd421353-8e9c-4660-8224-cb11e8575e7a\") " pod="openshift-multus/multus-additional-cni-plugins-6nx2l"
Sep 10 15:27:06.801495 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.801474    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8jbt\" (UniqueName: \"kubernetes.io/projected/6db07abf-0c03-4565-a661-9df53fbe5957-kube-api-access-b8jbt\") pod \"node-ca-pzn2d\" (UID: \"6db07abf-0c03-4565-a661-9df53fbe5957\") " pod="openshift-image-registry/node-ca-pzn2d"
Sep 10 15:27:06.803078 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.803055    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdccr\" (UniqueName: \"kubernetes.io/projected/87bd161e-974d-49ee-861d-22763253529c-kube-api-access-fdccr\") pod \"network-metrics-daemon-w24m7\" (UID: \"87bd161e-974d-49ee-861d-22763253529c\") " pod="openshift-multus/network-metrics-daemon-w24m7"
Sep 10 15:27:06.889224 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.889176    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/153abd57-b263-4386-9087-f9138c09e4a0-etc-systemd\") pod \"tuned-zxgvr\" (UID: \"153abd57-b263-4386-9087-f9138c09e4a0\") " pod="openshift-cluster-node-tuning-operator/tuned-zxgvr"
Sep 10 15:27:06.889399 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.889234    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/153abd57-b263-4386-9087-f9138c09e4a0-lib-modules\") pod \"tuned-zxgvr\" (UID: \"153abd57-b263-4386-9087-f9138c09e4a0\") " pod="openshift-cluster-node-tuning-operator/tuned-zxgvr"
Sep 10 15:27:06.889399 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.889262    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/96f18013-00ea-4663-87b5-4bca49f4295a-host-cni-netd\") pod \"ovnkube-node-sdjq9\" (UID: \"96f18013-00ea-4663-87b5-4bca49f4295a\") " pod="openshift-ovn-kubernetes/ovnkube-node-sdjq9"
Sep 10 15:27:06.889399 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.889291    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/96f18013-00ea-4663-87b5-4bca49f4295a-ovnkube-script-lib\") pod \"ovnkube-node-sdjq9\" (UID: \"96f18013-00ea-4663-87b5-4bca49f4295a\") " pod="openshift-ovn-kubernetes/ovnkube-node-sdjq9"
Sep 10 15:27:06.889399 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.889319    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1a08685b-d954-46be-8456-410c35f09c93-cnibin\") pod \"multus-vdtg4\" (UID: \"1a08685b-d954-46be-8456-410c35f09c93\") " pod="openshift-multus/multus-vdtg4"
Sep 10 15:27:06.889399 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.889332    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/153abd57-b263-4386-9087-f9138c09e4a0-etc-systemd\") pod \"tuned-zxgvr\" (UID: \"153abd57-b263-4386-9087-f9138c09e4a0\") " pod="openshift-cluster-node-tuning-operator/tuned-zxgvr"
Sep 10 15:27:06.889399 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.889345    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1a08685b-d954-46be-8456-410c35f09c93-host-run-netns\") pod \"multus-vdtg4\" (UID: \"1a08685b-d954-46be-8456-410c35f09c93\") " pod="openshift-multus/multus-vdtg4"
Sep 10 15:27:06.889399 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.889372    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0cf24a6f-168a-4076-a08e-6b30b6f34234-registration-dir\") pod \"aws-ebs-csi-driver-node-kcrzq\" (UID: \"0cf24a6f-168a-4076-a08e-6b30b6f34234\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kcrzq"
Sep 10 15:27:06.889399 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.889394    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/153abd57-b263-4386-9087-f9138c09e4a0-lib-modules\") pod \"tuned-zxgvr\" (UID: \"153abd57-b263-4386-9087-f9138c09e4a0\") " pod="openshift-cluster-node-tuning-operator/tuned-zxgvr"
Sep 10 15:27:06.889748 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.889398    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/153abd57-b263-4386-9087-f9138c09e4a0-etc-tuned\") pod \"tuned-zxgvr\" (UID: \"153abd57-b263-4386-9087-f9138c09e4a0\") " pod="openshift-cluster-node-tuning-operator/tuned-zxgvr"
Sep 10 15:27:06.889748 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.889418    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/96f18013-00ea-4663-87b5-4bca49f4295a-host-cni-netd\") pod \"ovnkube-node-sdjq9\" (UID: \"96f18013-00ea-4663-87b5-4bca49f4295a\") " pod="openshift-ovn-kubernetes/ovnkube-node-sdjq9"
Sep 10 15:27:06.889748 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.889446    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3f10d0b5-445b-4531-a2ff-ea0264c02ef5-host-slash\") pod \"iptables-alerter-fkmfw\" (UID: \"3f10d0b5-445b-4531-a2ff-ea0264c02ef5\") " pod="openshift-network-operator/iptables-alerter-fkmfw"
Sep 10 15:27:06.889748 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.889465    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1a08685b-d954-46be-8456-410c35f09c93-host-run-netns\") pod \"multus-vdtg4\" (UID: \"1a08685b-d954-46be-8456-410c35f09c93\") " pod="openshift-multus/multus-vdtg4"
Sep 10 15:27:06.889748 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.889477    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1a08685b-d954-46be-8456-410c35f09c93-host-run-multus-certs\") pod \"multus-vdtg4\" (UID: \"1a08685b-d954-46be-8456-410c35f09c93\") " pod="openshift-multus/multus-vdtg4"
Sep 10 15:27:06.889748 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.889505    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1a08685b-d954-46be-8456-410c35f09c93-etc-kubernetes\") pod \"multus-vdtg4\" (UID: \"1a08685b-d954-46be-8456-410c35f09c93\") " pod="openshift-multus/multus-vdtg4"
Sep 10 15:27:06.889748 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.889521    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1a08685b-d954-46be-8456-410c35f09c93-cnibin\") pod \"multus-vdtg4\" (UID: \"1a08685b-d954-46be-8456-410c35f09c93\") " pod="openshift-multus/multus-vdtg4"
Sep 10 15:27:06.889748 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.889534    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0cf24a6f-168a-4076-a08e-6b30b6f34234-socket-dir\") pod \"aws-ebs-csi-driver-node-kcrzq\" (UID: \"0cf24a6f-168a-4076-a08e-6b30b6f34234\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kcrzq"
Sep 10 15:27:06.889748 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.889578    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0cf24a6f-168a-4076-a08e-6b30b6f34234-registration-dir\") pod \"aws-ebs-csi-driver-node-kcrzq\" (UID: \"0cf24a6f-168a-4076-a08e-6b30b6f34234\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kcrzq"
Sep 10 15:27:06.889748 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.889591    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/153abd57-b263-4386-9087-f9138c09e4a0-etc-modprobe-d\") pod \"tuned-zxgvr\" (UID: \"153abd57-b263-4386-9087-f9138c09e4a0\") " pod="openshift-cluster-node-tuning-operator/tuned-zxgvr"
Sep 10 15:27:06.889748 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.889641    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/153abd57-b263-4386-9087-f9138c09e4a0-var-lib-kubelet\") pod \"tuned-zxgvr\" (UID: \"153abd57-b263-4386-9087-f9138c09e4a0\") " pod="openshift-cluster-node-tuning-operator/tuned-zxgvr"
Sep 10 15:27:06.889748 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.889649    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1a08685b-d954-46be-8456-410c35f09c93-host-run-multus-certs\") pod \"multus-vdtg4\" (UID: \"1a08685b-d954-46be-8456-410c35f09c93\") " pod="openshift-multus/multus-vdtg4"
Sep 10 15:27:06.889748 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.889669    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/96f18013-00ea-4663-87b5-4bca49f4295a-run-openvswitch\") pod \"ovnkube-node-sdjq9\" (UID: \"96f18013-00ea-4663-87b5-4bca49f4295a\") " pod="openshift-ovn-kubernetes/ovnkube-node-sdjq9"
Sep 10 15:27:06.889748 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.889686    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3f10d0b5-445b-4531-a2ff-ea0264c02ef5-host-slash\") pod \"iptables-alerter-fkmfw\" (UID: \"3f10d0b5-445b-4531-a2ff-ea0264c02ef5\") " pod="openshift-network-operator/iptables-alerter-fkmfw"
Sep 10 15:27:06.889748 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.889695    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/96f18013-00ea-4663-87b5-4bca49f4295a-env-overrides\") pod \"ovnkube-node-sdjq9\" (UID: \"96f18013-00ea-4663-87b5-4bca49f4295a\") " pod="openshift-ovn-kubernetes/ovnkube-node-sdjq9"
Sep 10 15:27:06.889748 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.889721    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1a08685b-d954-46be-8456-410c35f09c93-host-var-lib-kubelet\") pod \"multus-vdtg4\" (UID: \"1a08685b-d954-46be-8456-410c35f09c93\") " pod="openshift-multus/multus-vdtg4"
Sep 10 15:27:06.889748 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.889751    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0cf24a6f-168a-4076-a08e-6b30b6f34234-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kcrzq\" (UID: \"0cf24a6f-168a-4076-a08e-6b30b6f34234\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kcrzq"
Sep 10 15:27:06.890593 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.889787    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87pf9\" (UniqueName: \"kubernetes.io/projected/0cf24a6f-168a-4076-a08e-6b30b6f34234-kube-api-access-87pf9\") pod \"aws-ebs-csi-driver-node-kcrzq\" (UID: \"0cf24a6f-168a-4076-a08e-6b30b6f34234\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kcrzq"
Sep 10 15:27:06.890593 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.889787    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1a08685b-d954-46be-8456-410c35f09c93-etc-kubernetes\") pod \"multus-vdtg4\" (UID: \"1a08685b-d954-46be-8456-410c35f09c93\") " pod="openshift-multus/multus-vdtg4"
Sep 10 15:27:06.890593 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.889840    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/153abd57-b263-4386-9087-f9138c09e4a0-etc-sysctl-conf\") pod \"tuned-zxgvr\" (UID: \"153abd57-b263-4386-9087-f9138c09e4a0\") " pod="openshift-cluster-node-tuning-operator/tuned-zxgvr"
Sep 10 15:27:06.890593 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.889868    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1a08685b-d954-46be-8456-410c35f09c93-multus-conf-dir\") pod \"multus-vdtg4\" (UID: \"1a08685b-d954-46be-8456-410c35f09c93\") " pod="openshift-multus/multus-vdtg4"
Sep 10 15:27:06.890593 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.889897    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7a9f7d3b-8a21-40ed-ab47-067f97ed537a-konnectivity-ca\") pod \"konnectivity-agent-wgbw2\" (UID: \"7a9f7d3b-8a21-40ed-ab47-067f97ed537a\") " pod="kube-system/konnectivity-agent-wgbw2"
Sep 10 15:27:06.890593 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.889927    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/96f18013-00ea-4663-87b5-4bca49f4295a-etc-openvswitch\") pod \"ovnkube-node-sdjq9\" (UID: \"96f18013-00ea-4663-87b5-4bca49f4295a\") " pod="openshift-ovn-kubernetes/ovnkube-node-sdjq9"
Sep 10 15:27:06.890593 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.889944    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/153abd57-b263-4386-9087-f9138c09e4a0-etc-sysconfig\") pod \"tuned-zxgvr\" (UID: \"153abd57-b263-4386-9087-f9138c09e4a0\") " pod="openshift-cluster-node-tuning-operator/tuned-zxgvr"
Sep 10 15:27:06.890593 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.889961    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/96f18013-00ea-4663-87b5-4bca49f4295a-host-run-netns\") pod \"ovnkube-node-sdjq9\" (UID: \"96f18013-00ea-4663-87b5-4bca49f4295a\") " pod="openshift-ovn-kubernetes/ovnkube-node-sdjq9"
Sep 10 15:27:06.890593 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.889978    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/96f18013-00ea-4663-87b5-4bca49f4295a-node-log\") pod \"ovnkube-node-sdjq9\" (UID: \"96f18013-00ea-4663-87b5-4bca49f4295a\") " pod="openshift-ovn-kubernetes/ovnkube-node-sdjq9"
Sep 10 15:27:06.890593 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.890001    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/96f18013-00ea-4663-87b5-4bca49f4295a-ovnkube-config\") pod \"ovnkube-node-sdjq9\" (UID: \"96f18013-00ea-4663-87b5-4bca49f4295a\") " pod="openshift-ovn-kubernetes/ovnkube-node-sdjq9"
Sep 10 15:27:06.890593 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.890018    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/96f18013-00ea-4663-87b5-4bca49f4295a-ovn-node-metrics-cert\") pod \"ovnkube-node-sdjq9\" (UID: \"96f18013-00ea-4663-87b5-4bca49f4295a\") " pod="openshift-ovn-kubernetes/ovnkube-node-sdjq9"
Sep 10 15:27:06.890593 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.890038    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1a08685b-d954-46be-8456-410c35f09c93-host-var-lib-cni-multus\") pod \"multus-vdtg4\" (UID: \"1a08685b-d954-46be-8456-410c35f09c93\") " pod="openshift-multus/multus-vdtg4"
Sep 10 15:27:06.890593 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.890076    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nh2lw\" (UniqueName: \"kubernetes.io/projected/153abd57-b263-4386-9087-f9138c09e4a0-kube-api-access-nh2lw\") pod \"tuned-zxgvr\" (UID: \"153abd57-b263-4386-9087-f9138c09e4a0\") " pod="openshift-cluster-node-tuning-operator/tuned-zxgvr"
Sep 10 15:27:06.890593 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.890104    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2fsc\" (UniqueName: \"kubernetes.io/projected/96f18013-00ea-4663-87b5-4bca49f4295a-kube-api-access-p2fsc\") pod \"ovnkube-node-sdjq9\" (UID: \"96f18013-00ea-4663-87b5-4bca49f4295a\") " pod="openshift-ovn-kubernetes/ovnkube-node-sdjq9"
Sep 10 15:27:06.890593 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.890119    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/96f18013-00ea-4663-87b5-4bca49f4295a-run-openvswitch\") pod \"ovnkube-node-sdjq9\" (UID: \"96f18013-00ea-4663-87b5-4bca49f4295a\") " pod="openshift-ovn-kubernetes/ovnkube-node-sdjq9"
Sep 10 15:27:06.890593 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.890139    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1a08685b-d954-46be-8456-410c35f09c93-multus-daemon-config\") pod \"multus-vdtg4\" (UID: \"1a08685b-d954-46be-8456-410c35f09c93\") " pod="openshift-multus/multus-vdtg4"
Sep 10 15:27:06.890593 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.890176    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0cf24a6f-168a-4076-a08e-6b30b6f34234-device-dir\") pod \"aws-ebs-csi-driver-node-kcrzq\" (UID: \"0cf24a6f-168a-4076-a08e-6b30b6f34234\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kcrzq"
Sep 10 15:27:06.891372 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.890204    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/153abd57-b263-4386-9087-f9138c09e4a0-sys\") pod \"tuned-zxgvr\" (UID: \"153abd57-b263-4386-9087-f9138c09e4a0\") " pod="openshift-cluster-node-tuning-operator/tuned-zxgvr"
Sep 10 15:27:06.891372 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.890221    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/96f18013-00ea-4663-87b5-4bca49f4295a-host-slash\") pod \"ovnkube-node-sdjq9\" (UID: \"96f18013-00ea-4663-87b5-4bca49f4295a\") " pod="openshift-ovn-kubernetes/ovnkube-node-sdjq9"
Sep 10 15:27:06.891372 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.890237    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/96f18013-00ea-4663-87b5-4bca49f4295a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-sdjq9\" (UID: \"96f18013-00ea-4663-87b5-4bca49f4295a\") " pod="openshift-ovn-kubernetes/ovnkube-node-sdjq9"
Sep 10 15:27:06.891372 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.890263    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1a08685b-d954-46be-8456-410c35f09c93-cni-binary-copy\") pod \"multus-vdtg4\" (UID: \"1a08685b-d954-46be-8456-410c35f09c93\") " pod="openshift-multus/multus-vdtg4"
Sep 10 15:27:06.891372 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.890279    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1a08685b-d954-46be-8456-410c35f09c93-multus-socket-dir-parent\") pod \"multus-vdtg4\" (UID: \"1a08685b-d954-46be-8456-410c35f09c93\") " pod="openshift-multus/multus-vdtg4"
Sep 10 15:27:06.891372 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.890310    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0cf24a6f-168a-4076-a08e-6b30b6f34234-etc-selinux\") pod \"aws-ebs-csi-driver-node-kcrzq\" (UID: \"0cf24a6f-168a-4076-a08e-6b30b6f34234\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kcrzq"
Sep 10 15:27:06.891372 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.890336    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1a08685b-d954-46be-8456-410c35f09c93-host-var-lib-cni-bin\") pod \"multus-vdtg4\" (UID: \"1a08685b-d954-46be-8456-410c35f09c93\") " pod="openshift-multus/multus-vdtg4"
Sep 10 15:27:06.891372 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.890355    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/96f18013-00ea-4663-87b5-4bca49f4295a-ovnkube-script-lib\") pod \"ovnkube-node-sdjq9\" (UID: \"96f18013-00ea-4663-87b5-4bca49f4295a\") " pod="openshift-ovn-kubernetes/ovnkube-node-sdjq9"
Sep 10 15:27:06.891372 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.890366    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcnpj\" (UniqueName: \"kubernetes.io/projected/3f10d0b5-445b-4531-a2ff-ea0264c02ef5-kube-api-access-bcnpj\") pod \"iptables-alerter-fkmfw\" (UID: \"3f10d0b5-445b-4531-a2ff-ea0264c02ef5\") " pod="openshift-network-operator/iptables-alerter-fkmfw"
Sep 10 15:27:06.891372 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.890403    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0cf24a6f-168a-4076-a08e-6b30b6f34234-sys-fs\") pod \"aws-ebs-csi-driver-node-kcrzq\" (UID: \"0cf24a6f-168a-4076-a08e-6b30b6f34234\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kcrzq"
Sep 10 15:27:06.891372 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.890423    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/96f18013-00ea-4663-87b5-4bca49f4295a-run-systemd\") pod \"ovnkube-node-sdjq9\" (UID: \"96f18013-00ea-4663-87b5-4bca49f4295a\") " pod="openshift-ovn-kubernetes/ovnkube-node-sdjq9"
Sep 10 15:27:06.891372 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.890445    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/96f18013-00ea-4663-87b5-4bca49f4295a-run-ovn\") pod \"ovnkube-node-sdjq9\" (UID: \"96f18013-00ea-4663-87b5-4bca49f4295a\") " pod="openshift-ovn-kubernetes/ovnkube-node-sdjq9"
Sep 10 15:27:06.891372 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.890465    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/96f18013-00ea-4663-87b5-4bca49f4295a-host-run-ovn-kubernetes\") pod \"ovnkube-node-sdjq9\" (UID: \"96f18013-00ea-4663-87b5-4bca49f4295a\") " pod="openshift-ovn-kubernetes/ovnkube-node-sdjq9"
Sep 10 15:27:06.891372 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.890481    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1a08685b-d954-46be-8456-410c35f09c93-hostroot\") pod \"multus-vdtg4\" (UID: \"1a08685b-d954-46be-8456-410c35f09c93\") " pod="openshift-multus/multus-vdtg4"
Sep 10 15:27:06.891372 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.890497    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/96f18013-00ea-4663-87b5-4bca49f4295a-log-socket\") pod \"ovnkube-node-sdjq9\" (UID: \"96f18013-00ea-4663-87b5-4bca49f4295a\") " pod="openshift-ovn-kubernetes/ovnkube-node-sdjq9"
Sep 10 15:27:06.891372 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.890497    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/96f18013-00ea-4663-87b5-4bca49f4295a-env-overrides\") pod \"ovnkube-node-sdjq9\" (UID: \"96f18013-00ea-4663-87b5-4bca49f4295a\") " pod="openshift-ovn-kubernetes/ovnkube-node-sdjq9"
Sep 10 15:27:06.891372 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.890500    2559 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 15:27:06.891372 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.890542    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/153abd57-b263-4386-9087-f9138c09e4a0-var-lib-kubelet\") pod \"tuned-zxgvr\" (UID: \"153abd57-b263-4386-9087-f9138c09e4a0\") " pod="openshift-cluster-node-tuning-operator/tuned-zxgvr"
Sep 10 15:27:06.892169 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.890592    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1a08685b-d954-46be-8456-410c35f09c93-host-var-lib-kubelet\") pod \"multus-vdtg4\" (UID: \"1a08685b-d954-46be-8456-410c35f09c93\") " pod="openshift-multus/multus-vdtg4"
Sep 10 15:27:06.892169 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.890643    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/153abd57-b263-4386-9087-f9138c09e4a0-etc-sysctl-d\") pod \"tuned-zxgvr\" (UID: \"153abd57-b263-4386-9087-f9138c09e4a0\") " pod="openshift-cluster-node-tuning-operator/tuned-zxgvr"
Sep 10 15:27:06.892169 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.890759    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0cf24a6f-168a-4076-a08e-6b30b6f34234-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kcrzq\" (UID: \"0cf24a6f-168a-4076-a08e-6b30b6f34234\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kcrzq"
Sep 10 15:27:06.892169 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.890521    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/153abd57-b263-4386-9087-f9138c09e4a0-etc-sysctl-d\") pod \"tuned-zxgvr\" (UID: \"153abd57-b263-4386-9087-f9138c09e4a0\") " pod="openshift-cluster-node-tuning-operator/tuned-zxgvr"
Sep 10 15:27:06.892169 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.890790    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0cf24a6f-168a-4076-a08e-6b30b6f34234-socket-dir\") pod \"aws-ebs-csi-driver-node-kcrzq\" (UID: \"0cf24a6f-168a-4076-a08e-6b30b6f34234\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kcrzq"
Sep 10 15:27:06.892169 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.890815    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/96f18013-00ea-4663-87b5-4bca49f4295a-var-lib-openvswitch\") pod \"ovnkube-node-sdjq9\" (UID: \"96f18013-00ea-4663-87b5-4bca49f4295a\") " pod="openshift-ovn-kubernetes/ovnkube-node-sdjq9"
Sep 10 15:27:06.892169 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.890846    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fc4rw\" (UniqueName: \"kubernetes.io/projected/1a08685b-d954-46be-8456-410c35f09c93-kube-api-access-fc4rw\") pod \"multus-vdtg4\" (UID: \"1a08685b-d954-46be-8456-410c35f09c93\") " pod="openshift-multus/multus-vdtg4"
Sep 10 15:27:06.892169 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.890874    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/153abd57-b263-4386-9087-f9138c09e4a0-host\") pod \"tuned-zxgvr\" (UID: \"153abd57-b263-4386-9087-f9138c09e4a0\") " pod="openshift-cluster-node-tuning-operator/tuned-zxgvr"
Sep 10 15:27:06.892169 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.890893    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/153abd57-b263-4386-9087-f9138c09e4a0-etc-modprobe-d\") pod \"tuned-zxgvr\" (UID: \"153abd57-b263-4386-9087-f9138c09e4a0\") " pod="openshift-cluster-node-tuning-operator/tuned-zxgvr"
Sep 10 15:27:06.892169 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.890898    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/96f18013-00ea-4663-87b5-4bca49f4295a-host-kubelet\") pod \"ovnkube-node-sdjq9\" (UID: \"96f18013-00ea-4663-87b5-4bca49f4295a\") " pod="openshift-ovn-kubernetes/ovnkube-node-sdjq9"
Sep 10 15:27:06.892169 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.890946    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/96f18013-00ea-4663-87b5-4bca49f4295a-host-kubelet\") pod \"ovnkube-node-sdjq9\" (UID: \"96f18013-00ea-4663-87b5-4bca49f4295a\") " pod="openshift-ovn-kubernetes/ovnkube-node-sdjq9"
Sep 10 15:27:06.892169 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.890958    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/96f18013-00ea-4663-87b5-4bca49f4295a-systemd-units\") pod \"ovnkube-node-sdjq9\" (UID: \"96f18013-00ea-4663-87b5-4bca49f4295a\") " pod="openshift-ovn-kubernetes/ovnkube-node-sdjq9"
Sep 10 15:27:06.892169 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.890976    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1a08685b-d954-46be-8456-410c35f09c93-system-cni-dir\") pod \"multus-vdtg4\" (UID: \"1a08685b-d954-46be-8456-410c35f09c93\") " pod="openshift-multus/multus-vdtg4"
Sep 10 15:27:06.892169 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.890991    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/96f18013-00ea-4663-87b5-4bca49f4295a-var-lib-openvswitch\") pod \"ovnkube-node-sdjq9\" (UID: \"96f18013-00ea-4663-87b5-4bca49f4295a\") " pod="openshift-ovn-kubernetes/ovnkube-node-sdjq9"
Sep 10 15:27:06.892169 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.891000    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1a08685b-d954-46be-8456-410c35f09c93-multus-cni-dir\") pod \"multus-vdtg4\" (UID: \"1a08685b-d954-46be-8456-410c35f09c93\") " pod="openshift-multus/multus-vdtg4"
Sep 10 15:27:06.892169 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.891135    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1a08685b-d954-46be-8456-410c35f09c93-multus-cni-dir\") pod \"multus-vdtg4\" (UID: \"1a08685b-d954-46be-8456-410c35f09c93\") " pod="openshift-multus/multus-vdtg4"
Sep 10 15:27:06.892169 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.891241    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/153abd57-b263-4386-9087-f9138c09e4a0-etc-sysctl-conf\") pod \"tuned-zxgvr\" (UID: \"153abd57-b263-4386-9087-f9138c09e4a0\") " pod="openshift-cluster-node-tuning-operator/tuned-zxgvr"
Sep 10 15:27:06.892169 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.891256    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/153abd57-b263-4386-9087-f9138c09e4a0-host\") pod \"tuned-zxgvr\" (UID: \"153abd57-b263-4386-9087-f9138c09e4a0\") " pod="openshift-cluster-node-tuning-operator/tuned-zxgvr"
Sep 10 15:27:06.893222 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.891273    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1a08685b-d954-46be-8456-410c35f09c93-multus-conf-dir\") pod \"multus-vdtg4\" (UID: \"1a08685b-d954-46be-8456-410c35f09c93\") " pod="openshift-multus/multus-vdtg4"
Sep 10 15:27:06.893222 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.891309    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/153abd57-b263-4386-9087-f9138c09e4a0-etc-kubernetes\") pod \"tuned-zxgvr\" (UID: \"153abd57-b263-4386-9087-f9138c09e4a0\") " pod="openshift-cluster-node-tuning-operator/tuned-zxgvr"
Sep 10 15:27:06.893222 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.891328    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/96f18013-00ea-4663-87b5-4bca49f4295a-host-slash\") pod \"ovnkube-node-sdjq9\" (UID: \"96f18013-00ea-4663-87b5-4bca49f4295a\") " pod="openshift-ovn-kubernetes/ovnkube-node-sdjq9"
Sep 10 15:27:06.893222 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.891339    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1a08685b-d954-46be-8456-410c35f09c93-host-run-k8s-cni-cncf-io\") pod \"multus-vdtg4\" (UID: \"1a08685b-d954-46be-8456-410c35f09c93\") " pod="openshift-multus/multus-vdtg4"
Sep 10 15:27:06.893222 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.891332    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/96f18013-00ea-4663-87b5-4bca49f4295a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-sdjq9\" (UID: \"96f18013-00ea-4663-87b5-4bca49f4295a\") " pod="openshift-ovn-kubernetes/ovnkube-node-sdjq9"
Sep 10 15:27:06.893222 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.891368    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3f10d0b5-445b-4531-a2ff-ea0264c02ef5-iptables-alerter-script\") pod \"iptables-alerter-fkmfw\" (UID: \"3f10d0b5-445b-4531-a2ff-ea0264c02ef5\") " pod="openshift-network-operator/iptables-alerter-fkmfw"
Sep 10 15:27:06.893222 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.891402    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/153abd57-b263-4386-9087-f9138c09e4a0-tmp\") pod \"tuned-zxgvr\" (UID: \"153abd57-b263-4386-9087-f9138c09e4a0\") " pod="openshift-cluster-node-tuning-operator/tuned-zxgvr"
Sep 10 15:27:06.893222 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.891427    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7a9f7d3b-8a21-40ed-ab47-067f97ed537a-agent-certs\") pod \"konnectivity-agent-wgbw2\" (UID: \"7a9f7d3b-8a21-40ed-ab47-067f97ed537a\") " pod="kube-system/konnectivity-agent-wgbw2"
Sep 10 15:27:06.893222 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.891454    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/96f18013-00ea-4663-87b5-4bca49f4295a-host-cni-bin\") pod \"ovnkube-node-sdjq9\" (UID: \"96f18013-00ea-4663-87b5-4bca49f4295a\") " pod="openshift-ovn-kubernetes/ovnkube-node-sdjq9"
Sep 10 15:27:06.893222 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.891478    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/153abd57-b263-4386-9087-f9138c09e4a0-run\") pod \"tuned-zxgvr\" (UID: \"153abd57-b263-4386-9087-f9138c09e4a0\") " pod="openshift-cluster-node-tuning-operator/tuned-zxgvr"
Sep 10 15:27:06.893222 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.891574    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/153abd57-b263-4386-9087-f9138c09e4a0-run\") pod \"tuned-zxgvr\" (UID: \"153abd57-b263-4386-9087-f9138c09e4a0\") " pod="openshift-cluster-node-tuning-operator/tuned-zxgvr"
Sep 10 15:27:06.893222 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.891630    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/153abd57-b263-4386-9087-f9138c09e4a0-etc-kubernetes\") pod \"tuned-zxgvr\" (UID: \"153abd57-b263-4386-9087-f9138c09e4a0\") " pod="openshift-cluster-node-tuning-operator/tuned-zxgvr"
Sep 10 15:27:06.893222 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.891686    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1a08685b-d954-46be-8456-410c35f09c93-host-run-k8s-cni-cncf-io\") pod \"multus-vdtg4\" (UID: \"1a08685b-d954-46be-8456-410c35f09c93\") " pod="openshift-multus/multus-vdtg4"
Sep 10 15:27:06.893222 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.891738    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/153abd57-b263-4386-9087-f9138c09e4a0-etc-sysconfig\") pod \"tuned-zxgvr\" (UID: \"153abd57-b263-4386-9087-f9138c09e4a0\") " pod="openshift-cluster-node-tuning-operator/tuned-zxgvr"
Sep 10 15:27:06.893222 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.891785    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/96f18013-00ea-4663-87b5-4bca49f4295a-etc-openvswitch\") pod \"ovnkube-node-sdjq9\" (UID: \"96f18013-00ea-4663-87b5-4bca49f4295a\") " pod="openshift-ovn-kubernetes/ovnkube-node-sdjq9"
Sep 10 15:27:06.893222 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.891824    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/96f18013-00ea-4663-87b5-4bca49f4295a-host-run-netns\") pod \"ovnkube-node-sdjq9\" (UID: \"96f18013-00ea-4663-87b5-4bca49f4295a\") " pod="openshift-ovn-kubernetes/ovnkube-node-sdjq9"
Sep 10 15:27:06.893222 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.891865    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/96f18013-00ea-4663-87b5-4bca49f4295a-node-log\") pod \"ovnkube-node-sdjq9\" (UID: \"96f18013-00ea-4663-87b5-4bca49f4295a\") " pod="openshift-ovn-kubernetes/ovnkube-node-sdjq9"
Sep 10 15:27:06.893222 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.891875    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1a08685b-d954-46be-8456-410c35f09c93-cni-binary-copy\") pod \"multus-vdtg4\" (UID: \"1a08685b-d954-46be-8456-410c35f09c93\") " pod="openshift-multus/multus-vdtg4"
Sep 10 15:27:06.894207 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.891967    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1a08685b-d954-46be-8456-410c35f09c93-multus-socket-dir-parent\") pod \"multus-vdtg4\" (UID: \"1a08685b-d954-46be-8456-410c35f09c93\") " pod="openshift-multus/multus-vdtg4"
Sep 10 15:27:06.894207 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.892042    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0cf24a6f-168a-4076-a08e-6b30b6f34234-etc-selinux\") pod \"aws-ebs-csi-driver-node-kcrzq\" (UID: \"0cf24a6f-168a-4076-a08e-6b30b6f34234\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kcrzq"
Sep 10 15:27:06.894207 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.892073    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1a08685b-d954-46be-8456-410c35f09c93-host-var-lib-cni-bin\") pod \"multus-vdtg4\" (UID: \"1a08685b-d954-46be-8456-410c35f09c93\") " pod="openshift-multus/multus-vdtg4"
Sep 10 15:27:06.894207 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.892282    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3f10d0b5-445b-4531-a2ff-ea0264c02ef5-iptables-alerter-script\") pod \"iptables-alerter-fkmfw\" (UID: \"3f10d0b5-445b-4531-a2ff-ea0264c02ef5\") " pod="openshift-network-operator/iptables-alerter-fkmfw"
Sep 10 15:27:06.894207 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.892295    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0cf24a6f-168a-4076-a08e-6b30b6f34234-sys-fs\") pod \"aws-ebs-csi-driver-node-kcrzq\" (UID: \"0cf24a6f-168a-4076-a08e-6b30b6f34234\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kcrzq"
Sep 10 15:27:06.894207 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.892320    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/96f18013-00ea-4663-87b5-4bca49f4295a-ovnkube-config\") pod \"ovnkube-node-sdjq9\" (UID: \"96f18013-00ea-4663-87b5-4bca49f4295a\") " pod="openshift-ovn-kubernetes/ovnkube-node-sdjq9"
Sep 10 15:27:06.894207 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.892342    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/96f18013-00ea-4663-87b5-4bca49f4295a-run-ovn\") pod \"ovnkube-node-sdjq9\" (UID: \"96f18013-00ea-4663-87b5-4bca49f4295a\") " pod="openshift-ovn-kubernetes/ovnkube-node-sdjq9"
Sep 10 15:27:06.894207 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.892378    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/96f18013-00ea-4663-87b5-4bca49f4295a-host-run-ovn-kubernetes\") pod \"ovnkube-node-sdjq9\" (UID: \"96f18013-00ea-4663-87b5-4bca49f4295a\") " pod="openshift-ovn-kubernetes/ovnkube-node-sdjq9"
Sep 10 15:27:06.894207 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.892380    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/96f18013-00ea-4663-87b5-4bca49f4295a-host-cni-bin\") pod \"ovnkube-node-sdjq9\" (UID: \"96f18013-00ea-4663-87b5-4bca49f4295a\") " pod="openshift-ovn-kubernetes/ovnkube-node-sdjq9"
Sep 10 15:27:06.894207 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.892406    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1a08685b-d954-46be-8456-410c35f09c93-hostroot\") pod \"multus-vdtg4\" (UID: \"1a08685b-d954-46be-8456-410c35f09c93\") " pod="openshift-multus/multus-vdtg4"
Sep 10 15:27:06.894207 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.892432    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/96f18013-00ea-4663-87b5-4bca49f4295a-log-socket\") pod \"ovnkube-node-sdjq9\" (UID: \"96f18013-00ea-4663-87b5-4bca49f4295a\") " pod="openshift-ovn-kubernetes/ovnkube-node-sdjq9"
Sep 10 15:27:06.894207 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.892470    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/96f18013-00ea-4663-87b5-4bca49f4295a-systemd-units\") pod \"ovnkube-node-sdjq9\" (UID: \"96f18013-00ea-4663-87b5-4bca49f4295a\") " pod="openshift-ovn-kubernetes/ovnkube-node-sdjq9"
Sep 10 15:27:06.894207 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.892510    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1a08685b-d954-46be-8456-410c35f09c93-system-cni-dir\") pod \"multus-vdtg4\" (UID: \"1a08685b-d954-46be-8456-410c35f09c93\") " pod="openshift-multus/multus-vdtg4"
Sep 10 15:27:06.894207 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.892618    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1a08685b-d954-46be-8456-410c35f09c93-host-var-lib-cni-multus\") pod \"multus-vdtg4\" (UID: \"1a08685b-d954-46be-8456-410c35f09c93\") " pod="openshift-multus/multus-vdtg4"
Sep 10 15:27:06.894207 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.892752    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/153abd57-b263-4386-9087-f9138c09e4a0-etc-tuned\") pod \"tuned-zxgvr\" (UID: \"153abd57-b263-4386-9087-f9138c09e4a0\") " pod="openshift-cluster-node-tuning-operator/tuned-zxgvr"
Sep 10 15:27:06.894207 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.892827    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0cf24a6f-168a-4076-a08e-6b30b6f34234-device-dir\") pod \"aws-ebs-csi-driver-node-kcrzq\" (UID: \"0cf24a6f-168a-4076-a08e-6b30b6f34234\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kcrzq"
Sep 10 15:27:06.894207 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.892886    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/153abd57-b263-4386-9087-f9138c09e4a0-sys\") pod \"tuned-zxgvr\" (UID: \"153abd57-b263-4386-9087-f9138c09e4a0\") " pod="openshift-cluster-node-tuning-operator/tuned-zxgvr"
Sep 10 15:27:06.894207 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.893078    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/96f18013-00ea-4663-87b5-4bca49f4295a-run-systemd\") pod \"ovnkube-node-sdjq9\" (UID: \"96f18013-00ea-4663-87b5-4bca49f4295a\") " pod="openshift-ovn-kubernetes/ovnkube-node-sdjq9"
Sep 10 15:27:06.895137 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.893236    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7a9f7d3b-8a21-40ed-ab47-067f97ed537a-konnectivity-ca\") pod \"konnectivity-agent-wgbw2\" (UID: \"7a9f7d3b-8a21-40ed-ab47-067f97ed537a\") " pod="kube-system/konnectivity-agent-wgbw2"
Sep 10 15:27:06.895137 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.893303    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1a08685b-d954-46be-8456-410c35f09c93-multus-daemon-config\") pod \"multus-vdtg4\" (UID: \"1a08685b-d954-46be-8456-410c35f09c93\") " pod="openshift-multus/multus-vdtg4"
Sep 10 15:27:06.895137 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.894758    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/96f18013-00ea-4663-87b5-4bca49f4295a-ovn-node-metrics-cert\") pod \"ovnkube-node-sdjq9\" (UID: \"96f18013-00ea-4663-87b5-4bca49f4295a\") " pod="openshift-ovn-kubernetes/ovnkube-node-sdjq9"
Sep 10 15:27:06.895137 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.895015    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/153abd57-b263-4386-9087-f9138c09e4a0-tmp\") pod \"tuned-zxgvr\" (UID: \"153abd57-b263-4386-9087-f9138c09e4a0\") " pod="openshift-cluster-node-tuning-operator/tuned-zxgvr"
Sep 10 15:27:06.895615 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.895554    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7a9f7d3b-8a21-40ed-ab47-067f97ed537a-agent-certs\") pod \"konnectivity-agent-wgbw2\" (UID: \"7a9f7d3b-8a21-40ed-ab47-067f97ed537a\") " pod="kube-system/konnectivity-agent-wgbw2"
Sep 10 15:27:06.899100 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.899076    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-87pf9\" (UniqueName: \"kubernetes.io/projected/0cf24a6f-168a-4076-a08e-6b30b6f34234-kube-api-access-87pf9\") pod \"aws-ebs-csi-driver-node-kcrzq\" (UID: \"0cf24a6f-168a-4076-a08e-6b30b6f34234\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kcrzq"
Sep 10 15:27:06.904908 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.904584    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcnpj\" (UniqueName: \"kubernetes.io/projected/3f10d0b5-445b-4531-a2ff-ea0264c02ef5-kube-api-access-bcnpj\") pod \"iptables-alerter-fkmfw\" (UID: \"3f10d0b5-445b-4531-a2ff-ea0264c02ef5\") " pod="openshift-network-operator/iptables-alerter-fkmfw"
Sep 10 15:27:06.905317 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.905292    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fc4rw\" (UniqueName: \"kubernetes.io/projected/1a08685b-d954-46be-8456-410c35f09c93-kube-api-access-fc4rw\") pod \"multus-vdtg4\" (UID: \"1a08685b-d954-46be-8456-410c35f09c93\") " pod="openshift-multus/multus-vdtg4"
Sep 10 15:27:06.907561 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.907524    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2fsc\" (UniqueName: \"kubernetes.io/projected/96f18013-00ea-4663-87b5-4bca49f4295a-kube-api-access-p2fsc\") pod \"ovnkube-node-sdjq9\" (UID: \"96f18013-00ea-4663-87b5-4bca49f4295a\") " pod="openshift-ovn-kubernetes/ovnkube-node-sdjq9"
Sep 10 15:27:06.908287 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.908265    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nh2lw\" (UniqueName: \"kubernetes.io/projected/153abd57-b263-4386-9087-f9138c09e4a0-kube-api-access-nh2lw\") pod \"tuned-zxgvr\" (UID: \"153abd57-b263-4386-9087-f9138c09e4a0\") " pod="openshift-cluster-node-tuning-operator/tuned-zxgvr"
Sep 10 15:27:06.982388 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.982351    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6nx2l"
Sep 10 15:27:06.993628 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:06.993603    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-pzn2d"
Sep 10 15:27:07.005635 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:07.005609    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-md7dv"
Sep 10 15:27:07.014627 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:07.014598    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wgbw2"
Sep 10 15:27:07.022514 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:07.022487    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-fkmfw"
Sep 10 15:27:07.031695 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:07.031671    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-sdjq9"
Sep 10 15:27:07.042799 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:07.042772    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vdtg4"
Sep 10 15:27:07.050864 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:07.050841    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kcrzq"
Sep 10 15:27:07.057869 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:07.057843    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zxgvr"
Sep 10 15:27:07.295121 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:07.295078    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wx7jm\" (UniqueName: \"kubernetes.io/projected/008da33d-29cc-4f79-b53a-d1a674b604e8-kube-api-access-wx7jm\") pod \"network-check-target-qbx8c\" (UID: \"008da33d-29cc-4f79-b53a-d1a674b604e8\") " pod="openshift-network-diagnostics/network-check-target-qbx8c"
Sep 10 15:27:07.295321 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:07.295134    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87bd161e-974d-49ee-861d-22763253529c-metrics-certs\") pod \"network-metrics-daemon-w24m7\" (UID: \"87bd161e-974d-49ee-861d-22763253529c\") " pod="openshift-multus/network-metrics-daemon-w24m7"
Sep 10 15:27:07.295321 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:07.295184    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/96b96370-410d-447c-8fb0-63f56ab52310-original-pull-secret\") pod \"global-pull-secret-syncer-7kktn\" (UID: \"96b96370-410d-447c-8fb0-63f56ab52310\") " pod="kube-system/global-pull-secret-syncer-7kktn"
Sep 10 15:27:07.295321 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:07.295301    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 15:27:07.295473 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:07.295334    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 15:27:07.295473 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:07.295411    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87bd161e-974d-49ee-861d-22763253529c-metrics-certs podName:87bd161e-974d-49ee-861d-22763253529c nodeName:}" failed. No retries permitted until 2026-09-10 15:27:08.295389616 +0000 UTC m=+4.037354703 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/87bd161e-974d-49ee-861d-22763253529c-metrics-certs") pod "network-metrics-daemon-w24m7" (UID: "87bd161e-974d-49ee-861d-22763253529c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 15:27:07.295473 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:07.295344    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 15:27:07.295473 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:07.295436    2559 projected.go:196] Error preparing data for projected volume kube-api-access-wx7jm for pod openshift-network-diagnostics/network-check-target-qbx8c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 15:27:07.295674 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:07.295491    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/008da33d-29cc-4f79-b53a-d1a674b604e8-kube-api-access-wx7jm podName:008da33d-29cc-4f79-b53a-d1a674b604e8 nodeName:}" failed. No retries permitted until 2026-09-10 15:27:08.295473111 +0000 UTC m=+4.037438204 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-wx7jm" (UniqueName: "kubernetes.io/projected/008da33d-29cc-4f79-b53a-d1a674b604e8-kube-api-access-wx7jm") pod "network-check-target-qbx8c" (UID: "008da33d-29cc-4f79-b53a-d1a674b604e8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 15:27:07.295674 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:07.295308    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 15:27:07.295674 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:07.295527    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96b96370-410d-447c-8fb0-63f56ab52310-original-pull-secret podName:96b96370-410d-447c-8fb0-63f56ab52310 nodeName:}" failed. No retries permitted until 2026-09-10 15:27:08.295518315 +0000 UTC m=+4.037483398 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/96b96370-410d-447c-8fb0-63f56ab52310-original-pull-secret") pod "global-pull-secret-syncer-7kktn" (UID: "96b96370-410d-447c-8fb0-63f56ab52310") : object "kube-system"/"original-pull-secret" not registered
Sep 10 15:27:07.727877 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:07.727808    2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 15:22:07 +0000 UTC" deadline="2028-05-18 22:14:45.3421259 +0000 UTC"
Sep 10 15:27:07.727877 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:07.727872    2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14790h47m37.614258123s"
Sep 10 15:27:07.779397 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:07.779360    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qbx8c"
Sep 10 15:27:07.779574 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:07.779498    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qbx8c" podUID="008da33d-29cc-4f79-b53a-d1a674b604e8"
Sep 10 15:27:07.981184 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:07.981135    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96f18013_00ea_4663_87b5_4bca49f4295a.slice/crio-d428a25731a9c2b5b54b312cbe9481e40b2a1daf68f0fb738b8f640fe87e4c79 WatchSource:0}: Error finding container d428a25731a9c2b5b54b312cbe9481e40b2a1daf68f0fb738b8f640fe87e4c79: Status 404 returned error can't find the container with id d428a25731a9c2b5b54b312cbe9481e40b2a1daf68f0fb738b8f640fe87e4c79
Sep 10 15:27:07.982129 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:07.982100    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod153abd57_b263_4386_9087_f9138c09e4a0.slice/crio-4dcfb66b24f734963e0696eaf9ac02fb5fa671fb1438b06317a18c0c8096daf5 WatchSource:0}: Error finding container 4dcfb66b24f734963e0696eaf9ac02fb5fa671fb1438b06317a18c0c8096daf5: Status 404 returned error can't find the container with id 4dcfb66b24f734963e0696eaf9ac02fb5fa671fb1438b06317a18c0c8096daf5
Sep 10 15:27:08.005660 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:08.005632    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a9f7d3b_8a21_40ed_ab47_067f97ed537a.slice/crio-f57e0d62a0995855b57dd549fb26e2ad152fb47b72fafe33a81f4592a4b567fb WatchSource:0}: Error finding container f57e0d62a0995855b57dd549fb26e2ad152fb47b72fafe33a81f4592a4b567fb: Status 404 returned error can't find the container with id f57e0d62a0995855b57dd549fb26e2ad152fb47b72fafe33a81f4592a4b567fb
Sep 10 15:27:08.006783 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:08.006743    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cf24a6f_168a_4076_a08e_6b30b6f34234.slice/crio-722aca8ff4daa0df781948603b9794146bd1e4ad0e3acbc75fd8f7b850fc6de9 WatchSource:0}: Error finding container 722aca8ff4daa0df781948603b9794146bd1e4ad0e3acbc75fd8f7b850fc6de9: Status 404 returned error can't find the container with id 722aca8ff4daa0df781948603b9794146bd1e4ad0e3acbc75fd8f7b850fc6de9
Sep 10 15:27:08.008655 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:08.008421    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc56c45fc_a485_48b6_a88a_355ba442927f.slice/crio-5a055137c9c69cc6f25681bef5f96957763a29bc501d4b01135ab1fe1b0011cf WatchSource:0}: Error finding container 5a055137c9c69cc6f25681bef5f96957763a29bc501d4b01135ab1fe1b0011cf: Status 404 returned error can't find the container with id 5a055137c9c69cc6f25681bef5f96957763a29bc501d4b01135ab1fe1b0011cf
Sep 10 15:27:08.009070 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:08.009043    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd421353_8e9c_4660_8224_cb11e8575e7a.slice/crio-1bb1f252c0b94b3d725f5bc2930f7e7297757ea4acf60c2714d5944f0ec09bb6 WatchSource:0}: Error finding container 1bb1f252c0b94b3d725f5bc2930f7e7297757ea4acf60c2714d5944f0ec09bb6: Status 404 returned error can't find the container with id 1bb1f252c0b94b3d725f5bc2930f7e7297757ea4acf60c2714d5944f0ec09bb6
Sep 10 15:27:08.009915 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:08.009892    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f10d0b5_445b_4531_a2ff_ea0264c02ef5.slice/crio-8a846d67ceba82f945944141a10d7b74421be3d032c9c3ffcd517cc2df9431b1 WatchSource:0}: Error finding container 8a846d67ceba82f945944141a10d7b74421be3d032c9c3ffcd517cc2df9431b1: Status 404 returned error can't find the container with id 8a846d67ceba82f945944141a10d7b74421be3d032c9c3ffcd517cc2df9431b1
Sep 10 15:27:08.011471 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:08.011201    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a08685b_d954_46be_8456_410c35f09c93.slice/crio-527243b6b9ffe37c3428b6d2bf283f5e5ce0019fbb371950f7498d699290cc81 WatchSource:0}: Error finding container 527243b6b9ffe37c3428b6d2bf283f5e5ce0019fbb371950f7498d699290cc81: Status 404 returned error can't find the container with id 527243b6b9ffe37c3428b6d2bf283f5e5ce0019fbb371950f7498d699290cc81
Sep 10 15:27:08.013643 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:08.013362    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6db07abf_0c03_4565_a661_9df53fbe5957.slice/crio-aef3d3775d24fb6a4db812200cb922a95a68e0d5b1f4d0e082b6b6813f8df424 WatchSource:0}: Error finding container aef3d3775d24fb6a4db812200cb922a95a68e0d5b1f4d0e082b6b6813f8df424: Status 404 returned error can't find the container with id aef3d3775d24fb6a4db812200cb922a95a68e0d5b1f4d0e082b6b6813f8df424
Sep 10 15:27:08.304392 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:08.304363    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/96b96370-410d-447c-8fb0-63f56ab52310-original-pull-secret\") pod \"global-pull-secret-syncer-7kktn\" (UID: \"96b96370-410d-447c-8fb0-63f56ab52310\") " pod="kube-system/global-pull-secret-syncer-7kktn"
Sep 10 15:27:08.304510 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:08.304418    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wx7jm\" (UniqueName: \"kubernetes.io/projected/008da33d-29cc-4f79-b53a-d1a674b604e8-kube-api-access-wx7jm\") pod \"network-check-target-qbx8c\" (UID: \"008da33d-29cc-4f79-b53a-d1a674b604e8\") " pod="openshift-network-diagnostics/network-check-target-qbx8c"
Sep 10 15:27:08.304510 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:08.304471    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87bd161e-974d-49ee-861d-22763253529c-metrics-certs\") pod \"network-metrics-daemon-w24m7\" (UID: \"87bd161e-974d-49ee-861d-22763253529c\") " pod="openshift-multus/network-metrics-daemon-w24m7"
Sep 10 15:27:08.304588 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:08.304525    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 15:27:08.304588 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:08.304570    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 15:27:08.304668 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:08.304612    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 15:27:08.304668 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:08.304632    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 15:27:08.304668 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:08.304646    2559 projected.go:196] Error preparing data for projected volume kube-api-access-wx7jm for pod openshift-network-diagnostics/network-check-target-qbx8c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 15:27:08.304802 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:08.304620    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96b96370-410d-447c-8fb0-63f56ab52310-original-pull-secret podName:96b96370-410d-447c-8fb0-63f56ab52310 nodeName:}" failed. No retries permitted until 2026-09-10 15:27:10.304580259 +0000 UTC m=+6.046545362 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/96b96370-410d-447c-8fb0-63f56ab52310-original-pull-secret") pod "global-pull-secret-syncer-7kktn" (UID: "96b96370-410d-447c-8fb0-63f56ab52310") : object "kube-system"/"original-pull-secret" not registered
Sep 10 15:27:08.304802 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:08.304690    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87bd161e-974d-49ee-861d-22763253529c-metrics-certs podName:87bd161e-974d-49ee-861d-22763253529c nodeName:}" failed. No retries permitted until 2026-09-10 15:27:10.304678383 +0000 UTC m=+6.046643468 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/87bd161e-974d-49ee-861d-22763253529c-metrics-certs") pod "network-metrics-daemon-w24m7" (UID: "87bd161e-974d-49ee-861d-22763253529c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 15:27:08.304802 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:08.304704    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/008da33d-29cc-4f79-b53a-d1a674b604e8-kube-api-access-wx7jm podName:008da33d-29cc-4f79-b53a-d1a674b604e8 nodeName:}" failed. No retries permitted until 2026-09-10 15:27:10.304696635 +0000 UTC m=+6.046661718 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-wx7jm" (UniqueName: "kubernetes.io/projected/008da33d-29cc-4f79-b53a-d1a674b604e8-kube-api-access-wx7jm") pod "network-check-target-qbx8c" (UID: "008da33d-29cc-4f79-b53a-d1a674b604e8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 15:27:08.728763 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:08.728687    2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 15:22:07 +0000 UTC" deadline="2028-04-14 05:59:18.676695878 +0000 UTC"
Sep 10 15:27:08.728763 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:08.728728    2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13958h32m9.947972572s"
Sep 10 15:27:08.780374 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:08.779784    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w24m7"
Sep 10 15:27:08.780374 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:08.779935    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w24m7" podUID="87bd161e-974d-49ee-861d-22763253529c"
Sep 10 15:27:08.780969 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:08.780772    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7kktn"
Sep 10 15:27:08.780969 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:08.780887    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7kktn" podUID="96b96370-410d-447c-8fb0-63f56ab52310"
Sep 10 15:27:08.804996 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:08.804963    2559 generic.go:356] "Generic (PLEG): container finished" podID="bf09fa72a5150c7d471ca089e48ca49a" containerID="1524ad4215c389a6d4b86271c1fc9009d9c59c2b28a7cb3566118f28603b5083" exitCode=0
Sep 10 15:27:08.805123 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:08.805055    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-30.ec2.internal" event={"ID":"bf09fa72a5150c7d471ca089e48ca49a","Type":"ContainerDied","Data":"1524ad4215c389a6d4b86271c1fc9009d9c59c2b28a7cb3566118f28603b5083"}
Sep 10 15:27:08.812050 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:08.812021    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-30.ec2.internal" event={"ID":"5aa341fc62b3c7ff684a265510f7e3fe","Type":"ContainerStarted","Data":"cf9c3f27f735453d3410be416b8f2a863a4944153303a68dcd11525d56a4a8ca"}
Sep 10 15:27:08.821754 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:08.821676    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vdtg4" event={"ID":"1a08685b-d954-46be-8456-410c35f09c93","Type":"ContainerStarted","Data":"527243b6b9ffe37c3428b6d2bf283f5e5ce0019fbb371950f7498d699290cc81"}
Sep 10 15:27:08.835923 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:08.835891    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6nx2l" event={"ID":"dd421353-8e9c-4660-8224-cb11e8575e7a","Type":"ContainerStarted","Data":"1bb1f252c0b94b3d725f5bc2930f7e7297757ea4acf60c2714d5944f0ec09bb6"}
Sep 10 15:27:08.839224 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:08.839195    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kcrzq" event={"ID":"0cf24a6f-168a-4076-a08e-6b30b6f34234","Type":"ContainerStarted","Data":"722aca8ff4daa0df781948603b9794146bd1e4ad0e3acbc75fd8f7b850fc6de9"}
Sep 10 15:27:08.853167 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:08.853131    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zxgvr" event={"ID":"153abd57-b263-4386-9087-f9138c09e4a0","Type":"ContainerStarted","Data":"4dcfb66b24f734963e0696eaf9ac02fb5fa671fb1438b06317a18c0c8096daf5"}
Sep 10 15:27:08.863322 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:08.863290    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sdjq9" event={"ID":"96f18013-00ea-4663-87b5-4bca49f4295a","Type":"ContainerStarted","Data":"d428a25731a9c2b5b54b312cbe9481e40b2a1daf68f0fb738b8f640fe87e4c79"}
Sep 10 15:27:08.872251 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:08.872217    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-pzn2d" event={"ID":"6db07abf-0c03-4565-a661-9df53fbe5957","Type":"ContainerStarted","Data":"aef3d3775d24fb6a4db812200cb922a95a68e0d5b1f4d0e082b6b6813f8df424"}
Sep 10 15:27:08.886409 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:08.886377    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-fkmfw" event={"ID":"3f10d0b5-445b-4531-a2ff-ea0264c02ef5","Type":"ContainerStarted","Data":"8a846d67ceba82f945944141a10d7b74421be3d032c9c3ffcd517cc2df9431b1"}
Sep 10 15:27:08.898407 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:08.896411    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-md7dv" event={"ID":"c56c45fc-a485-48b6-a88a-355ba442927f","Type":"ContainerStarted","Data":"5a055137c9c69cc6f25681bef5f96957763a29bc501d4b01135ab1fe1b0011cf"}
Sep 10 15:27:08.918007 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:08.917973    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wgbw2" event={"ID":"7a9f7d3b-8a21-40ed-ab47-067f97ed537a","Type":"ContainerStarted","Data":"f57e0d62a0995855b57dd549fb26e2ad152fb47b72fafe33a81f4592a4b567fb"}
Sep 10 15:27:09.779299 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:09.779015    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qbx8c"
Sep 10 15:27:09.779835 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:09.779426    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qbx8c" podUID="008da33d-29cc-4f79-b53a-d1a674b604e8"
Sep 10 15:27:09.943117 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:09.943078    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-30.ec2.internal" event={"ID":"bf09fa72a5150c7d471ca089e48ca49a","Type":"ContainerStarted","Data":"d51874119325c9a82ea73925e6cab585df8f9a6333de14fa4190bc3dc97fa996"}
Sep 10 15:27:09.964111 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:09.964046    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-30.ec2.internal" podStartSLOduration=3.964007241 podStartE2EDuration="3.964007241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:27:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:27:08.849060306 +0000 UTC m=+4.591025414" watchObservedRunningTime="2026-09-10 15:27:09.964007241 +0000 UTC m=+5.705972350"
Sep 10 15:27:09.964900 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:09.964858    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-30.ec2.internal" podStartSLOduration=3.964847687 podStartE2EDuration="3.964847687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:27:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:27:09.96419726 +0000 UTC m=+5.706162356" watchObservedRunningTime="2026-09-10 15:27:09.964847687 +0000 UTC m=+5.706812798"
Sep 10 15:27:10.324552 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:10.323367    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/96b96370-410d-447c-8fb0-63f56ab52310-original-pull-secret\") pod \"global-pull-secret-syncer-7kktn\" (UID: \"96b96370-410d-447c-8fb0-63f56ab52310\") " pod="kube-system/global-pull-secret-syncer-7kktn"
Sep 10 15:27:10.324552 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:10.323444    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wx7jm\" (UniqueName: \"kubernetes.io/projected/008da33d-29cc-4f79-b53a-d1a674b604e8-kube-api-access-wx7jm\") pod \"network-check-target-qbx8c\" (UID: \"008da33d-29cc-4f79-b53a-d1a674b604e8\") " pod="openshift-network-diagnostics/network-check-target-qbx8c"
Sep 10 15:27:10.324552 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:10.323477    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87bd161e-974d-49ee-861d-22763253529c-metrics-certs\") pod \"network-metrics-daemon-w24m7\" (UID: \"87bd161e-974d-49ee-861d-22763253529c\") " pod="openshift-multus/network-metrics-daemon-w24m7"
Sep 10 15:27:10.324552 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:10.323662    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 15:27:10.324552 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:10.323740    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87bd161e-974d-49ee-861d-22763253529c-metrics-certs podName:87bd161e-974d-49ee-861d-22763253529c nodeName:}" failed. No retries permitted until 2026-09-10 15:27:14.323716631 +0000 UTC m=+10.065681742 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/87bd161e-974d-49ee-861d-22763253529c-metrics-certs") pod "network-metrics-daemon-w24m7" (UID: "87bd161e-974d-49ee-861d-22763253529c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 15:27:10.324552 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:10.324160    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 15:27:10.324552 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:10.324213    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96b96370-410d-447c-8fb0-63f56ab52310-original-pull-secret podName:96b96370-410d-447c-8fb0-63f56ab52310 nodeName:}" failed. No retries permitted until 2026-09-10 15:27:14.324195747 +0000 UTC m=+10.066160830 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/96b96370-410d-447c-8fb0-63f56ab52310-original-pull-secret") pod "global-pull-secret-syncer-7kktn" (UID: "96b96370-410d-447c-8fb0-63f56ab52310") : object "kube-system"/"original-pull-secret" not registered
Sep 10 15:27:10.324552 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:10.324301    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 15:27:10.324552 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:10.324317    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 15:27:10.324552 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:10.324331    2559 projected.go:196] Error preparing data for projected volume kube-api-access-wx7jm for pod openshift-network-diagnostics/network-check-target-qbx8c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 15:27:10.324552 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:10.324372    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/008da33d-29cc-4f79-b53a-d1a674b604e8-kube-api-access-wx7jm podName:008da33d-29cc-4f79-b53a-d1a674b604e8 nodeName:}" failed. No retries permitted until 2026-09-10 15:27:14.324358676 +0000 UTC m=+10.066323765 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-wx7jm" (UniqueName: "kubernetes.io/projected/008da33d-29cc-4f79-b53a-d1a674b604e8-kube-api-access-wx7jm") pod "network-check-target-qbx8c" (UID: "008da33d-29cc-4f79-b53a-d1a674b604e8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 15:27:10.780973 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:10.780935    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7kktn"
Sep 10 15:27:10.781487 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:10.781082    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7kktn" podUID="96b96370-410d-447c-8fb0-63f56ab52310"
Sep 10 15:27:10.781487 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:10.781167    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w24m7"
Sep 10 15:27:10.781487 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:10.781266    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w24m7" podUID="87bd161e-974d-49ee-861d-22763253529c"
Sep 10 15:27:11.778960 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:11.778918    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qbx8c"
Sep 10 15:27:11.779132 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:11.779103    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qbx8c" podUID="008da33d-29cc-4f79-b53a-d1a674b604e8"
Sep 10 15:27:12.778716 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:12.778678    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7kktn"
Sep 10 15:27:12.779141 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:12.778827    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7kktn" podUID="96b96370-410d-447c-8fb0-63f56ab52310"
Sep 10 15:27:12.779141 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:12.778865    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w24m7"
Sep 10 15:27:12.779141 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:12.778994    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w24m7" podUID="87bd161e-974d-49ee-861d-22763253529c"
Sep 10 15:27:13.779375 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:13.779341    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qbx8c"
Sep 10 15:27:13.779810 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:13.779481    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qbx8c" podUID="008da33d-29cc-4f79-b53a-d1a674b604e8"
Sep 10 15:27:14.356940 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:14.356324    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wx7jm\" (UniqueName: \"kubernetes.io/projected/008da33d-29cc-4f79-b53a-d1a674b604e8-kube-api-access-wx7jm\") pod \"network-check-target-qbx8c\" (UID: \"008da33d-29cc-4f79-b53a-d1a674b604e8\") " pod="openshift-network-diagnostics/network-check-target-qbx8c"
Sep 10 15:27:14.356940 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:14.356376    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87bd161e-974d-49ee-861d-22763253529c-metrics-certs\") pod \"network-metrics-daemon-w24m7\" (UID: \"87bd161e-974d-49ee-861d-22763253529c\") " pod="openshift-multus/network-metrics-daemon-w24m7"
Sep 10 15:27:14.356940 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:14.356437    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/96b96370-410d-447c-8fb0-63f56ab52310-original-pull-secret\") pod \"global-pull-secret-syncer-7kktn\" (UID: \"96b96370-410d-447c-8fb0-63f56ab52310\") " pod="kube-system/global-pull-secret-syncer-7kktn"
Sep 10 15:27:14.356940 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:14.356529    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 15:27:14.356940 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:14.356550    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 15:27:14.356940 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:14.356570    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 15:27:14.356940 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:14.356554    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 15:27:14.356940 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:14.356643    2559 projected.go:196] Error preparing data for projected volume kube-api-access-wx7jm for pod openshift-network-diagnostics/network-check-target-qbx8c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 15:27:14.356940 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:14.356632    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96b96370-410d-447c-8fb0-63f56ab52310-original-pull-secret podName:96b96370-410d-447c-8fb0-63f56ab52310 nodeName:}" failed. No retries permitted until 2026-09-10 15:27:22.356611676 +0000 UTC m=+18.098576836 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/96b96370-410d-447c-8fb0-63f56ab52310-original-pull-secret") pod "global-pull-secret-syncer-7kktn" (UID: "96b96370-410d-447c-8fb0-63f56ab52310") : object "kube-system"/"original-pull-secret" not registered
Sep 10 15:27:14.356940 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:14.356690    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87bd161e-974d-49ee-861d-22763253529c-metrics-certs podName:87bd161e-974d-49ee-861d-22763253529c nodeName:}" failed. No retries permitted until 2026-09-10 15:27:22.356673676 +0000 UTC m=+18.098638757 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/87bd161e-974d-49ee-861d-22763253529c-metrics-certs") pod "network-metrics-daemon-w24m7" (UID: "87bd161e-974d-49ee-861d-22763253529c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 15:27:14.356940 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:14.356704    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/008da33d-29cc-4f79-b53a-d1a674b604e8-kube-api-access-wx7jm podName:008da33d-29cc-4f79-b53a-d1a674b604e8 nodeName:}" failed. No retries permitted until 2026-09-10 15:27:22.356695707 +0000 UTC m=+18.098660789 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-wx7jm" (UniqueName: "kubernetes.io/projected/008da33d-29cc-4f79-b53a-d1a674b604e8-kube-api-access-wx7jm") pod "network-check-target-qbx8c" (UID: "008da33d-29cc-4f79-b53a-d1a674b604e8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 15:27:14.780493 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:14.779877    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7kktn"
Sep 10 15:27:14.780493 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:14.779984    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7kktn" podUID="96b96370-410d-447c-8fb0-63f56ab52310"
Sep 10 15:27:14.780493 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:14.780351    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w24m7"
Sep 10 15:27:14.780493 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:14.780450    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w24m7" podUID="87bd161e-974d-49ee-861d-22763253529c"
Sep 10 15:27:15.778938 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:15.778441    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qbx8c"
Sep 10 15:27:15.778938 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:15.778598    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qbx8c" podUID="008da33d-29cc-4f79-b53a-d1a674b604e8"
Sep 10 15:27:16.779216 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:16.779175    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7kktn"
Sep 10 15:27:16.779685 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:16.779312    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7kktn" podUID="96b96370-410d-447c-8fb0-63f56ab52310"
Sep 10 15:27:16.780049 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:16.779768    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w24m7"
Sep 10 15:27:16.780049 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:16.780004    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w24m7" podUID="87bd161e-974d-49ee-861d-22763253529c"
Sep 10 15:27:17.778703 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:17.778669    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qbx8c"
Sep 10 15:27:17.778901 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:17.778786    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qbx8c" podUID="008da33d-29cc-4f79-b53a-d1a674b604e8"
Sep 10 15:27:18.779134 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:18.779101    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7kktn"
Sep 10 15:27:18.779613 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:18.779243    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7kktn" podUID="96b96370-410d-447c-8fb0-63f56ab52310"
Sep 10 15:27:18.779759 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:18.779741    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w24m7"
Sep 10 15:27:18.779872 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:18.779850    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w24m7" podUID="87bd161e-974d-49ee-861d-22763253529c"
Sep 10 15:27:19.778656 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:19.778627    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qbx8c"
Sep 10 15:27:19.778838 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:19.778738    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qbx8c" podUID="008da33d-29cc-4f79-b53a-d1a674b604e8"
Sep 10 15:27:20.779125 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:20.779094    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w24m7"
Sep 10 15:27:20.779536 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:20.779243    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w24m7" podUID="87bd161e-974d-49ee-861d-22763253529c"
Sep 10 15:27:20.779536 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:20.779302    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7kktn"
Sep 10 15:27:20.779536 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:20.779396    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7kktn" podUID="96b96370-410d-447c-8fb0-63f56ab52310"
Sep 10 15:27:21.778635 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:21.778597    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qbx8c"
Sep 10 15:27:21.778879 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:21.778716    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qbx8c" podUID="008da33d-29cc-4f79-b53a-d1a674b604e8"
Sep 10 15:27:22.419057 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:22.419022    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wx7jm\" (UniqueName: \"kubernetes.io/projected/008da33d-29cc-4f79-b53a-d1a674b604e8-kube-api-access-wx7jm\") pod \"network-check-target-qbx8c\" (UID: \"008da33d-29cc-4f79-b53a-d1a674b604e8\") " pod="openshift-network-diagnostics/network-check-target-qbx8c"
Sep 10 15:27:22.419057 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:22.419064    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87bd161e-974d-49ee-861d-22763253529c-metrics-certs\") pod \"network-metrics-daemon-w24m7\" (UID: \"87bd161e-974d-49ee-861d-22763253529c\") " pod="openshift-multus/network-metrics-daemon-w24m7"
Sep 10 15:27:22.419669 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:22.419177    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 15:27:22.419669 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:22.419205    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 15:27:22.419669 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:22.419208    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/96b96370-410d-447c-8fb0-63f56ab52310-original-pull-secret\") pod \"global-pull-secret-syncer-7kktn\" (UID: \"96b96370-410d-447c-8fb0-63f56ab52310\") " pod="kube-system/global-pull-secret-syncer-7kktn"
Sep 10 15:27:22.419669 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:22.419224    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87bd161e-974d-49ee-861d-22763253529c-metrics-certs podName:87bd161e-974d-49ee-861d-22763253529c nodeName:}" failed. No retries permitted until 2026-09-10 15:27:38.419209185 +0000 UTC m=+34.161174270 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/87bd161e-974d-49ee-861d-22763253529c-metrics-certs") pod "network-metrics-daemon-w24m7" (UID: "87bd161e-974d-49ee-861d-22763253529c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 15:27:22.419669 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:22.419231    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 15:27:22.419669 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:22.419248    2559 projected.go:196] Error preparing data for projected volume kube-api-access-wx7jm for pod openshift-network-diagnostics/network-check-target-qbx8c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 15:27:22.419669 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:22.419298    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/008da33d-29cc-4f79-b53a-d1a674b604e8-kube-api-access-wx7jm podName:008da33d-29cc-4f79-b53a-d1a674b604e8 nodeName:}" failed. No retries permitted until 2026-09-10 15:27:38.419282755 +0000 UTC m=+34.161247858 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-wx7jm" (UniqueName: "kubernetes.io/projected/008da33d-29cc-4f79-b53a-d1a674b604e8-kube-api-access-wx7jm") pod "network-check-target-qbx8c" (UID: "008da33d-29cc-4f79-b53a-d1a674b604e8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 15:27:22.419669 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:22.419317    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 15:27:22.419669 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:22.419353    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96b96370-410d-447c-8fb0-63f56ab52310-original-pull-secret podName:96b96370-410d-447c-8fb0-63f56ab52310 nodeName:}" failed. No retries permitted until 2026-09-10 15:27:38.419343057 +0000 UTC m=+34.161308138 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/96b96370-410d-447c-8fb0-63f56ab52310-original-pull-secret") pod "global-pull-secret-syncer-7kktn" (UID: "96b96370-410d-447c-8fb0-63f56ab52310") : object "kube-system"/"original-pull-secret" not registered
Sep 10 15:27:22.778771 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:22.778674    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7kktn"
Sep 10 15:27:22.778944 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:22.778674    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w24m7"
Sep 10 15:27:22.778944 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:22.778806    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7kktn" podUID="96b96370-410d-447c-8fb0-63f56ab52310"
Sep 10 15:27:22.778944 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:22.778853    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w24m7" podUID="87bd161e-974d-49ee-861d-22763253529c"
Sep 10 15:27:23.778355 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:23.778324    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qbx8c"
Sep 10 15:27:23.778851 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:23.778459    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qbx8c" podUID="008da33d-29cc-4f79-b53a-d1a674b604e8"
Sep 10 15:27:24.779813 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:24.779782    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w24m7"
Sep 10 15:27:24.780254 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:24.779904    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w24m7" podUID="87bd161e-974d-49ee-861d-22763253529c"
Sep 10 15:27:24.780254 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:24.779948    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7kktn"
Sep 10 15:27:24.780254 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:24.780076    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7kktn" podUID="96b96370-410d-447c-8fb0-63f56ab52310"
Sep 10 15:27:25.778632 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:25.778603    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qbx8c"
Sep 10 15:27:25.778767 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:25.778694    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qbx8c" podUID="008da33d-29cc-4f79-b53a-d1a674b604e8"
Sep 10 15:27:26.779475 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:26.779215    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w24m7"
Sep 10 15:27:26.780372 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:26.779733    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7kktn"
Sep 10 15:27:26.780372 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:26.779733    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w24m7" podUID="87bd161e-974d-49ee-861d-22763253529c"
Sep 10 15:27:26.780372 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:26.779888    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7kktn" podUID="96b96370-410d-447c-8fb0-63f56ab52310"
Sep 10 15:27:26.979728 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:26.979686    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vdtg4" event={"ID":"1a08685b-d954-46be-8456-410c35f09c93","Type":"ContainerStarted","Data":"042072dee90cec1b9a02afcd1cb7371c1e51b4f996ae0ebc431ca4da4b93f4b4"}
Sep 10 15:27:26.981868 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:26.981841    2559 generic.go:356] "Generic (PLEG): container finished" podID="dd421353-8e9c-4660-8224-cb11e8575e7a" containerID="62fc84e0a892aa32c523e10138e2e3d47bac56066703591ccac020f4a2d94bc0" exitCode=0
Sep 10 15:27:26.982017 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:26.981913    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6nx2l" event={"ID":"dd421353-8e9c-4660-8224-cb11e8575e7a","Type":"ContainerDied","Data":"62fc84e0a892aa32c523e10138e2e3d47bac56066703591ccac020f4a2d94bc0"}
Sep 10 15:27:26.984009 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:26.983738    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kcrzq" event={"ID":"0cf24a6f-168a-4076-a08e-6b30b6f34234","Type":"ContainerStarted","Data":"0653e19b0193b5235ba3a0be42bb486c738c7ce34d9a3d0e319cbc8d3abaa152"}
Sep 10 15:27:26.985696 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:26.985666    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zxgvr" event={"ID":"153abd57-b263-4386-9087-f9138c09e4a0","Type":"ContainerStarted","Data":"cb6017165e4ee29d4ad806c707d1a5533270a932ba0a481b8c4d7c740f1910c7"}
Sep 10 15:27:26.990398 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:26.990362    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sdjq9" event={"ID":"96f18013-00ea-4663-87b5-4bca49f4295a","Type":"ContainerStarted","Data":"3495c3dc96e10c6887e56f276031a94dc93e59d63a779a5a1b4ca186a4f80848"}
Sep 10 15:27:26.990513 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:26.990448    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sdjq9" event={"ID":"96f18013-00ea-4663-87b5-4bca49f4295a","Type":"ContainerStarted","Data":"7af3e3c3077bbfcda7c9ef96c92f147cbad1f466ac0e0a110494b724ff347113"}
Sep 10 15:27:26.990513 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:26.990503    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sdjq9" event={"ID":"96f18013-00ea-4663-87b5-4bca49f4295a","Type":"ContainerStarted","Data":"7ecd8b6edddededfab644437b813ccf8401245e1006ced248ff600b58ee2750d"}
Sep 10 15:27:26.990646 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:26.990553    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sdjq9" event={"ID":"96f18013-00ea-4663-87b5-4bca49f4295a","Type":"ContainerStarted","Data":"f7625dc1528b70ab7ed67cefbc29dcaa9a1d8c6912476fed84ad7132f7f35c3b"}
Sep 10 15:27:26.990646 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:26.990622    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sdjq9" event={"ID":"96f18013-00ea-4663-87b5-4bca49f4295a","Type":"ContainerStarted","Data":"2b4849acb03d4ba27fd63c6a0d954e40e872fa3a33f3dc1f2dc964e91f15ef51"}
Sep 10 15:27:26.990763 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:26.990670    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sdjq9" event={"ID":"96f18013-00ea-4663-87b5-4bca49f4295a","Type":"ContainerStarted","Data":"87bb1fedc7da9efd464fccdfa8c29fd52b8f6838a9e35125b4f4dc13d40f9612"}
Sep 10 15:27:26.992330 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:26.992302    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-pzn2d" event={"ID":"6db07abf-0c03-4565-a661-9df53fbe5957","Type":"ContainerStarted","Data":"5ef6b6a79d8e69d917acb394eef775ac56ad4756ddf0148ef3cc499fcf93c263"}
Sep 10 15:27:26.994050 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:26.994030    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-md7dv" event={"ID":"c56c45fc-a485-48b6-a88a-355ba442927f","Type":"ContainerStarted","Data":"4ea685b8d80be1191e1c3b1254f7d87833edad8c7c7dcaf67dda5df1b761dc69"}
Sep 10 15:27:26.995732 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:26.995712    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wgbw2" event={"ID":"7a9f7d3b-8a21-40ed-ab47-067f97ed537a","Type":"ContainerStarted","Data":"18fa32cb70e0bc4db78858ea0ddce8994d7901dc61c61e1ca7d335e9face2302"}
Sep 10 15:27:27.004993 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:27.004958    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-vdtg4" podStartSLOduration=3.939463864 podStartE2EDuration="22.004946198s" totalImagesPullingTime="18.065482334s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:27:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:27:27.004379044 +0000 UTC m=+22.746344152" watchObservedRunningTime="2026-09-10 15:27:27.004946198 +0000 UTC m=+22.746911301"
Sep 10 15:27:27.023062 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:27.023027    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-zxgvr" podStartSLOduration=4.078848418 podStartE2EDuration="22.023015267s" totalImagesPullingTime="17.944166849s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:27:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:27:27.022748964 +0000 UTC m=+22.764714077" watchObservedRunningTime="2026-09-10 15:27:27.023015267 +0000 UTC m=+22.764980369"
Sep 10 15:27:27.037655 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:27.037568    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-pzn2d" podStartSLOduration=4.11675387 podStartE2EDuration="22.037559419s" totalImagesPullingTime="17.920805549s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:27:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:27:27.037215122 +0000 UTC m=+22.779180224" watchObservedRunningTime="2026-09-10 15:27:27.037559419 +0000 UTC m=+22.779524522"
Sep 10 15:27:27.054433 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:27.054382    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-md7dv" podStartSLOduration=4.136342749 podStartE2EDuration="22.054370582s" totalImagesPullingTime="17.918027833s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:27:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:27:27.054175297 +0000 UTC m=+22.796140400" watchObservedRunningTime="2026-09-10 15:27:27.054370582 +0000 UTC m=+22.796335685"
Sep 10 15:27:27.095640 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:27.095598    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-wgbw2" podStartSLOduration=11.488630434 podStartE2EDuration="22.095571813s" totalImagesPullingTime="10.606941379s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:27:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:27:27.071652626 +0000 UTC m=+22.813617728" watchObservedRunningTime="2026-09-10 15:27:27.095571813 +0000 UTC m=+22.837536915"
Sep 10 15:27:27.480911 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:27.480878    2559 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 10 15:27:27.747086 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:27.746975    2559 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-10T15:27:27.480908554Z","UUID":"673291d2-6322-4b7c-b211-e5df5650a07e","Handler":null,"Name":"","Endpoint":""}
Sep 10 15:27:27.748742 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:27.748718    2559 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 10 15:27:27.748742 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:27.748756    2559 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 10 15:27:27.778291 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:27.778265    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qbx8c"
Sep 10 15:27:27.778434 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:27.778353    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qbx8c" podUID="008da33d-29cc-4f79-b53a-d1a674b604e8"
Sep 10 15:27:28.000773 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:28.000680    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-fkmfw" event={"ID":"3f10d0b5-445b-4531-a2ff-ea0264c02ef5","Type":"ContainerStarted","Data":"9757410b2f662063a5029daa56d89ac691ee2d8a9134f63e873753f166e25633"}
Sep 10 15:27:28.003388 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:28.003361    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kcrzq" event={"ID":"0cf24a6f-168a-4076-a08e-6b30b6f34234","Type":"ContainerStarted","Data":"6f37ff4b9a8049691cc314588af146304d34a0c5556046251daa59bfc7d09fc3"}
Sep 10 15:27:28.778878 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:28.778806    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7kktn"
Sep 10 15:27:28.779032 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:28.778942    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7kktn" podUID="96b96370-410d-447c-8fb0-63f56ab52310"
Sep 10 15:27:28.779032 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:28.778956    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w24m7"
Sep 10 15:27:28.779148 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:28.779080    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w24m7" podUID="87bd161e-974d-49ee-861d-22763253529c"
Sep 10 15:27:28.844193 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:28.844155    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-wgbw2"
Sep 10 15:27:29.009696 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:29.009658    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sdjq9" event={"ID":"96f18013-00ea-4663-87b5-4bca49f4295a","Type":"ContainerStarted","Data":"c5f1d102e0afce00b7455e13813ac6f5585ee7d917f1199da1b18bf525f05488"}
Sep 10 15:27:29.011913 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:29.011886    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kcrzq" event={"ID":"0cf24a6f-168a-4076-a08e-6b30b6f34234","Type":"ContainerStarted","Data":"f678a55a0c6431018fc45d27a393b5b8a9db6ab60553edd795af8d2303c6bc19"}
Sep 10 15:27:29.035479 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:29.035405    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-fkmfw" podStartSLOduration=6.321919714 podStartE2EDuration="24.035392064s" totalImagesPullingTime="17.71347235s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:27:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:27:28.018345557 +0000 UTC m=+23.760310659" watchObservedRunningTime="2026-09-10 15:27:29.035392064 +0000 UTC m=+24.777357167"
Sep 10 15:27:29.035885 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:29.035862    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kcrzq" podStartSLOduration=4.08884736 podStartE2EDuration="24.035855279s" totalImagesPullingTime="19.947007919s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:27:05 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:27:29.035243176 +0000 UTC m=+24.777208280" watchObservedRunningTime="2026-09-10 15:27:29.035855279 +0000 UTC m=+24.777820382"
Sep 10 15:27:29.544391 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:29.544360    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-wgbw2"
Sep 10 15:27:29.544978 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:29.544958    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-wgbw2"
Sep 10 15:27:29.778843 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:29.778815    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qbx8c"
Sep 10 15:27:29.779016 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:29.778933    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qbx8c" podUID="008da33d-29cc-4f79-b53a-d1a674b604e8"
Sep 10 15:27:30.014746 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:30.014716    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-wgbw2"
Sep 10 15:27:30.778914 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:30.778881    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7kktn"
Sep 10 15:27:30.779081 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:30.778880    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w24m7"
Sep 10 15:27:30.779081 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:30.779023    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7kktn" podUID="96b96370-410d-447c-8fb0-63f56ab52310"
Sep 10 15:27:30.779179 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:30.779112    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w24m7" podUID="87bd161e-974d-49ee-861d-22763253529c"
Sep 10 15:27:31.778969 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:31.778932    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qbx8c"
Sep 10 15:27:31.779361 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:31.779043    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qbx8c" podUID="008da33d-29cc-4f79-b53a-d1a674b604e8"
Sep 10 15:27:32.778769 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:32.778600    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w24m7"
Sep 10 15:27:32.778916 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:32.778651    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7kktn"
Sep 10 15:27:32.778916 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:32.778855    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w24m7" podUID="87bd161e-974d-49ee-861d-22763253529c"
Sep 10 15:27:32.778916 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:32.778910    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7kktn" podUID="96b96370-410d-447c-8fb0-63f56ab52310"
Sep 10 15:27:33.021205 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:33.021172    2559 generic.go:356] "Generic (PLEG): container finished" podID="dd421353-8e9c-4660-8224-cb11e8575e7a" containerID="5030ed1b9091d0c67e9b770f5b1d8ec7b7336a9d7f7c5dab7e1670ecfe9015a4" exitCode=0
Sep 10 15:27:33.021997 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:33.021269    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6nx2l" event={"ID":"dd421353-8e9c-4660-8224-cb11e8575e7a","Type":"ContainerDied","Data":"5030ed1b9091d0c67e9b770f5b1d8ec7b7336a9d7f7c5dab7e1670ecfe9015a4"}
Sep 10 15:27:33.025467 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:33.025446    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sdjq9" event={"ID":"96f18013-00ea-4663-87b5-4bca49f4295a","Type":"ContainerStarted","Data":"ed3e151035f6e2f71ece3ec59586d53e2028ca8d2cd8a216a51fd69a080b20a8"}
Sep 10 15:27:33.025743 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:33.025727    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-sdjq9"
Sep 10 15:27:33.025842 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:33.025774    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-sdjq9"
Sep 10 15:27:33.040908 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:33.040850    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-sdjq9"
Sep 10 15:27:33.084385 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:33.084336    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-sdjq9" podStartSLOduration=10.111570098 podStartE2EDuration="28.084320206s" totalImagesPullingTime="17.972750108s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:27:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:27:33.084062187 +0000 UTC m=+28.826027300" watchObservedRunningTime="2026-09-10 15:27:33.084320206 +0000 UTC m=+28.826285310"
Sep 10 15:27:33.778909 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:33.778762    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qbx8c"
Sep 10 15:27:33.779006 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:33.778989    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qbx8c" podUID="008da33d-29cc-4f79-b53a-d1a674b604e8"
Sep 10 15:27:34.006681 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:34.006652    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qbx8c"]
Sep 10 15:27:34.009488 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:34.009462    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-7kktn"]
Sep 10 15:27:34.009611 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:34.009548    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7kktn"
Sep 10 15:27:34.009705 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:34.009677    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7kktn" podUID="96b96370-410d-447c-8fb0-63f56ab52310"
Sep 10 15:27:34.010528 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:34.010499    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-w24m7"]
Sep 10 15:27:34.010643 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:34.010602    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w24m7"
Sep 10 15:27:34.010729 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:34.010708    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w24m7" podUID="87bd161e-974d-49ee-861d-22763253529c"
Sep 10 15:27:34.027618 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:34.027597    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qbx8c"
Sep 10 15:27:34.027974 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:34.027682    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qbx8c" podUID="008da33d-29cc-4f79-b53a-d1a674b604e8"
Sep 10 15:27:34.028157 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:34.028128    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-sdjq9"
Sep 10 15:27:34.046370 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:34.046349    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-sdjq9"
Sep 10 15:27:35.031765 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:35.031729    2559 generic.go:356] "Generic (PLEG): container finished" podID="dd421353-8e9c-4660-8224-cb11e8575e7a" containerID="e5b64846357b28f70be7a1614d8f323ca31c10d04bf3741fa57a2fc46e77b54a" exitCode=0
Sep 10 15:27:35.032193 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:35.031817    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6nx2l" event={"ID":"dd421353-8e9c-4660-8224-cb11e8575e7a","Type":"ContainerDied","Data":"e5b64846357b28f70be7a1614d8f323ca31c10d04bf3741fa57a2fc46e77b54a"}
Sep 10 15:27:35.778690 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:35.778655    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qbx8c"
Sep 10 15:27:35.778690 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:35.778674    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7kktn"
Sep 10 15:27:35.778921 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:35.778674    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w24m7"
Sep 10 15:27:35.778921 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:35.778761    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qbx8c" podUID="008da33d-29cc-4f79-b53a-d1a674b604e8"
Sep 10 15:27:35.778921 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:35.778836    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7kktn" podUID="96b96370-410d-447c-8fb0-63f56ab52310"
Sep 10 15:27:35.778921 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:35.778895    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w24m7" podUID="87bd161e-974d-49ee-861d-22763253529c"
Sep 10 15:27:36.036331 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:36.036269    2559 generic.go:356] "Generic (PLEG): container finished" podID="dd421353-8e9c-4660-8224-cb11e8575e7a" containerID="2e90f29acaec505e32aa3bd5b91b79fa6f41745dc28269e28a97e2314cd080ba" exitCode=0
Sep 10 15:27:36.036666 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:36.036353    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6nx2l" event={"ID":"dd421353-8e9c-4660-8224-cb11e8575e7a","Type":"ContainerDied","Data":"2e90f29acaec505e32aa3bd5b91b79fa6f41745dc28269e28a97e2314cd080ba"}
Sep 10 15:27:37.779587 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:37.779334    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7kktn"
Sep 10 15:27:37.780116 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:37.779344    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qbx8c"
Sep 10 15:27:37.780116 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:37.779690    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7kktn" podUID="96b96370-410d-447c-8fb0-63f56ab52310"
Sep 10 15:27:37.780116 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:37.779334    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w24m7"
Sep 10 15:27:37.780116 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:37.779762    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qbx8c" podUID="008da33d-29cc-4f79-b53a-d1a674b604e8"
Sep 10 15:27:37.780116 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:37.779820    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w24m7" podUID="87bd161e-974d-49ee-861d-22763253529c"
Sep 10 15:27:38.435169 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:38.435137    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wx7jm\" (UniqueName: \"kubernetes.io/projected/008da33d-29cc-4f79-b53a-d1a674b604e8-kube-api-access-wx7jm\") pod \"network-check-target-qbx8c\" (UID: \"008da33d-29cc-4f79-b53a-d1a674b604e8\") " pod="openshift-network-diagnostics/network-check-target-qbx8c"
Sep 10 15:27:38.435366 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:38.435185    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87bd161e-974d-49ee-861d-22763253529c-metrics-certs\") pod \"network-metrics-daemon-w24m7\" (UID: \"87bd161e-974d-49ee-861d-22763253529c\") " pod="openshift-multus/network-metrics-daemon-w24m7"
Sep 10 15:27:38.435366 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:38.435222    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/96b96370-410d-447c-8fb0-63f56ab52310-original-pull-secret\") pod \"global-pull-secret-syncer-7kktn\" (UID: \"96b96370-410d-447c-8fb0-63f56ab52310\") " pod="kube-system/global-pull-secret-syncer-7kktn"
Sep 10 15:27:38.435366 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:38.435327    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 15:27:38.435366 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:38.435335    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 15:27:38.435366 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:38.435360    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 15:27:38.435636 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:38.435392    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96b96370-410d-447c-8fb0-63f56ab52310-original-pull-secret podName:96b96370-410d-447c-8fb0-63f56ab52310 nodeName:}" failed. No retries permitted until 2026-09-10 15:28:10.435373621 +0000 UTC m=+66.177338716 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/96b96370-410d-447c-8fb0-63f56ab52310-original-pull-secret") pod "global-pull-secret-syncer-7kktn" (UID: "96b96370-410d-447c-8fb0-63f56ab52310") : object "kube-system"/"original-pull-secret" not registered
Sep 10 15:27:38.435636 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:38.435425    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87bd161e-974d-49ee-861d-22763253529c-metrics-certs podName:87bd161e-974d-49ee-861d-22763253529c nodeName:}" failed. No retries permitted until 2026-09-10 15:28:10.435406374 +0000 UTC m=+66.177371470 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/87bd161e-974d-49ee-861d-22763253529c-metrics-certs") pod "network-metrics-daemon-w24m7" (UID: "87bd161e-974d-49ee-861d-22763253529c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 15:27:38.435636 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:38.435363    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 15:27:38.435636 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:38.435448    2559 projected.go:196] Error preparing data for projected volume kube-api-access-wx7jm for pod openshift-network-diagnostics/network-check-target-qbx8c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 15:27:38.435636 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:38.435490    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/008da33d-29cc-4f79-b53a-d1a674b604e8-kube-api-access-wx7jm podName:008da33d-29cc-4f79-b53a-d1a674b604e8 nodeName:}" failed. No retries permitted until 2026-09-10 15:28:10.435479678 +0000 UTC m=+66.177444773 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-wx7jm" (UniqueName: "kubernetes.io/projected/008da33d-29cc-4f79-b53a-d1a674b604e8-kube-api-access-wx7jm") pod "network-check-target-qbx8c" (UID: "008da33d-29cc-4f79-b53a-d1a674b604e8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 15:27:39.062662 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:39.062628    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-30.ec2.internal" event="NodeReady"
Sep 10 15:27:39.063087 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:39.062803    2559 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 10 15:27:39.114235 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:39.114163    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-tj846"]
Sep 10 15:27:39.124083 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:39.124061    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-5dnj9"]
Sep 10 15:27:39.124228 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:39.124207    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tj846"
Sep 10 15:27:39.126716 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:39.126687    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 10 15:27:39.126997 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:39.126975    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-w5mt4\""
Sep 10 15:27:39.127112 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:39.126975    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 10 15:27:39.127112 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:39.127024    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5dnj9"
Sep 10 15:27:39.129288 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:39.129266    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-tj846"]
Sep 10 15:27:39.130317 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:39.130121    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-8ztg8\""
Sep 10 15:27:39.130317 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:39.130169    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 10 15:27:39.130317 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:39.130226    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 10 15:27:39.130317 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:39.130123    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 10 15:27:39.136030 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:39.135981    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5dnj9"]
Sep 10 15:27:39.239804 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:39.239765    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/66850670-e07c-4df2-bfe0-fd3b5fcb3308-metrics-tls\") pod \"dns-default-tj846\" (UID: \"66850670-e07c-4df2-bfe0-fd3b5fcb3308\") " pod="openshift-dns/dns-default-tj846"
Sep 10 15:27:39.239965 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:39.239812    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n52fh\" (UniqueName: \"kubernetes.io/projected/280d97b4-f718-4bf7-a173-11822ff796fd-kube-api-access-n52fh\") pod \"ingress-canary-5dnj9\" (UID: \"280d97b4-f718-4bf7-a173-11822ff796fd\") " pod="openshift-ingress-canary/ingress-canary-5dnj9"
Sep 10 15:27:39.239965 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:39.239862    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/280d97b4-f718-4bf7-a173-11822ff796fd-cert\") pod \"ingress-canary-5dnj9\" (UID: \"280d97b4-f718-4bf7-a173-11822ff796fd\") " pod="openshift-ingress-canary/ingress-canary-5dnj9"
Sep 10 15:27:39.239965 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:39.239933    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/66850670-e07c-4df2-bfe0-fd3b5fcb3308-config-volume\") pod \"dns-default-tj846\" (UID: \"66850670-e07c-4df2-bfe0-fd3b5fcb3308\") " pod="openshift-dns/dns-default-tj846"
Sep 10 15:27:39.239965 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:39.239951    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66850670-e07c-4df2-bfe0-fd3b5fcb3308-tmp-dir\") pod \"dns-default-tj846\" (UID: \"66850670-e07c-4df2-bfe0-fd3b5fcb3308\") " pod="openshift-dns/dns-default-tj846"
Sep 10 15:27:39.240119 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:39.239996    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vr4z6\" (UniqueName: \"kubernetes.io/projected/66850670-e07c-4df2-bfe0-fd3b5fcb3308-kube-api-access-vr4z6\") pod \"dns-default-tj846\" (UID: \"66850670-e07c-4df2-bfe0-fd3b5fcb3308\") " pod="openshift-dns/dns-default-tj846"
Sep 10 15:27:39.341119 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:39.341077    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/280d97b4-f718-4bf7-a173-11822ff796fd-cert\") pod \"ingress-canary-5dnj9\" (UID: \"280d97b4-f718-4bf7-a173-11822ff796fd\") " pod="openshift-ingress-canary/ingress-canary-5dnj9"
Sep 10 15:27:39.341295 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:39.341167    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/66850670-e07c-4df2-bfe0-fd3b5fcb3308-config-volume\") pod \"dns-default-tj846\" (UID: \"66850670-e07c-4df2-bfe0-fd3b5fcb3308\") " pod="openshift-dns/dns-default-tj846"
Sep 10 15:27:39.341295 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:39.341188    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66850670-e07c-4df2-bfe0-fd3b5fcb3308-tmp-dir\") pod \"dns-default-tj846\" (UID: \"66850670-e07c-4df2-bfe0-fd3b5fcb3308\") " pod="openshift-dns/dns-default-tj846"
Sep 10 15:27:39.341295 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:39.341226    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vr4z6\" (UniqueName: \"kubernetes.io/projected/66850670-e07c-4df2-bfe0-fd3b5fcb3308-kube-api-access-vr4z6\") pod \"dns-default-tj846\" (UID: \"66850670-e07c-4df2-bfe0-fd3b5fcb3308\") " pod="openshift-dns/dns-default-tj846"
Sep 10 15:27:39.341295 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:39.341261    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/66850670-e07c-4df2-bfe0-fd3b5fcb3308-metrics-tls\") pod \"dns-default-tj846\" (UID: \"66850670-e07c-4df2-bfe0-fd3b5fcb3308\") " pod="openshift-dns/dns-default-tj846"
Sep 10 15:27:39.341295 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:39.341285    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n52fh\" (UniqueName: \"kubernetes.io/projected/280d97b4-f718-4bf7-a173-11822ff796fd-kube-api-access-n52fh\") pod \"ingress-canary-5dnj9\" (UID: \"280d97b4-f718-4bf7-a173-11822ff796fd\") " pod="openshift-ingress-canary/ingress-canary-5dnj9"
Sep 10 15:27:39.341534 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:39.341430    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 15:27:39.341534 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:39.341495    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66850670-e07c-4df2-bfe0-fd3b5fcb3308-metrics-tls podName:66850670-e07c-4df2-bfe0-fd3b5fcb3308 nodeName:}" failed. No retries permitted until 2026-09-10 15:27:39.841474332 +0000 UTC m=+35.583439427 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/66850670-e07c-4df2-bfe0-fd3b5fcb3308-metrics-tls") pod "dns-default-tj846" (UID: "66850670-e07c-4df2-bfe0-fd3b5fcb3308") : secret "dns-default-metrics-tls" not found
Sep 10 15:27:39.341649 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:39.341626    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66850670-e07c-4df2-bfe0-fd3b5fcb3308-tmp-dir\") pod \"dns-default-tj846\" (UID: \"66850670-e07c-4df2-bfe0-fd3b5fcb3308\") " pod="openshift-dns/dns-default-tj846"
Sep 10 15:27:39.341826 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:39.341806    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/66850670-e07c-4df2-bfe0-fd3b5fcb3308-config-volume\") pod \"dns-default-tj846\" (UID: \"66850670-e07c-4df2-bfe0-fd3b5fcb3308\") " pod="openshift-dns/dns-default-tj846"
Sep 10 15:27:39.346064 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:39.346042    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/280d97b4-f718-4bf7-a173-11822ff796fd-cert\") pod \"ingress-canary-5dnj9\" (UID: \"280d97b4-f718-4bf7-a173-11822ff796fd\") " pod="openshift-ingress-canary/ingress-canary-5dnj9"
Sep 10 15:27:39.353640 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:39.353617    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n52fh\" (UniqueName: \"kubernetes.io/projected/280d97b4-f718-4bf7-a173-11822ff796fd-kube-api-access-n52fh\") pod \"ingress-canary-5dnj9\" (UID: \"280d97b4-f718-4bf7-a173-11822ff796fd\") " pod="openshift-ingress-canary/ingress-canary-5dnj9"
Sep 10 15:27:39.354544 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:39.354524    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vr4z6\" (UniqueName: \"kubernetes.io/projected/66850670-e07c-4df2-bfe0-fd3b5fcb3308-kube-api-access-vr4z6\") pod \"dns-default-tj846\" (UID: \"66850670-e07c-4df2-bfe0-fd3b5fcb3308\") " pod="openshift-dns/dns-default-tj846"
Sep 10 15:27:39.443796 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:39.443760    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5dnj9"
Sep 10 15:27:39.778527 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:39.778442    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qbx8c"
Sep 10 15:27:39.778527 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:39.778496    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7kktn"
Sep 10 15:27:39.778757 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:39.778603    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w24m7"
Sep 10 15:27:39.782761 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:39.782744    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 10 15:27:39.782999 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:39.782985    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-5rkds\""
Sep 10 15:27:39.782999 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:39.782993    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 10 15:27:39.783136 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:39.783055    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-h8jcj\""
Sep 10 15:27:39.783136 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:39.782993    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 10 15:27:39.783205 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:39.783174    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 10 15:27:39.846477 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:39.846447    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/66850670-e07c-4df2-bfe0-fd3b5fcb3308-metrics-tls\") pod \"dns-default-tj846\" (UID: \"66850670-e07c-4df2-bfe0-fd3b5fcb3308\") " pod="openshift-dns/dns-default-tj846"
Sep 10 15:27:39.846693 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:39.846620    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 15:27:39.846693 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:39.846686    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66850670-e07c-4df2-bfe0-fd3b5fcb3308-metrics-tls podName:66850670-e07c-4df2-bfe0-fd3b5fcb3308 nodeName:}" failed. No retries permitted until 2026-09-10 15:27:40.846668176 +0000 UTC m=+36.588633265 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/66850670-e07c-4df2-bfe0-fd3b5fcb3308-metrics-tls") pod "dns-default-tj846" (UID: "66850670-e07c-4df2-bfe0-fd3b5fcb3308") : secret "dns-default-metrics-tls" not found
Sep 10 15:27:40.489706 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:40.489395    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5dnj9"]
Sep 10 15:27:40.492500 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:40.492464    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod280d97b4_f718_4bf7_a173_11822ff796fd.slice/crio-afe755829f19dbb9878fc12fed99c2234fe0cfe36932184e50c9c9515a9dc05e WatchSource:0}: Error finding container afe755829f19dbb9878fc12fed99c2234fe0cfe36932184e50c9c9515a9dc05e: Status 404 returned error can't find the container with id afe755829f19dbb9878fc12fed99c2234fe0cfe36932184e50c9c9515a9dc05e
Sep 10 15:27:40.854392 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:40.854297    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/66850670-e07c-4df2-bfe0-fd3b5fcb3308-metrics-tls\") pod \"dns-default-tj846\" (UID: \"66850670-e07c-4df2-bfe0-fd3b5fcb3308\") " pod="openshift-dns/dns-default-tj846"
Sep 10 15:27:40.854541 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:40.854468    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 15:27:40.854541 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:40.854532    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66850670-e07c-4df2-bfe0-fd3b5fcb3308-metrics-tls podName:66850670-e07c-4df2-bfe0-fd3b5fcb3308 nodeName:}" failed. No retries permitted until 2026-09-10 15:27:42.854514176 +0000 UTC m=+38.596479257 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/66850670-e07c-4df2-bfe0-fd3b5fcb3308-metrics-tls") pod "dns-default-tj846" (UID: "66850670-e07c-4df2-bfe0-fd3b5fcb3308") : secret "dns-default-metrics-tls" not found
Sep 10 15:27:41.052357 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:41.052320    2559 generic.go:356] "Generic (PLEG): container finished" podID="dd421353-8e9c-4660-8224-cb11e8575e7a" containerID="5fb742c5cd99e094d5f86a2c445cbb3132fedcda7d29ffc45859836941fa4fde" exitCode=0
Sep 10 15:27:41.052528 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:41.052373    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6nx2l" event={"ID":"dd421353-8e9c-4660-8224-cb11e8575e7a","Type":"ContainerDied","Data":"5fb742c5cd99e094d5f86a2c445cbb3132fedcda7d29ffc45859836941fa4fde"}
Sep 10 15:27:41.053866 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:41.053841    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5dnj9" event={"ID":"280d97b4-f718-4bf7-a173-11822ff796fd","Type":"ContainerStarted","Data":"afe755829f19dbb9878fc12fed99c2234fe0cfe36932184e50c9c9515a9dc05e"}
Sep 10 15:27:42.059340 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:42.059303    2559 generic.go:356] "Generic (PLEG): container finished" podID="dd421353-8e9c-4660-8224-cb11e8575e7a" containerID="9b704ae8063a86103cc24ce6a7130ef6587205ad207bbc12db824e2f3fb72492" exitCode=0
Sep 10 15:27:42.059838 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:42.059367    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6nx2l" event={"ID":"dd421353-8e9c-4660-8224-cb11e8575e7a","Type":"ContainerDied","Data":"9b704ae8063a86103cc24ce6a7130ef6587205ad207bbc12db824e2f3fb72492"}
Sep 10 15:27:42.869667 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:42.869632    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/66850670-e07c-4df2-bfe0-fd3b5fcb3308-metrics-tls\") pod \"dns-default-tj846\" (UID: \"66850670-e07c-4df2-bfe0-fd3b5fcb3308\") " pod="openshift-dns/dns-default-tj846"
Sep 10 15:27:42.869806 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:42.869760    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 15:27:42.869846 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:42.869821    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66850670-e07c-4df2-bfe0-fd3b5fcb3308-metrics-tls podName:66850670-e07c-4df2-bfe0-fd3b5fcb3308 nodeName:}" failed. No retries permitted until 2026-09-10 15:27:46.86980641 +0000 UTC m=+42.611771505 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/66850670-e07c-4df2-bfe0-fd3b5fcb3308-metrics-tls") pod "dns-default-tj846" (UID: "66850670-e07c-4df2-bfe0-fd3b5fcb3308") : secret "dns-default-metrics-tls" not found
Sep 10 15:27:43.064697 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:43.064665    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6nx2l" event={"ID":"dd421353-8e9c-4660-8224-cb11e8575e7a","Type":"ContainerStarted","Data":"4d02263637f5e0a60739532695e25ac3e131e01b26be5d1b6ab442f3fec97c39"}
Sep 10 15:27:43.087743 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:43.087642    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-6nx2l" podStartSLOduration=8.753173968 podStartE2EDuration="38.087621917s" totalImagesPullingTime="29.334447949s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:27:05 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:27:43.086140505 +0000 UTC m=+38.828105665" watchObservedRunningTime="2026-09-10 15:27:43.087621917 +0000 UTC m=+38.829587021"
Sep 10 15:27:44.068402 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:44.068365    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5dnj9" event={"ID":"280d97b4-f718-4bf7-a173-11822ff796fd","Type":"ContainerStarted","Data":"767cd01825f75c82ff7faf387a1fa8baacb6f4ddff16bc6f3819785936efae0a"}
Sep 10 15:27:44.086920 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:44.086864    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-5dnj9" podStartSLOduration=2.362165933 podStartE2EDuration="5.086847937s" totalImagesPullingTime="2.724682004s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:27:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:27:44.08671692 +0000 UTC m=+39.828682033" watchObservedRunningTime="2026-09-10 15:27:44.086847937 +0000 UTC m=+39.828813041"
Sep 10 15:27:46.778322 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:46.778288    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-bk47h"]
Sep 10 15:27:46.807272 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:46.807235    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-bk47h"
Sep 10 15:27:46.810032 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:46.810005    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 10 15:27:46.810506 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:46.810488    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-bk47h"]
Sep 10 15:27:46.811048 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:46.811035    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 10 15:27:46.811096 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:46.811059    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-gncfw\""
Sep 10 15:27:46.896872 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:46.896835    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/66850670-e07c-4df2-bfe0-fd3b5fcb3308-metrics-tls\") pod \"dns-default-tj846\" (UID: \"66850670-e07c-4df2-bfe0-fd3b5fcb3308\") " pod="openshift-dns/dns-default-tj846"
Sep 10 15:27:46.896872 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:46.896875    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxm8z\" (UniqueName: \"kubernetes.io/projected/31f6539c-3d4f-4bfc-a1d6-44d44e6d6e4a-kube-api-access-hxm8z\") pod \"volume-data-source-validator-695c895bd6-bk47h\" (UID: \"31f6539c-3d4f-4bfc-a1d6-44d44e6d6e4a\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-bk47h"
Sep 10 15:27:46.897071 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:46.896980    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 15:27:46.897071 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:46.897036    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66850670-e07c-4df2-bfe0-fd3b5fcb3308-metrics-tls podName:66850670-e07c-4df2-bfe0-fd3b5fcb3308 nodeName:}" failed. No retries permitted until 2026-09-10 15:27:54.897018418 +0000 UTC m=+50.638983510 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/66850670-e07c-4df2-bfe0-fd3b5fcb3308-metrics-tls") pod "dns-default-tj846" (UID: "66850670-e07c-4df2-bfe0-fd3b5fcb3308") : secret "dns-default-metrics-tls" not found
Sep 10 15:27:46.995363 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:46.995325    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-bvhhx"]
Sep 10 15:27:46.998004 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:46.997976    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxm8z\" (UniqueName: \"kubernetes.io/projected/31f6539c-3d4f-4bfc-a1d6-44d44e6d6e4a-kube-api-access-hxm8z\") pod \"volume-data-source-validator-695c895bd6-bk47h\" (UID: \"31f6539c-3d4f-4bfc-a1d6-44d44e6d6e4a\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-bk47h"
Sep 10 15:27:47.015048 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:47.015022    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxm8z\" (UniqueName: \"kubernetes.io/projected/31f6539c-3d4f-4bfc-a1d6-44d44e6d6e4a-kube-api-access-hxm8z\") pod \"volume-data-source-validator-695c895bd6-bk47h\" (UID: \"31f6539c-3d4f-4bfc-a1d6-44d44e6d6e4a\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-bk47h"
Sep 10 15:27:47.016666 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:47.016632    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-np6wn"]
Sep 10 15:27:47.016774 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:47.016747    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-bvhhx"
Sep 10 15:27:47.019777 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:47.019752    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 10 15:27:47.019777 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:47.019774    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-f8b94\""
Sep 10 15:27:47.019941 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:47.019787    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 10 15:27:47.019941 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:47.019831    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 10 15:27:47.042807 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:47.042750    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-bvhhx"]
Sep 10 15:27:47.042807 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:47.042785    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-np6wn"]
Sep 10 15:27:47.042929 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:47.042854    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-np6wn"
Sep 10 15:27:47.045494 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:47.045476    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 10 15:27:47.045630 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:47.045481    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 10 15:27:47.046562 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:47.046548    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-cnrr4\""
Sep 10 15:27:47.047139 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:47.047123    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 10 15:27:47.047263 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:47.047242    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 10 15:27:47.098791 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:47.098760    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nppcm\" (UniqueName: \"kubernetes.io/projected/91e049b8-e314-4efb-a0c6-df668d76028d-kube-api-access-nppcm\") pod \"cluster-samples-operator-7cdcfcf76c-bvhhx\" (UID: \"91e049b8-e314-4efb-a0c6-df668d76028d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-bvhhx"
Sep 10 15:27:47.098967 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:47.098814    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/91e049b8-e314-4efb-a0c6-df668d76028d-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-bvhhx\" (UID: \"91e049b8-e314-4efb-a0c6-df668d76028d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-bvhhx"
Sep 10 15:27:47.115843 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:47.115825    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-bk47h"
Sep 10 15:27:47.199161 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:47.199104    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/91e049b8-e314-4efb-a0c6-df668d76028d-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-bvhhx\" (UID: \"91e049b8-e314-4efb-a0c6-df668d76028d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-bvhhx"
Sep 10 15:27:47.199161 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:47.199143    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/e12b28c6-2050-4b61-9b4d-e87606ac4a97-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-np6wn\" (UID: \"e12b28c6-2050-4b61-9b4d-e87606ac4a97\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-np6wn"
Sep 10 15:27:47.199335 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:47.199262    2559 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 10 15:27:47.199335 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:47.199284    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e12b28c6-2050-4b61-9b4d-e87606ac4a97-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-np6wn\" (UID: \"e12b28c6-2050-4b61-9b4d-e87606ac4a97\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-np6wn"
Sep 10 15:27:47.199335 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:47.199305    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5mr6\" (UniqueName: \"kubernetes.io/projected/e12b28c6-2050-4b61-9b4d-e87606ac4a97-kube-api-access-m5mr6\") pod \"cluster-monitoring-operator-6cdcc96965-np6wn\" (UID: \"e12b28c6-2050-4b61-9b4d-e87606ac4a97\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-np6wn"
Sep 10 15:27:47.199335 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:47.199321    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91e049b8-e314-4efb-a0c6-df668d76028d-samples-operator-tls podName:91e049b8-e314-4efb-a0c6-df668d76028d nodeName:}" failed. No retries permitted until 2026-09-10 15:27:47.699304648 +0000 UTC m=+43.441269729 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/91e049b8-e314-4efb-a0c6-df668d76028d-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-bvhhx" (UID: "91e049b8-e314-4efb-a0c6-df668d76028d") : secret "samples-operator-tls" not found
Sep 10 15:27:47.199462 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:47.199357    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nppcm\" (UniqueName: \"kubernetes.io/projected/91e049b8-e314-4efb-a0c6-df668d76028d-kube-api-access-nppcm\") pod \"cluster-samples-operator-7cdcfcf76c-bvhhx\" (UID: \"91e049b8-e314-4efb-a0c6-df668d76028d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-bvhhx"
Sep 10 15:27:47.216372 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:47.216346    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nppcm\" (UniqueName: \"kubernetes.io/projected/91e049b8-e314-4efb-a0c6-df668d76028d-kube-api-access-nppcm\") pod \"cluster-samples-operator-7cdcfcf76c-bvhhx\" (UID: \"91e049b8-e314-4efb-a0c6-df668d76028d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-bvhhx"
Sep 10 15:27:47.299733 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:47.299644    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e12b28c6-2050-4b61-9b4d-e87606ac4a97-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-np6wn\" (UID: \"e12b28c6-2050-4b61-9b4d-e87606ac4a97\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-np6wn"
Sep 10 15:27:47.299733 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:47.299683    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5mr6\" (UniqueName: \"kubernetes.io/projected/e12b28c6-2050-4b61-9b4d-e87606ac4a97-kube-api-access-m5mr6\") pod \"cluster-monitoring-operator-6cdcc96965-np6wn\" (UID: \"e12b28c6-2050-4b61-9b4d-e87606ac4a97\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-np6wn"
Sep 10 15:27:47.299920 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:47.299746    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/e12b28c6-2050-4b61-9b4d-e87606ac4a97-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-np6wn\" (UID: \"e12b28c6-2050-4b61-9b4d-e87606ac4a97\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-np6wn"
Sep 10 15:27:47.299920 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:47.299800    2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 10 15:27:47.299920 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:47.299878    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e12b28c6-2050-4b61-9b4d-e87606ac4a97-cluster-monitoring-operator-tls podName:e12b28c6-2050-4b61-9b4d-e87606ac4a97 nodeName:}" failed. No retries permitted until 2026-09-10 15:27:47.799857881 +0000 UTC m=+43.541822989 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e12b28c6-2050-4b61-9b4d-e87606ac4a97-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-np6wn" (UID: "e12b28c6-2050-4b61-9b4d-e87606ac4a97") : secret "cluster-monitoring-operator-tls" not found
Sep 10 15:27:47.300500 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:47.300480    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/e12b28c6-2050-4b61-9b4d-e87606ac4a97-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-np6wn\" (UID: \"e12b28c6-2050-4b61-9b4d-e87606ac4a97\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-np6wn"
Sep 10 15:27:47.311135 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:47.311102    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5mr6\" (UniqueName: \"kubernetes.io/projected/e12b28c6-2050-4b61-9b4d-e87606ac4a97-kube-api-access-m5mr6\") pod \"cluster-monitoring-operator-6cdcc96965-np6wn\" (UID: \"e12b28c6-2050-4b61-9b4d-e87606ac4a97\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-np6wn"
Sep 10 15:27:47.403779 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:47.403747    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-bk47h"]
Sep 10 15:27:47.406685 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:47.406647    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31f6539c_3d4f_4bfc_a1d6_44d44e6d6e4a.slice/crio-fb12e954b87358b8d0965aedea9ba46c3ae7a02ea31a56abd3f4a306083dbe68 WatchSource:0}: Error finding container fb12e954b87358b8d0965aedea9ba46c3ae7a02ea31a56abd3f4a306083dbe68: Status 404 returned error can't find the container with id fb12e954b87358b8d0965aedea9ba46c3ae7a02ea31a56abd3f4a306083dbe68
Sep 10 15:27:47.702725 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:47.702687    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/91e049b8-e314-4efb-a0c6-df668d76028d-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-bvhhx\" (UID: \"91e049b8-e314-4efb-a0c6-df668d76028d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-bvhhx"
Sep 10 15:27:47.705721 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:47.703151    2559 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 10 15:27:47.705721 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:47.703250    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91e049b8-e314-4efb-a0c6-df668d76028d-samples-operator-tls podName:91e049b8-e314-4efb-a0c6-df668d76028d nodeName:}" failed. No retries permitted until 2026-09-10 15:27:48.703227985 +0000 UTC m=+44.445193083 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/91e049b8-e314-4efb-a0c6-df668d76028d-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-bvhhx" (UID: "91e049b8-e314-4efb-a0c6-df668d76028d") : secret "samples-operator-tls" not found
Sep 10 15:27:47.803469 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:47.803427    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e12b28c6-2050-4b61-9b4d-e87606ac4a97-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-np6wn\" (UID: \"e12b28c6-2050-4b61-9b4d-e87606ac4a97\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-np6wn"
Sep 10 15:27:47.803938 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:47.803547    2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 10 15:27:47.803938 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:47.803635    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e12b28c6-2050-4b61-9b4d-e87606ac4a97-cluster-monitoring-operator-tls podName:e12b28c6-2050-4b61-9b4d-e87606ac4a97 nodeName:}" failed. No retries permitted until 2026-09-10 15:27:48.803615194 +0000 UTC m=+44.545580292 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e12b28c6-2050-4b61-9b4d-e87606ac4a97-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-np6wn" (UID: "e12b28c6-2050-4b61-9b4d-e87606ac4a97") : secret "cluster-monitoring-operator-tls" not found
Sep 10 15:27:48.078691 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:48.078604    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-bk47h" event={"ID":"31f6539c-3d4f-4bfc-a1d6-44d44e6d6e4a","Type":"ContainerStarted","Data":"fb12e954b87358b8d0965aedea9ba46c3ae7a02ea31a56abd3f4a306083dbe68"}
Sep 10 15:27:48.711191 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:48.711151    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/91e049b8-e314-4efb-a0c6-df668d76028d-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-bvhhx\" (UID: \"91e049b8-e314-4efb-a0c6-df668d76028d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-bvhhx"
Sep 10 15:27:48.711378 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:48.711324    2559 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 10 15:27:48.711435 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:48.711388    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91e049b8-e314-4efb-a0c6-df668d76028d-samples-operator-tls podName:91e049b8-e314-4efb-a0c6-df668d76028d nodeName:}" failed. No retries permitted until 2026-09-10 15:27:50.711370198 +0000 UTC m=+46.453335294 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/91e049b8-e314-4efb-a0c6-df668d76028d-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-bvhhx" (UID: "91e049b8-e314-4efb-a0c6-df668d76028d") : secret "samples-operator-tls" not found
Sep 10 15:27:48.812480 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:48.812440    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e12b28c6-2050-4b61-9b4d-e87606ac4a97-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-np6wn\" (UID: \"e12b28c6-2050-4b61-9b4d-e87606ac4a97\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-np6wn"
Sep 10 15:27:48.812845 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:48.812597    2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 10 15:27:48.812845 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:48.812662    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e12b28c6-2050-4b61-9b4d-e87606ac4a97-cluster-monitoring-operator-tls podName:e12b28c6-2050-4b61-9b4d-e87606ac4a97 nodeName:}" failed. No retries permitted until 2026-09-10 15:27:50.812644814 +0000 UTC m=+46.554609903 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e12b28c6-2050-4b61-9b4d-e87606ac4a97-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-np6wn" (UID: "e12b28c6-2050-4b61-9b4d-e87606ac4a97") : secret "cluster-monitoring-operator-tls" not found
Sep 10 15:27:50.085069 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:50.085029    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-bk47h" event={"ID":"31f6539c-3d4f-4bfc-a1d6-44d44e6d6e4a","Type":"ContainerStarted","Data":"5c1dbad7ed0880697e2953fcaf4f4a5ac38ac1053edbe4816266cf2a7bedf0d3"}
Sep 10 15:27:50.103484 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:50.103432    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-bk47h" podStartSLOduration=2.381865107 podStartE2EDuration="4.103418308s" totalImagesPullingTime="1.721553201s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:27:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:27:50.102650197 +0000 UTC m=+45.844615303" watchObservedRunningTime="2026-09-10 15:27:50.103418308 +0000 UTC m=+45.845383411"
Sep 10 15:27:50.729268 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:50.729228    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/91e049b8-e314-4efb-a0c6-df668d76028d-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-bvhhx\" (UID: \"91e049b8-e314-4efb-a0c6-df668d76028d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-bvhhx"
Sep 10 15:27:50.729452 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:50.729344    2559 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 10 15:27:50.729452 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:50.729397    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91e049b8-e314-4efb-a0c6-df668d76028d-samples-operator-tls podName:91e049b8-e314-4efb-a0c6-df668d76028d nodeName:}" failed. No retries permitted until 2026-09-10 15:27:54.729383175 +0000 UTC m=+50.471348256 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/91e049b8-e314-4efb-a0c6-df668d76028d-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-bvhhx" (UID: "91e049b8-e314-4efb-a0c6-df668d76028d") : secret "samples-operator-tls" not found
Sep 10 15:27:50.771592 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:50.771550    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-m8vxc"]
Sep 10 15:27:50.809156 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:50.809128    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-m8vxc"
Sep 10 15:27:50.811795 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:50.811769    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 10 15:27:50.811937 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:50.811774    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 10 15:27:50.811937 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:50.811849    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-lgskh\""
Sep 10 15:27:50.812770 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:50.812750    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 10 15:27:50.812862 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:50.812793    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-m8vxc"]
Sep 10 15:27:50.812862 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:50.812845    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 10 15:27:50.830051 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:50.830017    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e12b28c6-2050-4b61-9b4d-e87606ac4a97-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-np6wn\" (UID: \"e12b28c6-2050-4b61-9b4d-e87606ac4a97\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-np6wn"
Sep 10 15:27:50.830191 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:50.830176    2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 10 15:27:50.830271 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:50.830261    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e12b28c6-2050-4b61-9b4d-e87606ac4a97-cluster-monitoring-operator-tls podName:e12b28c6-2050-4b61-9b4d-e87606ac4a97 nodeName:}" failed. No retries permitted until 2026-09-10 15:27:54.830238959 +0000 UTC m=+50.572204043 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e12b28c6-2050-4b61-9b4d-e87606ac4a97-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-np6wn" (UID: "e12b28c6-2050-4b61-9b4d-e87606ac4a97") : secret "cluster-monitoring-operator-tls" not found
Sep 10 15:27:50.931123 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:50.931082    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b1d3b8a5-5a0d-4a47-b185-0934868a3e5d-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-m8vxc\" (UID: \"b1d3b8a5-5a0d-4a47-b185-0934868a3e5d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-m8vxc"
Sep 10 15:27:50.931123 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:50.931125    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjprm\" (UniqueName: \"kubernetes.io/projected/b1d3b8a5-5a0d-4a47-b185-0934868a3e5d-kube-api-access-kjprm\") pod \"kube-storage-version-migrator-operator-8d7dd4675-m8vxc\" (UID: \"b1d3b8a5-5a0d-4a47-b185-0934868a3e5d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-m8vxc"
Sep 10 15:27:50.931329 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:50.931208    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b1d3b8a5-5a0d-4a47-b185-0934868a3e5d-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-m8vxc\" (UID: \"b1d3b8a5-5a0d-4a47-b185-0934868a3e5d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-m8vxc"
Sep 10 15:27:51.032633 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:51.032534    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjprm\" (UniqueName: \"kubernetes.io/projected/b1d3b8a5-5a0d-4a47-b185-0934868a3e5d-kube-api-access-kjprm\") pod \"kube-storage-version-migrator-operator-8d7dd4675-m8vxc\" (UID: \"b1d3b8a5-5a0d-4a47-b185-0934868a3e5d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-m8vxc"
Sep 10 15:27:51.032794 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:51.032651    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b1d3b8a5-5a0d-4a47-b185-0934868a3e5d-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-m8vxc\" (UID: \"b1d3b8a5-5a0d-4a47-b185-0934868a3e5d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-m8vxc"
Sep 10 15:27:51.032794 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:51.032753    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b1d3b8a5-5a0d-4a47-b185-0934868a3e5d-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-m8vxc\" (UID: \"b1d3b8a5-5a0d-4a47-b185-0934868a3e5d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-m8vxc"
Sep 10 15:27:51.033226 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:51.033201    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b1d3b8a5-5a0d-4a47-b185-0934868a3e5d-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-m8vxc\" (UID: \"b1d3b8a5-5a0d-4a47-b185-0934868a3e5d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-m8vxc"
Sep 10 15:27:51.035166 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:51.035145    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b1d3b8a5-5a0d-4a47-b185-0934868a3e5d-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-m8vxc\" (UID: \"b1d3b8a5-5a0d-4a47-b185-0934868a3e5d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-m8vxc"
Sep 10 15:27:51.042309 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:51.042276    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjprm\" (UniqueName: \"kubernetes.io/projected/b1d3b8a5-5a0d-4a47-b185-0934868a3e5d-kube-api-access-kjprm\") pod \"kube-storage-version-migrator-operator-8d7dd4675-m8vxc\" (UID: \"b1d3b8a5-5a0d-4a47-b185-0934868a3e5d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-m8vxc"
Sep 10 15:27:51.118118 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:51.118087    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-m8vxc"
Sep 10 15:27:51.410307 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:51.410137    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-m8vxc"]
Sep 10 15:27:51.412692 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:51.412663    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1d3b8a5_5a0d_4a47_b185_0934868a3e5d.slice/crio-7c2c1476a169e28c752c66b240628f41de67cc2f2500b672403d368e3ced1aae WatchSource:0}: Error finding container 7c2c1476a169e28c752c66b240628f41de67cc2f2500b672403d368e3ced1aae: Status 404 returned error can't find the container with id 7c2c1476a169e28c752c66b240628f41de67cc2f2500b672403d368e3ced1aae
Sep 10 15:27:51.478259 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:51.478232    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-f69vm"]
Sep 10 15:27:51.503984 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:51.503958    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-f69vm"]
Sep 10 15:27:51.504120 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:51.504080    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-f69vm"
Sep 10 15:27:51.506612 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:51.506569    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 10 15:27:51.506834 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:51.506819    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-xrkll\""
Sep 10 15:27:51.506834 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:51.506831    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 10 15:27:51.506936 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:51.506865    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 10 15:27:51.507529 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:51.507515    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 10 15:27:51.637587 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:51.637550    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f7537246-aff7-4408-be0d-7a1d69496837-data-volume\") pod \"insights-runtime-extractor-f69vm\" (UID: \"f7537246-aff7-4408-be0d-7a1d69496837\") " pod="openshift-insights/insights-runtime-extractor-f69vm"
Sep 10 15:27:51.637792 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:51.637608    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f7537246-aff7-4408-be0d-7a1d69496837-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f69vm\" (UID: \"f7537246-aff7-4408-be0d-7a1d69496837\") " pod="openshift-insights/insights-runtime-extractor-f69vm"
Sep 10 15:27:51.637792 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:51.637724    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f7537246-aff7-4408-be0d-7a1d69496837-crio-socket\") pod \"insights-runtime-extractor-f69vm\" (UID: \"f7537246-aff7-4408-be0d-7a1d69496837\") " pod="openshift-insights/insights-runtime-extractor-f69vm"
Sep 10 15:27:51.637792 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:51.637781    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f7537246-aff7-4408-be0d-7a1d69496837-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f69vm\" (UID: \"f7537246-aff7-4408-be0d-7a1d69496837\") " pod="openshift-insights/insights-runtime-extractor-f69vm"
Sep 10 15:27:51.637918 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:51.637813    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qctzt\" (UniqueName: \"kubernetes.io/projected/f7537246-aff7-4408-be0d-7a1d69496837-kube-api-access-qctzt\") pod \"insights-runtime-extractor-f69vm\" (UID: \"f7537246-aff7-4408-be0d-7a1d69496837\") " pod="openshift-insights/insights-runtime-extractor-f69vm"
Sep 10 15:27:51.738667 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:51.738555    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f7537246-aff7-4408-be0d-7a1d69496837-crio-socket\") pod \"insights-runtime-extractor-f69vm\" (UID: \"f7537246-aff7-4408-be0d-7a1d69496837\") " pod="openshift-insights/insights-runtime-extractor-f69vm"
Sep 10 15:27:51.738667 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:51.738615    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f7537246-aff7-4408-be0d-7a1d69496837-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f69vm\" (UID: \"f7537246-aff7-4408-be0d-7a1d69496837\") " pod="openshift-insights/insights-runtime-extractor-f69vm"
Sep 10 15:27:51.738667 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:51.738637    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qctzt\" (UniqueName: \"kubernetes.io/projected/f7537246-aff7-4408-be0d-7a1d69496837-kube-api-access-qctzt\") pod \"insights-runtime-extractor-f69vm\" (UID: \"f7537246-aff7-4408-be0d-7a1d69496837\") " pod="openshift-insights/insights-runtime-extractor-f69vm"
Sep 10 15:27:51.738939 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:51.738684    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f7537246-aff7-4408-be0d-7a1d69496837-data-volume\") pod \"insights-runtime-extractor-f69vm\" (UID: \"f7537246-aff7-4408-be0d-7a1d69496837\") " pod="openshift-insights/insights-runtime-extractor-f69vm"
Sep 10 15:27:51.738939 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:51.738704    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f7537246-aff7-4408-be0d-7a1d69496837-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f69vm\" (UID: \"f7537246-aff7-4408-be0d-7a1d69496837\") " pod="openshift-insights/insights-runtime-extractor-f69vm"
Sep 10 15:27:51.738939 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:51.738749    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 15:27:51.738939 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:51.738766    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f7537246-aff7-4408-be0d-7a1d69496837-crio-socket\") pod \"insights-runtime-extractor-f69vm\" (UID: \"f7537246-aff7-4408-be0d-7a1d69496837\") " pod="openshift-insights/insights-runtime-extractor-f69vm"
Sep 10 15:27:51.738939 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:51.738791    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 15:27:51.738939 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:51.738829    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7537246-aff7-4408-be0d-7a1d69496837-insights-runtime-extractor-tls podName:f7537246-aff7-4408-be0d-7a1d69496837 nodeName:}" failed. No retries permitted until 2026-09-10 15:27:52.238808296 +0000 UTC m=+47.980773392 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f7537246-aff7-4408-be0d-7a1d69496837-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f69vm" (UID: "f7537246-aff7-4408-be0d-7a1d69496837") : secret "insights-runtime-extractor-tls" not found
Sep 10 15:27:51.738939 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:51.738848    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f7537246-aff7-4408-be0d-7a1d69496837-kube-rbac-proxy-cm podName:f7537246-aff7-4408-be0d-7a1d69496837 nodeName:}" failed. No retries permitted until 2026-09-10 15:27:52.238838942 +0000 UTC m=+47.980804033 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f7537246-aff7-4408-be0d-7a1d69496837-kube-rbac-proxy-cm") pod "insights-runtime-extractor-f69vm" (UID: "f7537246-aff7-4408-be0d-7a1d69496837") : configmap "kube-rbac-proxy" not found
Sep 10 15:27:51.739197 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:51.739053    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f7537246-aff7-4408-be0d-7a1d69496837-data-volume\") pod \"insights-runtime-extractor-f69vm\" (UID: \"f7537246-aff7-4408-be0d-7a1d69496837\") " pod="openshift-insights/insights-runtime-extractor-f69vm"
Sep 10 15:27:51.740700 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:51.740681    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-98lqw"]
Sep 10 15:27:51.750248 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:51.750231    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qctzt\" (UniqueName: \"kubernetes.io/projected/f7537246-aff7-4408-be0d-7a1d69496837-kube-api-access-qctzt\") pod \"insights-runtime-extractor-f69vm\" (UID: \"f7537246-aff7-4408-be0d-7a1d69496837\") " pod="openshift-insights/insights-runtime-extractor-f69vm"
Sep 10 15:27:51.757985 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:51.757963    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-98lqw"]
Sep 10 15:27:51.758105 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:51.758053    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-98lqw"
Sep 10 15:27:51.761127 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:51.761111    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-txtjp\""
Sep 10 15:27:51.839706 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:51.839663    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbhqd\" (UniqueName: \"kubernetes.io/projected/6ead4605-6faa-4ced-9677-48bfd5760935-kube-api-access-kbhqd\") pod \"network-check-source-685456655f-98lqw\" (UID: \"6ead4605-6faa-4ced-9677-48bfd5760935\") " pod="openshift-network-diagnostics/network-check-source-685456655f-98lqw"
Sep 10 15:27:51.940663 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:51.940628    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbhqd\" (UniqueName: \"kubernetes.io/projected/6ead4605-6faa-4ced-9677-48bfd5760935-kube-api-access-kbhqd\") pod \"network-check-source-685456655f-98lqw\" (UID: \"6ead4605-6faa-4ced-9677-48bfd5760935\") " pod="openshift-network-diagnostics/network-check-source-685456655f-98lqw"
Sep 10 15:27:51.950384 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:51.950357    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbhqd\" (UniqueName: \"kubernetes.io/projected/6ead4605-6faa-4ced-9677-48bfd5760935-kube-api-access-kbhqd\") pod \"network-check-source-685456655f-98lqw\" (UID: \"6ead4605-6faa-4ced-9677-48bfd5760935\") " pod="openshift-network-diagnostics/network-check-source-685456655f-98lqw"
Sep 10 15:27:52.066979 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:52.066898    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-98lqw"
Sep 10 15:27:52.092037 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:52.092000    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-m8vxc" event={"ID":"b1d3b8a5-5a0d-4a47-b185-0934868a3e5d","Type":"ContainerStarted","Data":"7c2c1476a169e28c752c66b240628f41de67cc2f2500b672403d368e3ced1aae"}
Sep 10 15:27:52.243171 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:52.243122    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f7537246-aff7-4408-be0d-7a1d69496837-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f69vm\" (UID: \"f7537246-aff7-4408-be0d-7a1d69496837\") " pod="openshift-insights/insights-runtime-extractor-f69vm"
Sep 10 15:27:52.243567 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:52.243261    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f7537246-aff7-4408-be0d-7a1d69496837-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f69vm\" (UID: \"f7537246-aff7-4408-be0d-7a1d69496837\") " pod="openshift-insights/insights-runtime-extractor-f69vm"
Sep 10 15:27:52.243567 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:52.243309    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 15:27:52.243567 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:52.243350    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 15:27:52.243567 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:52.243399    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7537246-aff7-4408-be0d-7a1d69496837-insights-runtime-extractor-tls podName:f7537246-aff7-4408-be0d-7a1d69496837 nodeName:}" failed. No retries permitted until 2026-09-10 15:27:53.243376903 +0000 UTC m=+48.985341988 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f7537246-aff7-4408-be0d-7a1d69496837-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f69vm" (UID: "f7537246-aff7-4408-be0d-7a1d69496837") : secret "insights-runtime-extractor-tls" not found
Sep 10 15:27:52.243567 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:52.243420    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f7537246-aff7-4408-be0d-7a1d69496837-kube-rbac-proxy-cm podName:f7537246-aff7-4408-be0d-7a1d69496837 nodeName:}" failed. No retries permitted until 2026-09-10 15:27:53.243409819 +0000 UTC m=+48.985374908 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f7537246-aff7-4408-be0d-7a1d69496837-kube-rbac-proxy-cm") pod "insights-runtime-extractor-f69vm" (UID: "f7537246-aff7-4408-be0d-7a1d69496837") : configmap "kube-rbac-proxy" not found
Sep 10 15:27:52.365368 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:52.365282    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-98lqw"]
Sep 10 15:27:52.368804 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:52.368749    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ead4605_6faa_4ced_9677_48bfd5760935.slice/crio-40cec39744749f553e7d6d6f3d4585dc1adeac71c4f6cb53f55aa56b4c552870 WatchSource:0}: Error finding container 40cec39744749f553e7d6d6f3d4585dc1adeac71c4f6cb53f55aa56b4c552870: Status 404 returned error can't find the container with id 40cec39744749f553e7d6d6f3d4585dc1adeac71c4f6cb53f55aa56b4c552870
Sep 10 15:27:53.095826 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:53.095794    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-98lqw" event={"ID":"6ead4605-6faa-4ced-9677-48bfd5760935","Type":"ContainerStarted","Data":"40cec39744749f553e7d6d6f3d4585dc1adeac71c4f6cb53f55aa56b4c552870"}
Sep 10 15:27:53.197310 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:53.197284    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-md7dv_c56c45fc-a485-48b6-a88a-355ba442927f/dns-node-resolver/0.log"
Sep 10 15:27:53.252933 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:53.252894    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f7537246-aff7-4408-be0d-7a1d69496837-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f69vm\" (UID: \"f7537246-aff7-4408-be0d-7a1d69496837\") " pod="openshift-insights/insights-runtime-extractor-f69vm"
Sep 10 15:27:53.253392 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:53.253052    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 15:27:53.253392 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:53.253082    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f7537246-aff7-4408-be0d-7a1d69496837-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f69vm\" (UID: \"f7537246-aff7-4408-be0d-7a1d69496837\") " pod="openshift-insights/insights-runtime-extractor-f69vm"
Sep 10 15:27:53.253392 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:53.253121    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7537246-aff7-4408-be0d-7a1d69496837-insights-runtime-extractor-tls podName:f7537246-aff7-4408-be0d-7a1d69496837 nodeName:}" failed. No retries permitted until 2026-09-10 15:27:55.253097168 +0000 UTC m=+50.995062248 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f7537246-aff7-4408-be0d-7a1d69496837-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f69vm" (UID: "f7537246-aff7-4408-be0d-7a1d69496837") : secret "insights-runtime-extractor-tls" not found
Sep 10 15:27:53.253392 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:53.253149    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 15:27:53.253392 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:53.253202    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f7537246-aff7-4408-be0d-7a1d69496837-kube-rbac-proxy-cm podName:f7537246-aff7-4408-be0d-7a1d69496837 nodeName:}" failed. No retries permitted until 2026-09-10 15:27:55.253185216 +0000 UTC m=+50.995150321 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f7537246-aff7-4408-be0d-7a1d69496837-kube-rbac-proxy-cm") pod "insights-runtime-extractor-f69vm" (UID: "f7537246-aff7-4408-be0d-7a1d69496837") : configmap "kube-rbac-proxy" not found
Sep 10 15:27:53.767136 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:53.767100    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-dzbgv"]
Sep 10 15:27:53.775521 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:53.775502    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-dzbgv"
Sep 10 15:27:53.777818 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:53.777791    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-dzbgv"]
Sep 10 15:27:53.778203 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:53.778185    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 10 15:27:53.779451 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:53.779430    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 10 15:27:53.779544 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:53.779471    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 10 15:27:53.779544 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:53.779500    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 10 15:27:53.779544 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:53.779518    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-8kppb\""
Sep 10 15:27:53.959177 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:53.959150    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3a58e876-d7a9-4bc4-8f33-a253b1fe149e-serving-cert\") pod \"service-ca-operator-859f94c4b-dzbgv\" (UID: \"3a58e876-d7a9-4bc4-8f33-a253b1fe149e\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-dzbgv"
Sep 10 15:27:53.959381 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:53.959179    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3a58e876-d7a9-4bc4-8f33-a253b1fe149e-config\") pod \"service-ca-operator-859f94c4b-dzbgv\" (UID: \"3a58e876-d7a9-4bc4-8f33-a253b1fe149e\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-dzbgv"
Sep 10 15:27:53.959381 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:53.959281    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpdqh\" (UniqueName: \"kubernetes.io/projected/3a58e876-d7a9-4bc4-8f33-a253b1fe149e-kube-api-access-dpdqh\") pod \"service-ca-operator-859f94c4b-dzbgv\" (UID: \"3a58e876-d7a9-4bc4-8f33-a253b1fe149e\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-dzbgv"
Sep 10 15:27:54.060381 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:54.060295    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpdqh\" (UniqueName: \"kubernetes.io/projected/3a58e876-d7a9-4bc4-8f33-a253b1fe149e-kube-api-access-dpdqh\") pod \"service-ca-operator-859f94c4b-dzbgv\" (UID: \"3a58e876-d7a9-4bc4-8f33-a253b1fe149e\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-dzbgv"
Sep 10 15:27:54.060381 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:54.060375    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3a58e876-d7a9-4bc4-8f33-a253b1fe149e-serving-cert\") pod \"service-ca-operator-859f94c4b-dzbgv\" (UID: \"3a58e876-d7a9-4bc4-8f33-a253b1fe149e\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-dzbgv"
Sep 10 15:27:54.060637 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:54.060400    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3a58e876-d7a9-4bc4-8f33-a253b1fe149e-config\") pod \"service-ca-operator-859f94c4b-dzbgv\" (UID: \"3a58e876-d7a9-4bc4-8f33-a253b1fe149e\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-dzbgv"
Sep 10 15:27:54.068301 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:54.068266    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3a58e876-d7a9-4bc4-8f33-a253b1fe149e-config\") pod \"service-ca-operator-859f94c4b-dzbgv\" (UID: \"3a58e876-d7a9-4bc4-8f33-a253b1fe149e\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-dzbgv"
Sep 10 15:27:54.070298 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:54.070272    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3a58e876-d7a9-4bc4-8f33-a253b1fe149e-serving-cert\") pod \"service-ca-operator-859f94c4b-dzbgv\" (UID: \"3a58e876-d7a9-4bc4-8f33-a253b1fe149e\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-dzbgv"
Sep 10 15:27:54.071423 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:54.071401    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpdqh\" (UniqueName: \"kubernetes.io/projected/3a58e876-d7a9-4bc4-8f33-a253b1fe149e-kube-api-access-dpdqh\") pod \"service-ca-operator-859f94c4b-dzbgv\" (UID: \"3a58e876-d7a9-4bc4-8f33-a253b1fe149e\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-dzbgv"
Sep 10 15:27:54.085510 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:54.085484    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-dzbgv"
Sep 10 15:27:54.100506 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:54.100478    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-m8vxc" event={"ID":"b1d3b8a5-5a0d-4a47-b185-0934868a3e5d","Type":"ContainerStarted","Data":"3f29cc3e16f08c9df36c24fe52a9db2b24ea21e5b208788c84d39fc911bbe574"}
Sep 10 15:27:54.120410 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:54.120352    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-m8vxc" podStartSLOduration=1.6340782790000001 podStartE2EDuration="4.120330552s" totalImagesPullingTime="2.486252273s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:27:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:27:54.120133589 +0000 UTC m=+49.862098693" watchObservedRunningTime="2026-09-10 15:27:54.120330552 +0000 UTC m=+49.862295657"
Sep 10 15:27:54.196466 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:54.196437    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-pzn2d_6db07abf-0c03-4565-a661-9df53fbe5957/node-ca/0.log"
Sep 10 15:27:54.371633 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:54.371537    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-dzbgv"]
Sep 10 15:27:54.374572 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:27:54.374542    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a58e876_d7a9_4bc4_8f33_a253b1fe149e.slice/crio-8f3282d8147505db7e2e1467c1d7bc5c810f3061ce6df8895e5b9998ebba0fad WatchSource:0}: Error finding container 8f3282d8147505db7e2e1467c1d7bc5c810f3061ce6df8895e5b9998ebba0fad: Status 404 returned error can't find the container with id 8f3282d8147505db7e2e1467c1d7bc5c810f3061ce6df8895e5b9998ebba0fad
Sep 10 15:27:54.767643 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:54.767613    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/91e049b8-e314-4efb-a0c6-df668d76028d-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-bvhhx\" (UID: \"91e049b8-e314-4efb-a0c6-df668d76028d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-bvhhx"
Sep 10 15:27:54.767805 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:54.767725    2559 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 10 15:27:54.767805 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:54.767781    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91e049b8-e314-4efb-a0c6-df668d76028d-samples-operator-tls podName:91e049b8-e314-4efb-a0c6-df668d76028d nodeName:}" failed. No retries permitted until 2026-09-10 15:28:02.767764894 +0000 UTC m=+58.509729994 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/91e049b8-e314-4efb-a0c6-df668d76028d-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-bvhhx" (UID: "91e049b8-e314-4efb-a0c6-df668d76028d") : secret "samples-operator-tls" not found
Sep 10 15:27:54.797071 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:54.797039    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5dnj9_280d97b4-f718-4bf7-a173-11822ff796fd/serve-healthcheck-canary/0.log"
Sep 10 15:27:54.867966 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:54.867920    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e12b28c6-2050-4b61-9b4d-e87606ac4a97-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-np6wn\" (UID: \"e12b28c6-2050-4b61-9b4d-e87606ac4a97\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-np6wn"
Sep 10 15:27:54.868124 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:54.868063    2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 10 15:27:54.868170 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:54.868131    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e12b28c6-2050-4b61-9b4d-e87606ac4a97-cluster-monitoring-operator-tls podName:e12b28c6-2050-4b61-9b4d-e87606ac4a97 nodeName:}" failed. No retries permitted until 2026-09-10 15:28:02.86811505 +0000 UTC m=+58.610080130 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e12b28c6-2050-4b61-9b4d-e87606ac4a97-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-np6wn" (UID: "e12b28c6-2050-4b61-9b4d-e87606ac4a97") : secret "cluster-monitoring-operator-tls" not found
Sep 10 15:27:54.968438 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:54.968405    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/66850670-e07c-4df2-bfe0-fd3b5fcb3308-metrics-tls\") pod \"dns-default-tj846\" (UID: \"66850670-e07c-4df2-bfe0-fd3b5fcb3308\") " pod="openshift-dns/dns-default-tj846"
Sep 10 15:27:54.968614 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:54.968591    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 15:27:54.968683 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:54.968671    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66850670-e07c-4df2-bfe0-fd3b5fcb3308-metrics-tls podName:66850670-e07c-4df2-bfe0-fd3b5fcb3308 nodeName:}" failed. No retries permitted until 2026-09-10 15:28:10.968647424 +0000 UTC m=+66.710612509 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/66850670-e07c-4df2-bfe0-fd3b5fcb3308-metrics-tls") pod "dns-default-tj846" (UID: "66850670-e07c-4df2-bfe0-fd3b5fcb3308") : secret "dns-default-metrics-tls" not found
Sep 10 15:27:55.104993 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:55.104886    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-dzbgv" event={"ID":"3a58e876-d7a9-4bc4-8f33-a253b1fe149e","Type":"ContainerStarted","Data":"8f3282d8147505db7e2e1467c1d7bc5c810f3061ce6df8895e5b9998ebba0fad"}
Sep 10 15:27:55.271231 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:55.271193    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f7537246-aff7-4408-be0d-7a1d69496837-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f69vm\" (UID: \"f7537246-aff7-4408-be0d-7a1d69496837\") " pod="openshift-insights/insights-runtime-extractor-f69vm"
Sep 10 15:27:55.271416 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:55.271325    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 15:27:55.271416 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:55.271397    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7537246-aff7-4408-be0d-7a1d69496837-insights-runtime-extractor-tls podName:f7537246-aff7-4408-be0d-7a1d69496837 nodeName:}" failed. No retries permitted until 2026-09-10 15:27:59.27137648 +0000 UTC m=+55.013341574 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f7537246-aff7-4408-be0d-7a1d69496837-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f69vm" (UID: "f7537246-aff7-4408-be0d-7a1d69496837") : secret "insights-runtime-extractor-tls" not found
Sep 10 15:27:55.271550 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:55.271472    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f7537246-aff7-4408-be0d-7a1d69496837-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f69vm\" (UID: \"f7537246-aff7-4408-be0d-7a1d69496837\") " pod="openshift-insights/insights-runtime-extractor-f69vm"
Sep 10 15:27:55.271627 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:55.271610    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 15:27:55.271689 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:55.271653    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f7537246-aff7-4408-be0d-7a1d69496837-kube-rbac-proxy-cm podName:f7537246-aff7-4408-be0d-7a1d69496837 nodeName:}" failed. No retries permitted until 2026-09-10 15:27:59.271639468 +0000 UTC m=+55.013604550 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f7537246-aff7-4408-be0d-7a1d69496837-kube-rbac-proxy-cm") pod "insights-runtime-extractor-f69vm" (UID: "f7537246-aff7-4408-be0d-7a1d69496837") : configmap "kube-rbac-proxy" not found
Sep 10 15:27:57.111695 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:57.111661    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-98lqw" event={"ID":"6ead4605-6faa-4ced-9677-48bfd5760935","Type":"ContainerStarted","Data":"271f8a89c39ba6fda30ffbf0362d7cc524905a5b5d924c02c5305a3a0b1bd799"}
Sep 10 15:27:57.130857 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:57.130805    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-685456655f-98lqw" podStartSLOduration=2.298498271 podStartE2EDuration="6.130789161s" totalImagesPullingTime="3.83229089s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:27:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:27:57.129254038 +0000 UTC m=+52.871219140" watchObservedRunningTime="2026-09-10 15:27:57.130789161 +0000 UTC m=+52.872754263"
Sep 10 15:27:58.115645 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:58.115612    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-dzbgv" event={"ID":"3a58e876-d7a9-4bc4-8f33-a253b1fe149e","Type":"ContainerStarted","Data":"9f2b034aa3fc1877b6e72b11fc194b593ab4cd4b5140cb5264cbac4c365e75ad"}
Sep 10 15:27:58.134085 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:58.134037    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-dzbgv" podStartSLOduration=1.457483048 podStartE2EDuration="5.134020311s" totalImagesPullingTime="3.676537263s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:27:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:27:58.132900467 +0000 UTC m=+53.874865570" watchObservedRunningTime="2026-09-10 15:27:58.134020311 +0000 UTC m=+53.875985413"
Sep 10 15:27:59.309207 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:59.309157    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f7537246-aff7-4408-be0d-7a1d69496837-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f69vm\" (UID: \"f7537246-aff7-4408-be0d-7a1d69496837\") " pod="openshift-insights/insights-runtime-extractor-f69vm"
Sep 10 15:27:59.309633 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:59.309285    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 15:27:59.309633 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:59.309336    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7537246-aff7-4408-be0d-7a1d69496837-insights-runtime-extractor-tls podName:f7537246-aff7-4408-be0d-7a1d69496837 nodeName:}" failed. No retries permitted until 2026-09-10 15:28:07.309323195 +0000 UTC m=+63.051288279 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f7537246-aff7-4408-be0d-7a1d69496837-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f69vm" (UID: "f7537246-aff7-4408-be0d-7a1d69496837") : secret "insights-runtime-extractor-tls" not found
Sep 10 15:27:59.309633 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:27:59.309360    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f7537246-aff7-4408-be0d-7a1d69496837-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f69vm\" (UID: \"f7537246-aff7-4408-be0d-7a1d69496837\") " pod="openshift-insights/insights-runtime-extractor-f69vm"
Sep 10 15:27:59.309633 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:59.309429    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 15:27:59.309633 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:27:59.309490    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f7537246-aff7-4408-be0d-7a1d69496837-kube-rbac-proxy-cm podName:f7537246-aff7-4408-be0d-7a1d69496837 nodeName:}" failed. No retries permitted until 2026-09-10 15:28:07.309473914 +0000 UTC m=+63.051439000 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f7537246-aff7-4408-be0d-7a1d69496837-kube-rbac-proxy-cm") pod "insights-runtime-extractor-f69vm" (UID: "f7537246-aff7-4408-be0d-7a1d69496837") : configmap "kube-rbac-proxy" not found
Sep 10 15:28:02.834728 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:02.834675    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/91e049b8-e314-4efb-a0c6-df668d76028d-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-bvhhx\" (UID: \"91e049b8-e314-4efb-a0c6-df668d76028d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-bvhhx"
Sep 10 15:28:02.835145 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:28:02.834826    2559 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 10 15:28:02.835145 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:28:02.834898    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91e049b8-e314-4efb-a0c6-df668d76028d-samples-operator-tls podName:91e049b8-e314-4efb-a0c6-df668d76028d nodeName:}" failed. No retries permitted until 2026-09-10 15:28:18.834880725 +0000 UTC m=+74.576845806 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/91e049b8-e314-4efb-a0c6-df668d76028d-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-bvhhx" (UID: "91e049b8-e314-4efb-a0c6-df668d76028d") : secret "samples-operator-tls" not found
Sep 10 15:28:02.935503 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:02.935454    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e12b28c6-2050-4b61-9b4d-e87606ac4a97-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-np6wn\" (UID: \"e12b28c6-2050-4b61-9b4d-e87606ac4a97\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-np6wn"
Sep 10 15:28:02.935683 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:28:02.935659    2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 10 15:28:02.935759 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:28:02.935747    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e12b28c6-2050-4b61-9b4d-e87606ac4a97-cluster-monitoring-operator-tls podName:e12b28c6-2050-4b61-9b4d-e87606ac4a97 nodeName:}" failed. No retries permitted until 2026-09-10 15:28:18.935722421 +0000 UTC m=+74.677687527 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e12b28c6-2050-4b61-9b4d-e87606ac4a97-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-np6wn" (UID: "e12b28c6-2050-4b61-9b4d-e87606ac4a97") : secret "cluster-monitoring-operator-tls" not found
Sep 10 15:28:06.046710 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:06.046682    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-sdjq9"
Sep 10 15:28:07.369039 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:07.368997    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f7537246-aff7-4408-be0d-7a1d69496837-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f69vm\" (UID: \"f7537246-aff7-4408-be0d-7a1d69496837\") " pod="openshift-insights/insights-runtime-extractor-f69vm"
Sep 10 15:28:07.369452 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:07.369074    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f7537246-aff7-4408-be0d-7a1d69496837-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f69vm\" (UID: \"f7537246-aff7-4408-be0d-7a1d69496837\") " pod="openshift-insights/insights-runtime-extractor-f69vm"
Sep 10 15:28:07.369452 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:28:07.369193    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 15:28:07.369452 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:28:07.369245    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7537246-aff7-4408-be0d-7a1d69496837-insights-runtime-extractor-tls podName:f7537246-aff7-4408-be0d-7a1d69496837 nodeName:}" failed. No retries permitted until 2026-09-10 15:28:23.369231652 +0000 UTC m=+79.111196732 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f7537246-aff7-4408-be0d-7a1d69496837-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f69vm" (UID: "f7537246-aff7-4408-be0d-7a1d69496837") : secret "insights-runtime-extractor-tls" not found
Sep 10 15:28:07.369613 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:07.369498    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f7537246-aff7-4408-be0d-7a1d69496837-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f69vm\" (UID: \"f7537246-aff7-4408-be0d-7a1d69496837\") " pod="openshift-insights/insights-runtime-extractor-f69vm"
Sep 10 15:28:10.496291 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:10.496249    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wx7jm\" (UniqueName: \"kubernetes.io/projected/008da33d-29cc-4f79-b53a-d1a674b604e8-kube-api-access-wx7jm\") pod \"network-check-target-qbx8c\" (UID: \"008da33d-29cc-4f79-b53a-d1a674b604e8\") " pod="openshift-network-diagnostics/network-check-target-qbx8c"
Sep 10 15:28:10.496291 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:10.496296    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87bd161e-974d-49ee-861d-22763253529c-metrics-certs\") pod \"network-metrics-daemon-w24m7\" (UID: \"87bd161e-974d-49ee-861d-22763253529c\") " pod="openshift-multus/network-metrics-daemon-w24m7"
Sep 10 15:28:10.496880 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:10.496342    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/96b96370-410d-447c-8fb0-63f56ab52310-original-pull-secret\") pod \"global-pull-secret-syncer-7kktn\" (UID: \"96b96370-410d-447c-8fb0-63f56ab52310\") " pod="kube-system/global-pull-secret-syncer-7kktn"
Sep 10 15:28:10.498814 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:10.498775    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wx7jm\" (UniqueName: \"kubernetes.io/projected/008da33d-29cc-4f79-b53a-d1a674b604e8-kube-api-access-wx7jm\") pod \"network-check-target-qbx8c\" (UID: \"008da33d-29cc-4f79-b53a-d1a674b604e8\") " pod="openshift-network-diagnostics/network-check-target-qbx8c"
Sep 10 15:28:10.498994 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:10.498973    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 10 15:28:10.499065 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:10.499008    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 10 15:28:10.507269 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:28:10.507253    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 10 15:28:10.507344 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:28:10.507320    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87bd161e-974d-49ee-861d-22763253529c-metrics-certs podName:87bd161e-974d-49ee-861d-22763253529c nodeName:}" failed. No retries permitted until 2026-09-10 15:29:14.50730478 +0000 UTC m=+130.249269866 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/87bd161e-974d-49ee-861d-22763253529c-metrics-certs") pod "network-metrics-daemon-w24m7" (UID: "87bd161e-974d-49ee-861d-22763253529c") : secret "metrics-daemon-secret" not found
Sep 10 15:28:10.509185 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:10.509165    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/96b96370-410d-447c-8fb0-63f56ab52310-original-pull-secret\") pod \"global-pull-secret-syncer-7kktn\" (UID: \"96b96370-410d-447c-8fb0-63f56ab52310\") " pod="kube-system/global-pull-secret-syncer-7kktn"
Sep 10 15:28:10.694033 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:10.693998    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-5rkds\""
Sep 10 15:28:10.698008 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:10.697989    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7kktn"
Sep 10 15:28:10.702140 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:10.702123    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qbx8c"
Sep 10 15:28:10.993651 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:10.993540    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-7kktn"]
Sep 10 15:28:10.996093 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:28:10.996062    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96b96370_410d_447c_8fb0_63f56ab52310.slice/crio-426780acbc53ec7b8ff78c129a874e9231785b55a44e2e4edaef6037c12ed9d6 WatchSource:0}: Error finding container 426780acbc53ec7b8ff78c129a874e9231785b55a44e2e4edaef6037c12ed9d6: Status 404 returned error can't find the container with id 426780acbc53ec7b8ff78c129a874e9231785b55a44e2e4edaef6037c12ed9d6
Sep 10 15:28:11.000765 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:11.000634    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/66850670-e07c-4df2-bfe0-fd3b5fcb3308-metrics-tls\") pod \"dns-default-tj846\" (UID: \"66850670-e07c-4df2-bfe0-fd3b5fcb3308\") " pod="openshift-dns/dns-default-tj846"
Sep 10 15:28:11.000765 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:28:11.000734    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 15:28:11.000874 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:28:11.000786    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66850670-e07c-4df2-bfe0-fd3b5fcb3308-metrics-tls podName:66850670-e07c-4df2-bfe0-fd3b5fcb3308 nodeName:}" failed. No retries permitted until 2026-09-10 15:28:43.000767225 +0000 UTC m=+98.742732340 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/66850670-e07c-4df2-bfe0-fd3b5fcb3308-metrics-tls") pod "dns-default-tj846" (UID: "66850670-e07c-4df2-bfe0-fd3b5fcb3308") : secret "dns-default-metrics-tls" not found
Sep 10 15:28:11.016569 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:11.016550    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qbx8c"]
Sep 10 15:28:11.018591 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:28:11.018553    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod008da33d_29cc_4f79_b53a_d1a674b604e8.slice/crio-7f1c504ee1d7aa1e361c0ea01b2345d4bfc1cf4444c9620c9c07f502e1ceffaf WatchSource:0}: Error finding container 7f1c504ee1d7aa1e361c0ea01b2345d4bfc1cf4444c9620c9c07f502e1ceffaf: Status 404 returned error can't find the container with id 7f1c504ee1d7aa1e361c0ea01b2345d4bfc1cf4444c9620c9c07f502e1ceffaf
Sep 10 15:28:11.147754 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:11.147721    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qbx8c" event={"ID":"008da33d-29cc-4f79-b53a-d1a674b604e8","Type":"ContainerStarted","Data":"63439a2495de1325e623f72fda8025b59e56ae18a69ee29aac7573b5cbb70361"}
Sep 10 15:28:11.147893 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:11.147776    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qbx8c" event={"ID":"008da33d-29cc-4f79-b53a-d1a674b604e8","Type":"ContainerStarted","Data":"7f1c504ee1d7aa1e361c0ea01b2345d4bfc1cf4444c9620c9c07f502e1ceffaf"}
Sep 10 15:28:11.147893 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:11.147819    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-qbx8c"
Sep 10 15:28:11.148994 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:11.148973    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-7kktn" event={"ID":"96b96370-410d-447c-8fb0-63f56ab52310","Type":"ContainerStarted","Data":"426780acbc53ec7b8ff78c129a874e9231785b55a44e2e4edaef6037c12ed9d6"}
Sep 10 15:28:11.164091 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:11.164048    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-qbx8c" podStartSLOduration=66.164037416 podStartE2EDuration="1m6.164037416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:27:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:28:11.16318858 +0000 UTC m=+66.905153684" watchObservedRunningTime="2026-09-10 15:28:11.164037416 +0000 UTC m=+66.906002509"
Sep 10 15:28:16.167562 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:16.167524    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-7kktn" event={"ID":"96b96370-410d-447c-8fb0-63f56ab52310","Type":"ContainerStarted","Data":"56db430fa2d4d4c30eee35a9cdec1161d934d9f6d1630b6a3fb1e1a64fdc33d1"}
Sep 10 15:28:16.183502 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:16.183449    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-7kktn" podStartSLOduration=66.850727059 podStartE2EDuration="1m11.18343478s" totalImagesPullingTime="4.332707721s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:27:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:28:16.182840979 +0000 UTC m=+71.924806081" watchObservedRunningTime="2026-09-10 15:28:16.18343478 +0000 UTC m=+71.925399884"
Sep 10 15:28:18.865133 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:18.865085    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/91e049b8-e314-4efb-a0c6-df668d76028d-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-bvhhx\" (UID: \"91e049b8-e314-4efb-a0c6-df668d76028d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-bvhhx"
Sep 10 15:28:18.867567 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:18.867543    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/91e049b8-e314-4efb-a0c6-df668d76028d-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-bvhhx\" (UID: \"91e049b8-e314-4efb-a0c6-df668d76028d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-bvhhx"
Sep 10 15:28:18.965787 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:18.965737    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e12b28c6-2050-4b61-9b4d-e87606ac4a97-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-np6wn\" (UID: \"e12b28c6-2050-4b61-9b4d-e87606ac4a97\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-np6wn"
Sep 10 15:28:18.968409 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:18.968389    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e12b28c6-2050-4b61-9b4d-e87606ac4a97-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-np6wn\" (UID: \"e12b28c6-2050-4b61-9b4d-e87606ac4a97\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-np6wn"
Sep 10 15:28:19.128898 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:19.128820    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-f8b94\""
Sep 10 15:28:19.137004 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:19.136978    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-bvhhx"
Sep 10 15:28:19.154267 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:19.154243    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-cnrr4\""
Sep 10 15:28:19.161561 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:19.161542    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-np6wn"
Sep 10 15:28:19.407388 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:19.407312    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-bvhhx"]
Sep 10 15:28:19.450594 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:19.450556    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-np6wn"]
Sep 10 15:28:19.453496 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:28:19.453463    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode12b28c6_2050_4b61_9b4d_e87606ac4a97.slice/crio-f9760bdb8f57ebe2d924d3234e2e6eeef2b64afb52b106530008d7d9e8eb1d29 WatchSource:0}: Error finding container f9760bdb8f57ebe2d924d3234e2e6eeef2b64afb52b106530008d7d9e8eb1d29: Status 404 returned error can't find the container with id f9760bdb8f57ebe2d924d3234e2e6eeef2b64afb52b106530008d7d9e8eb1d29
Sep 10 15:28:20.182884 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:20.182845    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-np6wn" event={"ID":"e12b28c6-2050-4b61-9b4d-e87606ac4a97","Type":"ContainerStarted","Data":"f9760bdb8f57ebe2d924d3234e2e6eeef2b64afb52b106530008d7d9e8eb1d29"}
Sep 10 15:28:20.184018 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:20.183994    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-bvhhx" event={"ID":"91e049b8-e314-4efb-a0c6-df668d76028d","Type":"ContainerStarted","Data":"7e9bc64173b5c2b7a078dad5221d77709a1d07020a4a8e42219e61f2cca7b485"}
Sep 10 15:28:22.192687 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:22.192653    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-np6wn" event={"ID":"e12b28c6-2050-4b61-9b4d-e87606ac4a97","Type":"ContainerStarted","Data":"33ab08e2e2fa9bbc4fbb5538fd8915c59c9705e37cd25ee18a36de904218bd0a"}
Sep 10 15:28:22.194636 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:22.194606    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-bvhhx" event={"ID":"91e049b8-e314-4efb-a0c6-df668d76028d","Type":"ContainerStarted","Data":"526bb376662f67d4d9e67a2e89426d82c6a677941990e055caf928b2f337236b"}
Sep 10 15:28:22.194750 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:22.194662    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-bvhhx" event={"ID":"91e049b8-e314-4efb-a0c6-df668d76028d","Type":"ContainerStarted","Data":"6734d1d169b83862fafedf3a02fb686f2b8c64c0fecfe8e19dba155f499f1827"}
Sep 10 15:28:22.211185 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:22.211142    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-np6wn" podStartSLOduration=33.824918292 podStartE2EDuration="36.211129575s" totalImagesPullingTime="2.386211283s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:27:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:28:22.21012806 +0000 UTC m=+77.952093162" watchObservedRunningTime="2026-09-10 15:28:22.211129575 +0000 UTC m=+77.953094677"
Sep 10 15:28:22.227989 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:22.227944    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-bvhhx" podStartSLOduration=34.517977568 podStartE2EDuration="36.22792999s" totalImagesPullingTime="1.709952422s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:27:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:28:22.227261954 +0000 UTC m=+77.969227071" watchObservedRunningTime="2026-09-10 15:28:22.22792999 +0000 UTC m=+77.969895094"
Sep 10 15:28:23.087620 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:23.087567    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6bb57f748b-qh5hs"]
Sep 10 15:28:23.089929 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:23.089902    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-755bdb9977-82cjl"]
Sep 10 15:28:23.090059 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:23.089992    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6bb57f748b-qh5hs"
Sep 10 15:28:23.092091 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:23.092073    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-755bdb9977-82cjl"
Sep 10 15:28:23.095247 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:23.095226    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 10 15:28:23.095522 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:23.095497    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 10 15:28:23.096300 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:23.096282    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 10 15:28:23.096384 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:23.096307    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 10 15:28:23.096449 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:23.096430    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 10 15:28:23.096518 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:23.096501    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 10 15:28:23.096592 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:23.096503    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 10 15:28:23.096592 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:23.096506    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-zw2gz\""
Sep 10 15:28:23.096698 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:23.096611    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 10 15:28:23.099266 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:23.099242    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6bb57f748b-qh5hs"]
Sep 10 15:28:23.101561 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:23.101540    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-755bdb9977-82cjl"]
Sep 10 15:28:23.197813 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:23.197784    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/610ec9e2-fc3d-4449-8780-ccce94c839f7-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-755bdb9977-82cjl\" (UID: \"610ec9e2-fc3d-4449-8780-ccce94c839f7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-755bdb9977-82cjl"
Sep 10 15:28:23.197813 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:23.197814    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/610ec9e2-fc3d-4449-8780-ccce94c839f7-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-755bdb9977-82cjl\" (UID: \"610ec9e2-fc3d-4449-8780-ccce94c839f7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-755bdb9977-82cjl"
Sep 10 15:28:23.198253 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:23.197850    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/610ec9e2-fc3d-4449-8780-ccce94c839f7-ca\") pod \"cluster-proxy-proxy-agent-755bdb9977-82cjl\" (UID: \"610ec9e2-fc3d-4449-8780-ccce94c839f7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-755bdb9977-82cjl"
Sep 10 15:28:23.198253 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:23.197891    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcd4r\" (UniqueName: \"kubernetes.io/projected/610ec9e2-fc3d-4449-8780-ccce94c839f7-kube-api-access-jcd4r\") pod \"cluster-proxy-proxy-agent-755bdb9977-82cjl\" (UID: \"610ec9e2-fc3d-4449-8780-ccce94c839f7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-755bdb9977-82cjl"
Sep 10 15:28:23.198253 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:23.198027    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/610ec9e2-fc3d-4449-8780-ccce94c839f7-hub\") pod \"cluster-proxy-proxy-agent-755bdb9977-82cjl\" (UID: \"610ec9e2-fc3d-4449-8780-ccce94c839f7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-755bdb9977-82cjl"
Sep 10 15:28:23.198253 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:23.198113    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/610ec9e2-fc3d-4449-8780-ccce94c839f7-additional-service-ca\") pod \"cluster-proxy-proxy-agent-755bdb9977-82cjl\" (UID: \"610ec9e2-fc3d-4449-8780-ccce94c839f7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-755bdb9977-82cjl"
Sep 10 15:28:23.198253 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:23.198141    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69b45\" (UniqueName: \"kubernetes.io/projected/856dc1f8-4d8d-4466-ad4b-d7385182838f-kube-api-access-69b45\") pod \"managed-serviceaccount-addon-agent-6bb57f748b-qh5hs\" (UID: \"856dc1f8-4d8d-4466-ad4b-d7385182838f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6bb57f748b-qh5hs"
Sep 10 15:28:23.198253 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:23.198159    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/856dc1f8-4d8d-4466-ad4b-d7385182838f-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6bb57f748b-qh5hs\" (UID: \"856dc1f8-4d8d-4466-ad4b-d7385182838f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6bb57f748b-qh5hs"
Sep 10 15:28:23.298738 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:23.298709    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcd4r\" (UniqueName: \"kubernetes.io/projected/610ec9e2-fc3d-4449-8780-ccce94c839f7-kube-api-access-jcd4r\") pod \"cluster-proxy-proxy-agent-755bdb9977-82cjl\" (UID: \"610ec9e2-fc3d-4449-8780-ccce94c839f7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-755bdb9977-82cjl"
Sep 10 15:28:23.298909 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:23.298751    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/610ec9e2-fc3d-4449-8780-ccce94c839f7-hub\") pod \"cluster-proxy-proxy-agent-755bdb9977-82cjl\" (UID: \"610ec9e2-fc3d-4449-8780-ccce94c839f7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-755bdb9977-82cjl"
Sep 10 15:28:23.298909 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:23.298820    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/610ec9e2-fc3d-4449-8780-ccce94c839f7-additional-service-ca\") pod \"cluster-proxy-proxy-agent-755bdb9977-82cjl\" (UID: \"610ec9e2-fc3d-4449-8780-ccce94c839f7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-755bdb9977-82cjl"
Sep 10 15:28:23.299116 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:23.299093    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69b45\" (UniqueName: \"kubernetes.io/projected/856dc1f8-4d8d-4466-ad4b-d7385182838f-kube-api-access-69b45\") pod \"managed-serviceaccount-addon-agent-6bb57f748b-qh5hs\" (UID: \"856dc1f8-4d8d-4466-ad4b-d7385182838f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6bb57f748b-qh5hs"
Sep 10 15:28:23.299160 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:23.299133    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/856dc1f8-4d8d-4466-ad4b-d7385182838f-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6bb57f748b-qh5hs\" (UID: \"856dc1f8-4d8d-4466-ad4b-d7385182838f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6bb57f748b-qh5hs"
Sep 10 15:28:23.299201 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:23.299177    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/610ec9e2-fc3d-4449-8780-ccce94c839f7-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-755bdb9977-82cjl\" (UID: \"610ec9e2-fc3d-4449-8780-ccce94c839f7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-755bdb9977-82cjl"
Sep 10 15:28:23.299249 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:23.299204    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/610ec9e2-fc3d-4449-8780-ccce94c839f7-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-755bdb9977-82cjl\" (UID: \"610ec9e2-fc3d-4449-8780-ccce94c839f7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-755bdb9977-82cjl"
Sep 10 15:28:23.299465 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:23.299441    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/610ec9e2-fc3d-4449-8780-ccce94c839f7-ca\") pod \"cluster-proxy-proxy-agent-755bdb9977-82cjl\" (UID: \"610ec9e2-fc3d-4449-8780-ccce94c839f7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-755bdb9977-82cjl"
Sep 10 15:28:23.299675 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:23.299649    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/610ec9e2-fc3d-4449-8780-ccce94c839f7-additional-service-ca\") pod \"cluster-proxy-proxy-agent-755bdb9977-82cjl\" (UID: \"610ec9e2-fc3d-4449-8780-ccce94c839f7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-755bdb9977-82cjl"
Sep 10 15:28:23.301609 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:23.301566    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/610ec9e2-fc3d-4449-8780-ccce94c839f7-hub\") pod \"cluster-proxy-proxy-agent-755bdb9977-82cjl\" (UID: \"610ec9e2-fc3d-4449-8780-ccce94c839f7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-755bdb9977-82cjl"
Sep 10 15:28:23.301894 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:23.301877    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/610ec9e2-fc3d-4449-8780-ccce94c839f7-ca\") pod \"cluster-proxy-proxy-agent-755bdb9977-82cjl\" (UID: \"610ec9e2-fc3d-4449-8780-ccce94c839f7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-755bdb9977-82cjl"
Sep 10 15:28:23.302108 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:23.302088    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/610ec9e2-fc3d-4449-8780-ccce94c839f7-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-755bdb9977-82cjl\" (UID: \"610ec9e2-fc3d-4449-8780-ccce94c839f7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-755bdb9977-82cjl"
Sep 10 15:28:23.302157 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:23.302134    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/610ec9e2-fc3d-4449-8780-ccce94c839f7-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-755bdb9977-82cjl\" (UID: \"610ec9e2-fc3d-4449-8780-ccce94c839f7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-755bdb9977-82cjl"
Sep 10 15:28:23.302192 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:23.302177    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/856dc1f8-4d8d-4466-ad4b-d7385182838f-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6bb57f748b-qh5hs\" (UID: \"856dc1f8-4d8d-4466-ad4b-d7385182838f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6bb57f748b-qh5hs"
Sep 10 15:28:23.311034 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:23.311009    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcd4r\" (UniqueName: \"kubernetes.io/projected/610ec9e2-fc3d-4449-8780-ccce94c839f7-kube-api-access-jcd4r\") pod \"cluster-proxy-proxy-agent-755bdb9977-82cjl\" (UID: \"610ec9e2-fc3d-4449-8780-ccce94c839f7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-755bdb9977-82cjl"
Sep 10 15:28:23.312459 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:23.312442    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-69b45\" (UniqueName: \"kubernetes.io/projected/856dc1f8-4d8d-4466-ad4b-d7385182838f-kube-api-access-69b45\") pod \"managed-serviceaccount-addon-agent-6bb57f748b-qh5hs\" (UID: \"856dc1f8-4d8d-4466-ad4b-d7385182838f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6bb57f748b-qh5hs"
Sep 10 15:28:23.400230 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:23.400144    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f7537246-aff7-4408-be0d-7a1d69496837-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f69vm\" (UID: \"f7537246-aff7-4408-be0d-7a1d69496837\") " pod="openshift-insights/insights-runtime-extractor-f69vm"
Sep 10 15:28:23.402244 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:23.402226    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f7537246-aff7-4408-be0d-7a1d69496837-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f69vm\" (UID: \"f7537246-aff7-4408-be0d-7a1d69496837\") " pod="openshift-insights/insights-runtime-extractor-f69vm"
Sep 10 15:28:23.407203 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:23.407184    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6bb57f748b-qh5hs"
Sep 10 15:28:23.411678 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:23.411662    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-755bdb9977-82cjl"
Sep 10 15:28:23.615007 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:23.614977    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-xrkll\""
Sep 10 15:28:23.622967 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:23.622948    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-f69vm"
Sep 10 15:28:23.684886 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:23.684855    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6bb57f748b-qh5hs"]
Sep 10 15:28:23.688484 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:23.687653    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-755bdb9977-82cjl"]
Sep 10 15:28:23.690842 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:28:23.690803    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod856dc1f8_4d8d_4466_ad4b_d7385182838f.slice/crio-1b20766e8ff4c879bb9643ec61f332843d5efd71d7c1cdf58723320f99c6721b WatchSource:0}: Error finding container 1b20766e8ff4c879bb9643ec61f332843d5efd71d7c1cdf58723320f99c6721b: Status 404 returned error can't find the container with id 1b20766e8ff4c879bb9643ec61f332843d5efd71d7c1cdf58723320f99c6721b
Sep 10 15:28:23.694106 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:28:23.694083    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod610ec9e2_fc3d_4449_8780_ccce94c839f7.slice/crio-0deb053b6520968f99ea01b36e71c975c52a2d7d60a5e2f0ada7f430ff6b9c24 WatchSource:0}: Error finding container 0deb053b6520968f99ea01b36e71c975c52a2d7d60a5e2f0ada7f430ff6b9c24: Status 404 returned error can't find the container with id 0deb053b6520968f99ea01b36e71c975c52a2d7d60a5e2f0ada7f430ff6b9c24
Sep 10 15:28:23.891794 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:23.891772    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-f69vm"]
Sep 10 15:28:23.894175 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:28:23.894148    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7537246_aff7_4408_be0d_7a1d69496837.slice/crio-2d57f4dc0acd418f3a712ab91df01d23be10cbf67cd2d73b05c25f2e9e74be3d WatchSource:0}: Error finding container 2d57f4dc0acd418f3a712ab91df01d23be10cbf67cd2d73b05c25f2e9e74be3d: Status 404 returned error can't find the container with id 2d57f4dc0acd418f3a712ab91df01d23be10cbf67cd2d73b05c25f2e9e74be3d
Sep 10 15:28:24.202985 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:24.202894    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-755bdb9977-82cjl" event={"ID":"610ec9e2-fc3d-4449-8780-ccce94c839f7","Type":"ContainerStarted","Data":"0deb053b6520968f99ea01b36e71c975c52a2d7d60a5e2f0ada7f430ff6b9c24"}
Sep 10 15:28:24.204111 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:24.204085    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6bb57f748b-qh5hs" event={"ID":"856dc1f8-4d8d-4466-ad4b-d7385182838f","Type":"ContainerStarted","Data":"1b20766e8ff4c879bb9643ec61f332843d5efd71d7c1cdf58723320f99c6721b"}
Sep 10 15:28:24.205533 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:24.205513    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-f69vm" event={"ID":"f7537246-aff7-4408-be0d-7a1d69496837","Type":"ContainerStarted","Data":"c087dbfcba6a70a542b301cffd644186d2a715d181dd449628bdaa1359daf8e1"}
Sep 10 15:28:24.205627 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:24.205558    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-f69vm" event={"ID":"f7537246-aff7-4408-be0d-7a1d69496837","Type":"ContainerStarted","Data":"2d57f4dc0acd418f3a712ab91df01d23be10cbf67cd2d73b05c25f2e9e74be3d"}
Sep 10 15:28:25.214658 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:25.214505    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-f69vm" event={"ID":"f7537246-aff7-4408-be0d-7a1d69496837","Type":"ContainerStarted","Data":"181845dfc3b4fc106ecc5229fa0d54580066612e20bc7e0c83e3116610462cca"}
Sep 10 15:28:29.230053 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:29.230018    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-f69vm" event={"ID":"f7537246-aff7-4408-be0d-7a1d69496837","Type":"ContainerStarted","Data":"414fc34b149414d134dc85f3e5a734669caa07bde8b944921d28e8c1db920f54"}
Sep 10 15:28:29.232235 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:29.232210    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-755bdb9977-82cjl" event={"ID":"610ec9e2-fc3d-4449-8780-ccce94c839f7","Type":"ContainerStarted","Data":"372fc6fd34ca59bbc73f5ad09391dd7b890285f2365295e049132d52a462efb9"}
Sep 10 15:28:29.232366 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:29.232247    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-755bdb9977-82cjl" event={"ID":"610ec9e2-fc3d-4449-8780-ccce94c839f7","Type":"ContainerStarted","Data":"14bddb64f71568d3f948501b3544c745998d43b97e8c8a8dae8171b421fafba3"}
Sep 10 15:28:29.232366 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:29.232261    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-755bdb9977-82cjl" event={"ID":"610ec9e2-fc3d-4449-8780-ccce94c839f7","Type":"ContainerStarted","Data":"d66fcd1c1d966ea3c84a0f8e4a923dc1cbafbb9a9984d7a7efb84383624ff7c6"}
Sep 10 15:28:29.233652 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:29.233635    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6bb57f748b-qh5hs" event={"ID":"856dc1f8-4d8d-4466-ad4b-d7385182838f","Type":"ContainerStarted","Data":"37db8b44757c73749d9efe08491868ee8f66376d6b2c8f1bf84684bd5e6044d5"}
Sep 10 15:28:29.249735 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:29.249696    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-f69vm" podStartSLOduration=34.137459136 podStartE2EDuration="38.249683756s" totalImagesPullingTime="4.11222462s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:27:51 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:28:29.247847845 +0000 UTC m=+84.989812948" watchObservedRunningTime="2026-09-10 15:28:29.249683756 +0000 UTC m=+84.991648859"
Sep 10 15:28:29.263200 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:29.263166    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6bb57f748b-qh5hs" podStartSLOduration=1.780029037 podStartE2EDuration="6.26315408s" totalImagesPullingTime="4.483125043s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:28:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:28:29.26268918 +0000 UTC m=+85.004654283" watchObservedRunningTime="2026-09-10 15:28:29.26315408 +0000 UTC m=+85.005119183"
Sep 10 15:28:29.282926 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:29.282890    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-755bdb9977-82cjl" podStartSLOduration=1.785475981 podStartE2EDuration="6.282879745s" totalImagesPullingTime="4.497403764s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:28:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:28:29.282721198 +0000 UTC m=+85.024686300" watchObservedRunningTime="2026-09-10 15:28:29.282879745 +0000 UTC m=+85.024844848"
Sep 10 15:28:32.886508 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:32.886461    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-hwn4s"]
Sep 10 15:28:32.889732 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:32.889711    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-hwn4s"
Sep 10 15:28:32.892839 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:32.892822    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 10 15:28:32.892839 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:32.892830    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 10 15:28:32.893009 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:32.892880    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 10 15:28:32.893009 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:32.892885    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 10 15:28:32.893009 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:32.893006    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-nl9w4\""
Sep 10 15:28:32.981488 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:32.981447    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/59556fec-b586-41af-b85f-0e370544f35a-metrics-client-ca\") pod \"node-exporter-hwn4s\" (UID: \"59556fec-b586-41af-b85f-0e370544f35a\") " pod="openshift-monitoring/node-exporter-hwn4s"
Sep 10 15:28:32.981488 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:32.981486    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/59556fec-b586-41af-b85f-0e370544f35a-node-exporter-accelerators-collector-config\") pod \"node-exporter-hwn4s\" (UID: \"59556fec-b586-41af-b85f-0e370544f35a\") " pod="openshift-monitoring/node-exporter-hwn4s"
Sep 10 15:28:32.981728 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:32.981603    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/59556fec-b586-41af-b85f-0e370544f35a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hwn4s\" (UID: \"59556fec-b586-41af-b85f-0e370544f35a\") " pod="openshift-monitoring/node-exporter-hwn4s"
Sep 10 15:28:32.981728 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:32.981647    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rk8z5\" (UniqueName: \"kubernetes.io/projected/59556fec-b586-41af-b85f-0e370544f35a-kube-api-access-rk8z5\") pod \"node-exporter-hwn4s\" (UID: \"59556fec-b586-41af-b85f-0e370544f35a\") " pod="openshift-monitoring/node-exporter-hwn4s"
Sep 10 15:28:32.981728 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:32.981698    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/59556fec-b586-41af-b85f-0e370544f35a-sys\") pod \"node-exporter-hwn4s\" (UID: \"59556fec-b586-41af-b85f-0e370544f35a\") " pod="openshift-monitoring/node-exporter-hwn4s"
Sep 10 15:28:32.981728 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:32.981716    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/59556fec-b586-41af-b85f-0e370544f35a-root\") pod \"node-exporter-hwn4s\" (UID: \"59556fec-b586-41af-b85f-0e370544f35a\") " pod="openshift-monitoring/node-exporter-hwn4s"
Sep 10 15:28:32.981912 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:32.981742    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/59556fec-b586-41af-b85f-0e370544f35a-node-exporter-tls\") pod \"node-exporter-hwn4s\" (UID: \"59556fec-b586-41af-b85f-0e370544f35a\") " pod="openshift-monitoring/node-exporter-hwn4s"
Sep 10 15:28:32.981912 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:32.981771    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/59556fec-b586-41af-b85f-0e370544f35a-node-exporter-textfile\") pod \"node-exporter-hwn4s\" (UID: \"59556fec-b586-41af-b85f-0e370544f35a\") " pod="openshift-monitoring/node-exporter-hwn4s"
Sep 10 15:28:32.981912 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:32.981791    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/59556fec-b586-41af-b85f-0e370544f35a-node-exporter-wtmp\") pod \"node-exporter-hwn4s\" (UID: \"59556fec-b586-41af-b85f-0e370544f35a\") " pod="openshift-monitoring/node-exporter-hwn4s"
Sep 10 15:28:33.083121 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:33.083086    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rk8z5\" (UniqueName: \"kubernetes.io/projected/59556fec-b586-41af-b85f-0e370544f35a-kube-api-access-rk8z5\") pod \"node-exporter-hwn4s\" (UID: \"59556fec-b586-41af-b85f-0e370544f35a\") " pod="openshift-monitoring/node-exporter-hwn4s"
Sep 10 15:28:33.083121 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:33.083128    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/59556fec-b586-41af-b85f-0e370544f35a-sys\") pod \"node-exporter-hwn4s\" (UID: \"59556fec-b586-41af-b85f-0e370544f35a\") " pod="openshift-monitoring/node-exporter-hwn4s"
Sep 10 15:28:33.083361 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:33.083148    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/59556fec-b586-41af-b85f-0e370544f35a-root\") pod \"node-exporter-hwn4s\" (UID: \"59556fec-b586-41af-b85f-0e370544f35a\") " pod="openshift-monitoring/node-exporter-hwn4s"
Sep 10 15:28:33.083361 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:33.083192    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/59556fec-b586-41af-b85f-0e370544f35a-node-exporter-tls\") pod \"node-exporter-hwn4s\" (UID: \"59556fec-b586-41af-b85f-0e370544f35a\") " pod="openshift-monitoring/node-exporter-hwn4s"
Sep 10 15:28:33.083361 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:33.083217    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/59556fec-b586-41af-b85f-0e370544f35a-sys\") pod \"node-exporter-hwn4s\" (UID: \"59556fec-b586-41af-b85f-0e370544f35a\") " pod="openshift-monitoring/node-exporter-hwn4s"
Sep 10 15:28:33.083361 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:33.083199    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/59556fec-b586-41af-b85f-0e370544f35a-root\") pod \"node-exporter-hwn4s\" (UID: \"59556fec-b586-41af-b85f-0e370544f35a\") " pod="openshift-monitoring/node-exporter-hwn4s"
Sep 10 15:28:33.083361 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:33.083241    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/59556fec-b586-41af-b85f-0e370544f35a-node-exporter-textfile\") pod \"node-exporter-hwn4s\" (UID: \"59556fec-b586-41af-b85f-0e370544f35a\") " pod="openshift-monitoring/node-exporter-hwn4s"
Sep 10 15:28:33.083361 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:33.083281    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/59556fec-b586-41af-b85f-0e370544f35a-node-exporter-wtmp\") pod \"node-exporter-hwn4s\" (UID: \"59556fec-b586-41af-b85f-0e370544f35a\") " pod="openshift-monitoring/node-exporter-hwn4s"
Sep 10 15:28:33.083361 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:33.083324    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/59556fec-b586-41af-b85f-0e370544f35a-metrics-client-ca\") pod \"node-exporter-hwn4s\" (UID: \"59556fec-b586-41af-b85f-0e370544f35a\") " pod="openshift-monitoring/node-exporter-hwn4s"
Sep 10 15:28:33.083361 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:33.083351    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/59556fec-b586-41af-b85f-0e370544f35a-node-exporter-accelerators-collector-config\") pod \"node-exporter-hwn4s\" (UID: \"59556fec-b586-41af-b85f-0e370544f35a\") " pod="openshift-monitoring/node-exporter-hwn4s"
Sep 10 15:28:33.083776 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:28:33.083379    2559 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 10 15:28:33.083776 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:28:33.083448    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59556fec-b586-41af-b85f-0e370544f35a-node-exporter-tls podName:59556fec-b586-41af-b85f-0e370544f35a nodeName:}" failed. No retries permitted until 2026-09-10 15:28:33.583425399 +0000 UTC m=+89.325390499 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/59556fec-b586-41af-b85f-0e370544f35a-node-exporter-tls") pod "node-exporter-hwn4s" (UID: "59556fec-b586-41af-b85f-0e370544f35a") : secret "node-exporter-tls" not found
Sep 10 15:28:33.083776 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:33.083464    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/59556fec-b586-41af-b85f-0e370544f35a-node-exporter-wtmp\") pod \"node-exporter-hwn4s\" (UID: \"59556fec-b586-41af-b85f-0e370544f35a\") " pod="openshift-monitoring/node-exporter-hwn4s"
Sep 10 15:28:33.083776 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:33.083471    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/59556fec-b586-41af-b85f-0e370544f35a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hwn4s\" (UID: \"59556fec-b586-41af-b85f-0e370544f35a\") " pod="openshift-monitoring/node-exporter-hwn4s"
Sep 10 15:28:33.083776 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:33.083648    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/59556fec-b586-41af-b85f-0e370544f35a-node-exporter-textfile\") pod \"node-exporter-hwn4s\" (UID: \"59556fec-b586-41af-b85f-0e370544f35a\") " pod="openshift-monitoring/node-exporter-hwn4s"
Sep 10 15:28:33.083981 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:33.083928    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/59556fec-b586-41af-b85f-0e370544f35a-node-exporter-accelerators-collector-config\") pod \"node-exporter-hwn4s\" (UID: \"59556fec-b586-41af-b85f-0e370544f35a\") " pod="openshift-monitoring/node-exporter-hwn4s"
Sep 10 15:28:33.083981 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:33.083928    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/59556fec-b586-41af-b85f-0e370544f35a-metrics-client-ca\") pod \"node-exporter-hwn4s\" (UID: \"59556fec-b586-41af-b85f-0e370544f35a\") " pod="openshift-monitoring/node-exporter-hwn4s"
Sep 10 15:28:33.085668 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:33.085647    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/59556fec-b586-41af-b85f-0e370544f35a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hwn4s\" (UID: \"59556fec-b586-41af-b85f-0e370544f35a\") " pod="openshift-monitoring/node-exporter-hwn4s"
Sep 10 15:28:33.092166 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:33.092142    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rk8z5\" (UniqueName: \"kubernetes.io/projected/59556fec-b586-41af-b85f-0e370544f35a-kube-api-access-rk8z5\") pod \"node-exporter-hwn4s\" (UID: \"59556fec-b586-41af-b85f-0e370544f35a\") " pod="openshift-monitoring/node-exporter-hwn4s"
Sep 10 15:28:33.587522 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:33.587485    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/59556fec-b586-41af-b85f-0e370544f35a-node-exporter-tls\") pod \"node-exporter-hwn4s\" (UID: \"59556fec-b586-41af-b85f-0e370544f35a\") " pod="openshift-monitoring/node-exporter-hwn4s"
Sep 10 15:28:33.589864 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:33.589837    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/59556fec-b586-41af-b85f-0e370544f35a-node-exporter-tls\") pod \"node-exporter-hwn4s\" (UID: \"59556fec-b586-41af-b85f-0e370544f35a\") " pod="openshift-monitoring/node-exporter-hwn4s"
Sep 10 15:28:33.798855 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:33.798820    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-hwn4s"
Sep 10 15:28:33.810074 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:28:33.810046    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59556fec_b586_41af_b85f_0e370544f35a.slice/crio-8a92cf1839f3cc25dac9a5e9646cea2e9eae928154f8b58ecaa5ffc741299b7b WatchSource:0}: Error finding container 8a92cf1839f3cc25dac9a5e9646cea2e9eae928154f8b58ecaa5ffc741299b7b: Status 404 returned error can't find the container with id 8a92cf1839f3cc25dac9a5e9646cea2e9eae928154f8b58ecaa5ffc741299b7b
Sep 10 15:28:34.254825 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:34.254787    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hwn4s" event={"ID":"59556fec-b586-41af-b85f-0e370544f35a","Type":"ContainerStarted","Data":"8a92cf1839f3cc25dac9a5e9646cea2e9eae928154f8b58ecaa5ffc741299b7b"}
Sep 10 15:28:35.259973 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:35.259881    2559 generic.go:356] "Generic (PLEG): container finished" podID="59556fec-b586-41af-b85f-0e370544f35a" containerID="71bdbf450ec9a852a4e27763609032cb39d3b824a0f58e91c4cb83cf4ce41692" exitCode=0
Sep 10 15:28:35.259973 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:35.259944    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hwn4s" event={"ID":"59556fec-b586-41af-b85f-0e370544f35a","Type":"ContainerDied","Data":"71bdbf450ec9a852a4e27763609032cb39d3b824a0f58e91c4cb83cf4ce41692"}
Sep 10 15:28:36.265013 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:36.264980    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hwn4s" event={"ID":"59556fec-b586-41af-b85f-0e370544f35a","Type":"ContainerStarted","Data":"d1af1507b47304aa2b45bf464d336cf2cf0e8d3ce114b205056ef7d86ac3ce82"}
Sep 10 15:28:36.265389 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:36.265030    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hwn4s" event={"ID":"59556fec-b586-41af-b85f-0e370544f35a","Type":"ContainerStarted","Data":"7aa75d7c2039afb44155d288fee6a5fe5a849de97a06e64b2eecd211904226f1"}
Sep 10 15:28:36.287824 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:36.287776    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-hwn4s" podStartSLOduration=3.086088941 podStartE2EDuration="4.287760831s" totalImagesPullingTime="1.140149325s" totalInitContainerRuntime="61.522565ms" isStatefulPod=true podCreationTimestamp="2026-09-10 15:28:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:28:36.28638874 +0000 UTC m=+92.028353842" watchObservedRunningTime="2026-09-10 15:28:36.287760831 +0000 UTC m=+92.029725934"
Sep 10 15:28:42.155539 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:42.155507    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-qbx8c"
Sep 10 15:28:43.065791 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:43.065750    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/66850670-e07c-4df2-bfe0-fd3b5fcb3308-metrics-tls\") pod \"dns-default-tj846\" (UID: \"66850670-e07c-4df2-bfe0-fd3b5fcb3308\") " pod="openshift-dns/dns-default-tj846"
Sep 10 15:28:43.067978 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:43.067954    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/66850670-e07c-4df2-bfe0-fd3b5fcb3308-metrics-tls\") pod \"dns-default-tj846\" (UID: \"66850670-e07c-4df2-bfe0-fd3b5fcb3308\") " pod="openshift-dns/dns-default-tj846"
Sep 10 15:28:43.340173 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:43.340082    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-w5mt4\""
Sep 10 15:28:43.347385 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:43.347360    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tj846"
Sep 10 15:28:43.616368 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:43.616344    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-tj846"]
Sep 10 15:28:43.619678 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:28:43.619649    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66850670_e07c_4df2_bfe0_fd3b5fcb3308.slice/crio-b01394b4413a0cad79134746c2b11a7b1e8ec03b51c557578f4cd7ec29f2ec50 WatchSource:0}: Error finding container b01394b4413a0cad79134746c2b11a7b1e8ec03b51c557578f4cd7ec29f2ec50: Status 404 returned error can't find the container with id b01394b4413a0cad79134746c2b11a7b1e8ec03b51c557578f4cd7ec29f2ec50
Sep 10 15:28:44.293784 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:44.293747    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tj846" event={"ID":"66850670-e07c-4df2-bfe0-fd3b5fcb3308","Type":"ContainerStarted","Data":"b01394b4413a0cad79134746c2b11a7b1e8ec03b51c557578f4cd7ec29f2ec50"}
Sep 10 15:28:45.298662 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:45.298626    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tj846" event={"ID":"66850670-e07c-4df2-bfe0-fd3b5fcb3308","Type":"ContainerStarted","Data":"762ac4b428445e7d3034db2ed0db78d2431d9b8a94bf1b5cbd9e82c5f08c88d0"}
Sep 10 15:28:45.299124 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:45.298688    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tj846" event={"ID":"66850670-e07c-4df2-bfe0-fd3b5fcb3308","Type":"ContainerStarted","Data":"6646dc0539ac753529074009e697ed04c11bd7d314bff66b19198de570ba6d51"}
Sep 10 15:28:45.299124 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:45.298748    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-tj846"
Sep 10 15:28:45.316333 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:45.316284    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-tj846" podStartSLOduration=64.845020987 podStartE2EDuration="1m6.316268258s" totalImagesPullingTime="1.471247271s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:27:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:28:45.315570349 +0000 UTC m=+101.057535456" watchObservedRunningTime="2026-09-10 15:28:45.316268258 +0000 UTC m=+101.058233361"
Sep 10 15:28:55.304800 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:28:55.304770    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-tj846"
Sep 10 15:29:05.281833 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:29:05.281799    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-8d4799894-zb5jb"]
Sep 10 15:29:05.285999 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:29:05.285978    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8d4799894-zb5jb"
Sep 10 15:29:05.288512 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:29:05.288480    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 10 15:29:05.289561 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:29:05.289515    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 10 15:29:05.289561 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:29:05.289548    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-57xrq\""
Sep 10 15:29:05.289561 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:29:05.289558    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 10 15:29:05.289820 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:29:05.289672    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 10 15:29:05.289900 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:29:05.289885    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 10 15:29:05.289961 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:29:05.289912    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 10 15:29:05.289961 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:29:05.289937    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 10 15:29:05.293852 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:29:05.293832    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 10 15:29:05.295073 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:29:05.295051    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8d4799894-zb5jb"]
Sep 10 15:29:05.337795 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:29:05.337751    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ee296cec-5fae-45cd-9519-a490a805b307-trusted-ca-bundle\") pod \"console-8d4799894-zb5jb\" (UID: \"ee296cec-5fae-45cd-9519-a490a805b307\") " pod="openshift-console/console-8d4799894-zb5jb"
Sep 10 15:29:05.337795 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:29:05.337797    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ee296cec-5fae-45cd-9519-a490a805b307-console-serving-cert\") pod \"console-8d4799894-zb5jb\" (UID: \"ee296cec-5fae-45cd-9519-a490a805b307\") " pod="openshift-console/console-8d4799894-zb5jb"
Sep 10 15:29:05.338038 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:29:05.337825    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ee296cec-5fae-45cd-9519-a490a805b307-console-config\") pod \"console-8d4799894-zb5jb\" (UID: \"ee296cec-5fae-45cd-9519-a490a805b307\") " pod="openshift-console/console-8d4799894-zb5jb"
Sep 10 15:29:05.338038 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:29:05.337854    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ee296cec-5fae-45cd-9519-a490a805b307-console-oauth-config\") pod \"console-8d4799894-zb5jb\" (UID: \"ee296cec-5fae-45cd-9519-a490a805b307\") " pod="openshift-console/console-8d4799894-zb5jb"
Sep 10 15:29:05.338038 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:29:05.337872    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ee296cec-5fae-45cd-9519-a490a805b307-service-ca\") pod \"console-8d4799894-zb5jb\" (UID: \"ee296cec-5fae-45cd-9519-a490a805b307\") " pod="openshift-console/console-8d4799894-zb5jb"
Sep 10 15:29:05.338038 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:29:05.337894    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktkz6\" (UniqueName: \"kubernetes.io/projected/ee296cec-5fae-45cd-9519-a490a805b307-kube-api-access-ktkz6\") pod \"console-8d4799894-zb5jb\" (UID: \"ee296cec-5fae-45cd-9519-a490a805b307\") " pod="openshift-console/console-8d4799894-zb5jb"
Sep 10 15:29:05.338038 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:29:05.338003    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ee296cec-5fae-45cd-9519-a490a805b307-oauth-serving-cert\") pod \"console-8d4799894-zb5jb\" (UID: \"ee296cec-5fae-45cd-9519-a490a805b307\") " pod="openshift-console/console-8d4799894-zb5jb"
Sep 10 15:29:05.439204 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:29:05.439155    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ee296cec-5fae-45cd-9519-a490a805b307-trusted-ca-bundle\") pod \"console-8d4799894-zb5jb\" (UID: \"ee296cec-5fae-45cd-9519-a490a805b307\") " pod="openshift-console/console-8d4799894-zb5jb"
Sep 10 15:29:05.439204 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:29:05.439205    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ee296cec-5fae-45cd-9519-a490a805b307-console-serving-cert\") pod \"console-8d4799894-zb5jb\" (UID: \"ee296cec-5fae-45cd-9519-a490a805b307\") " pod="openshift-console/console-8d4799894-zb5jb"
Sep 10 15:29:05.439475 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:29:05.439328    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ee296cec-5fae-45cd-9519-a490a805b307-console-config\") pod \"console-8d4799894-zb5jb\" (UID: \"ee296cec-5fae-45cd-9519-a490a805b307\") " pod="openshift-console/console-8d4799894-zb5jb"
Sep 10 15:29:05.439475 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:29:05.439376    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ee296cec-5fae-45cd-9519-a490a805b307-console-oauth-config\") pod \"console-8d4799894-zb5jb\" (UID: \"ee296cec-5fae-45cd-9519-a490a805b307\") " pod="openshift-console/console-8d4799894-zb5jb"
Sep 10 15:29:05.439475 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:29:05.439401    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ee296cec-5fae-45cd-9519-a490a805b307-service-ca\") pod \"console-8d4799894-zb5jb\" (UID: \"ee296cec-5fae-45cd-9519-a490a805b307\") " pod="openshift-console/console-8d4799894-zb5jb"
Sep 10 15:29:05.439475 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:29:05.439445    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktkz6\" (UniqueName: \"kubernetes.io/projected/ee296cec-5fae-45cd-9519-a490a805b307-kube-api-access-ktkz6\") pod \"console-8d4799894-zb5jb\" (UID: \"ee296cec-5fae-45cd-9519-a490a805b307\") " pod="openshift-console/console-8d4799894-zb5jb"
Sep 10 15:29:05.439776 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:29:05.439484    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ee296cec-5fae-45cd-9519-a490a805b307-oauth-serving-cert\") pod \"console-8d4799894-zb5jb\" (UID: \"ee296cec-5fae-45cd-9519-a490a805b307\") " pod="openshift-console/console-8d4799894-zb5jb"
Sep 10 15:29:05.440103 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:29:05.440076    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ee296cec-5fae-45cd-9519-a490a805b307-console-config\") pod \"console-8d4799894-zb5jb\" (UID: \"ee296cec-5fae-45cd-9519-a490a805b307\") " pod="openshift-console/console-8d4799894-zb5jb"
Sep 10 15:29:05.440194 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:29:05.440142    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ee296cec-5fae-45cd-9519-a490a805b307-trusted-ca-bundle\") pod \"console-8d4799894-zb5jb\" (UID: \"ee296cec-5fae-45cd-9519-a490a805b307\") " pod="openshift-console/console-8d4799894-zb5jb"
Sep 10 15:29:05.440194 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:29:05.440167    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ee296cec-5fae-45cd-9519-a490a805b307-service-ca\") pod \"console-8d4799894-zb5jb\" (UID: \"ee296cec-5fae-45cd-9519-a490a805b307\") " pod="openshift-console/console-8d4799894-zb5jb"
Sep 10 15:29:05.440302 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:29:05.440228    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ee296cec-5fae-45cd-9519-a490a805b307-oauth-serving-cert\") pod \"console-8d4799894-zb5jb\" (UID: \"ee296cec-5fae-45cd-9519-a490a805b307\") " pod="openshift-console/console-8d4799894-zb5jb"
Sep 10 15:29:05.441728 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:29:05.441707    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ee296cec-5fae-45cd-9519-a490a805b307-console-oauth-config\") pod \"console-8d4799894-zb5jb\" (UID: \"ee296cec-5fae-45cd-9519-a490a805b307\") " pod="openshift-console/console-8d4799894-zb5jb"
Sep 10 15:29:05.441823 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:29:05.441807    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ee296cec-5fae-45cd-9519-a490a805b307-console-serving-cert\") pod \"console-8d4799894-zb5jb\" (UID: \"ee296cec-5fae-45cd-9519-a490a805b307\") " pod="openshift-console/console-8d4799894-zb5jb"
Sep 10 15:29:05.448435 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:29:05.448407    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktkz6\" (UniqueName: \"kubernetes.io/projected/ee296cec-5fae-45cd-9519-a490a805b307-kube-api-access-ktkz6\") pod \"console-8d4799894-zb5jb\" (UID: \"ee296cec-5fae-45cd-9519-a490a805b307\") " pod="openshift-console/console-8d4799894-zb5jb"
Sep 10 15:29:05.595422 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:29:05.595317    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8d4799894-zb5jb"
Sep 10 15:29:05.866891 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:29:05.866679    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8d4799894-zb5jb"]
Sep 10 15:29:05.869308 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:29:05.869272    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee296cec_5fae_45cd_9519_a490a805b307.slice/crio-c5e0eadc2b5741c5959f5f78dfefaca933b03e8c0516eef7e587b37f8f84f0f9 WatchSource:0}: Error finding container c5e0eadc2b5741c5959f5f78dfefaca933b03e8c0516eef7e587b37f8f84f0f9: Status 404 returned error can't find the container with id c5e0eadc2b5741c5959f5f78dfefaca933b03e8c0516eef7e587b37f8f84f0f9
Sep 10 15:29:06.380923 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:29:06.380876    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8d4799894-zb5jb" event={"ID":"ee296cec-5fae-45cd-9519-a490a805b307","Type":"ContainerStarted","Data":"c5e0eadc2b5741c5959f5f78dfefaca933b03e8c0516eef7e587b37f8f84f0f9"}
Sep 10 15:29:09.393114 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:29:09.393082    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8d4799894-zb5jb" event={"ID":"ee296cec-5fae-45cd-9519-a490a805b307","Type":"ContainerStarted","Data":"d5f80c1ac9a08f7e86724a027ede6425d269eac99bc6ec7e3ea25f8039fbf7a5"}
Sep 10 15:29:09.394622 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:29:09.394600    2559 generic.go:356] "Generic (PLEG): container finished" podID="b1d3b8a5-5a0d-4a47-b185-0934868a3e5d" containerID="3f29cc3e16f08c9df36c24fe52a9db2b24ea21e5b208788c84d39fc911bbe574" exitCode=0
Sep 10 15:29:09.394690 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:29:09.394665    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-m8vxc" event={"ID":"b1d3b8a5-5a0d-4a47-b185-0934868a3e5d","Type":"ContainerDied","Data":"3f29cc3e16f08c9df36c24fe52a9db2b24ea21e5b208788c84d39fc911bbe574"}
Sep 10 15:29:09.394938 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:29:09.394925    2559 scope.go:117] "RemoveContainer" containerID="3f29cc3e16f08c9df36c24fe52a9db2b24ea21e5b208788c84d39fc911bbe574"
Sep 10 15:29:09.413290 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:29:09.413245    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-8d4799894-zb5jb" podStartSLOduration=1.6329410420000001 podStartE2EDuration="4.41323108s" totalImagesPullingTime="2.780290038s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:29:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:29:09.410126844 +0000 UTC m=+125.152091946" watchObservedRunningTime="2026-09-10 15:29:09.41323108 +0000 UTC m=+125.155196182"
Sep 10 15:29:10.400245 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:29:10.400212    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-m8vxc" event={"ID":"b1d3b8a5-5a0d-4a47-b185-0934868a3e5d","Type":"ContainerStarted","Data":"9dede174776206d136150cf8d85c2aed7d03e3eb84ac441fa00b020d70833bb2"}
Sep 10 15:29:14.512802 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:29:14.512757    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87bd161e-974d-49ee-861d-22763253529c-metrics-certs\") pod \"network-metrics-daemon-w24m7\" (UID: \"87bd161e-974d-49ee-861d-22763253529c\") " pod="openshift-multus/network-metrics-daemon-w24m7"
Sep 10 15:29:14.515023 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:29:14.515003    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87bd161e-974d-49ee-861d-22763253529c-metrics-certs\") pod \"network-metrics-daemon-w24m7\" (UID: \"87bd161e-974d-49ee-861d-22763253529c\") " pod="openshift-multus/network-metrics-daemon-w24m7"
Sep 10 15:29:14.604944 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:29:14.604906    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-h8jcj\""
Sep 10 15:29:14.613107 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:29:14.613086    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w24m7"
Sep 10 15:29:14.880548 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:29:14.880477    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-w24m7"]
Sep 10 15:29:14.883223 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:29:14.883190    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87bd161e_974d_49ee_861d_22763253529c.slice/crio-b2f3fdf76f6630730e59bee89ed285e193c1b0d745b7321b962b529695c6de34 WatchSource:0}: Error finding container b2f3fdf76f6630730e59bee89ed285e193c1b0d745b7321b962b529695c6de34: Status 404 returned error can't find the container with id b2f3fdf76f6630730e59bee89ed285e193c1b0d745b7321b962b529695c6de34
Sep 10 15:29:15.419952 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:29:15.419914    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-w24m7" event={"ID":"87bd161e-974d-49ee-861d-22763253529c","Type":"ContainerStarted","Data":"b2f3fdf76f6630730e59bee89ed285e193c1b0d745b7321b962b529695c6de34"}
Sep 10 15:29:15.595610 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:29:15.595561    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-8d4799894-zb5jb"
Sep 10 15:29:15.596034 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:29:15.595673    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-8d4799894-zb5jb"
Sep 10 15:29:15.601239 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:29:15.601209    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-8d4799894-zb5jb"
Sep 10 15:29:16.425546 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:29:16.425513    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-w24m7" event={"ID":"87bd161e-974d-49ee-861d-22763253529c","Type":"ContainerStarted","Data":"e69efb1b100e309f9d7487ea858cc65a30ced86589d978df412e8bab30194c6e"}
Sep 10 15:29:16.425760 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:29:16.425565    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-w24m7" event={"ID":"87bd161e-974d-49ee-861d-22763253529c","Type":"ContainerStarted","Data":"568f97789eb72f01c6ce7dead5fd7613166a527bd4aed8c862314a6bea13217a"}
Sep 10 15:29:16.429545 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:29:16.429522    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-8d4799894-zb5jb"
Sep 10 15:29:16.442857 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:29:16.442810    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-w24m7" podStartSLOduration=130.555242618 podStartE2EDuration="2m11.442793209s" totalImagesPullingTime="887.550591ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:27:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:29:16.44062237 +0000 UTC m=+132.182587472" watchObservedRunningTime="2026-09-10 15:29:16.442793209 +0000 UTC m=+132.184758311"
Sep 10 15:29:17.952907 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:29:17.952879    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cdcc96965-np6wn_e12b28c6-2050-4b61-9b4d-e87606ac4a97/cluster-monitoring-operator/0.log"
Sep 10 15:29:18.435057 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:29:18.435029    2559 generic.go:356] "Generic (PLEG): container finished" podID="3a58e876-d7a9-4bc4-8f33-a253b1fe149e" containerID="9f2b034aa3fc1877b6e72b11fc194b593ab4cd4b5140cb5264cbac4c365e75ad" exitCode=0
Sep 10 15:29:18.435159 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:29:18.435105    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-dzbgv" event={"ID":"3a58e876-d7a9-4bc4-8f33-a253b1fe149e","Type":"ContainerDied","Data":"9f2b034aa3fc1877b6e72b11fc194b593ab4cd4b5140cb5264cbac4c365e75ad"}
Sep 10 15:29:18.435466 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:29:18.435450    2559 scope.go:117] "RemoveContainer" containerID="9f2b034aa3fc1877b6e72b11fc194b593ab4cd4b5140cb5264cbac4c365e75ad"
Sep 10 15:29:19.440203 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:29:19.440167    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-dzbgv" event={"ID":"3a58e876-d7a9-4bc4-8f33-a253b1fe149e","Type":"ContainerStarted","Data":"8e0a393c9f27c6ccce09cc4675ab09d49e46fc66e5571a8394fde1e93292e871"}
Sep 10 15:29:19.749874 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:29:19.749800    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hwn4s_59556fec-b586-41af-b85f-0e370544f35a/init-textfile/0.log"
Sep 10 15:29:19.951473 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:29:19.951444    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hwn4s_59556fec-b586-41af-b85f-0e370544f35a/node-exporter/0.log"
Sep 10 15:29:20.150730 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:29:20.150653    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hwn4s_59556fec-b586-41af-b85f-0e370544f35a/kube-rbac-proxy/0.log"
Sep 10 15:29:25.550122 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:29:25.550090    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-8d4799894-zb5jb_ee296cec-5fae-45cd-9519-a490a805b307/console/0.log"
Sep 10 15:30:12.712137 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:30:12.712102    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8d4799894-zb5jb"]
Sep 10 15:30:37.733878 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:30:37.733819    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-8d4799894-zb5jb" podUID="ee296cec-5fae-45cd-9519-a490a805b307" containerName="console" containerID="cri-o://d5f80c1ac9a08f7e86724a027ede6425d269eac99bc6ec7e3ea25f8039fbf7a5" gracePeriod=15
Sep 10 15:30:37.972357 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:30:37.972330    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8d4799894-zb5jb"
Sep 10 15:30:38.023429 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:30:38.023332    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ee296cec-5fae-45cd-9519-a490a805b307-service-ca\") pod \"ee296cec-5fae-45cd-9519-a490a805b307\" (UID: \"ee296cec-5fae-45cd-9519-a490a805b307\") "
Sep 10 15:30:38.023429 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:30:38.023370    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ee296cec-5fae-45cd-9519-a490a805b307-oauth-serving-cert\") pod \"ee296cec-5fae-45cd-9519-a490a805b307\" (UID: \"ee296cec-5fae-45cd-9519-a490a805b307\") "
Sep 10 15:30:38.023429 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:30:38.023398    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ee296cec-5fae-45cd-9519-a490a805b307-console-config\") pod \"ee296cec-5fae-45cd-9519-a490a805b307\" (UID: \"ee296cec-5fae-45cd-9519-a490a805b307\") "
Sep 10 15:30:38.023429 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:30:38.023426    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ee296cec-5fae-45cd-9519-a490a805b307-console-serving-cert\") pod \"ee296cec-5fae-45cd-9519-a490a805b307\" (UID: \"ee296cec-5fae-45cd-9519-a490a805b307\") "
Sep 10 15:30:38.023788 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:30:38.023570    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ee296cec-5fae-45cd-9519-a490a805b307-console-oauth-config\") pod \"ee296cec-5fae-45cd-9519-a490a805b307\" (UID: \"ee296cec-5fae-45cd-9519-a490a805b307\") "
Sep 10 15:30:38.023788 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:30:38.023642    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ktkz6\" (UniqueName: \"kubernetes.io/projected/ee296cec-5fae-45cd-9519-a490a805b307-kube-api-access-ktkz6\") pod \"ee296cec-5fae-45cd-9519-a490a805b307\" (UID: \"ee296cec-5fae-45cd-9519-a490a805b307\") "
Sep 10 15:30:38.023788 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:30:38.023686    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ee296cec-5fae-45cd-9519-a490a805b307-trusted-ca-bundle\") pod \"ee296cec-5fae-45cd-9519-a490a805b307\" (UID: \"ee296cec-5fae-45cd-9519-a490a805b307\") "
Sep 10 15:30:38.023937 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:30:38.023839    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee296cec-5fae-45cd-9519-a490a805b307-service-ca" (OuterVolumeSpecName: "service-ca") pod "ee296cec-5fae-45cd-9519-a490a805b307" (UID: "ee296cec-5fae-45cd-9519-a490a805b307"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:30:38.023937 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:30:38.023850    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee296cec-5fae-45cd-9519-a490a805b307-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "ee296cec-5fae-45cd-9519-a490a805b307" (UID: "ee296cec-5fae-45cd-9519-a490a805b307"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:30:38.023937 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:30:38.023853    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee296cec-5fae-45cd-9519-a490a805b307-console-config" (OuterVolumeSpecName: "console-config") pod "ee296cec-5fae-45cd-9519-a490a805b307" (UID: "ee296cec-5fae-45cd-9519-a490a805b307"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:30:38.024255 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:30:38.024230    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee296cec-5fae-45cd-9519-a490a805b307-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "ee296cec-5fae-45cd-9519-a490a805b307" (UID: "ee296cec-5fae-45cd-9519-a490a805b307"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:30:38.025846 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:30:38.025818    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee296cec-5fae-45cd-9519-a490a805b307-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "ee296cec-5fae-45cd-9519-a490a805b307" (UID: "ee296cec-5fae-45cd-9519-a490a805b307"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:30:38.025846 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:30:38.025834    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee296cec-5fae-45cd-9519-a490a805b307-kube-api-access-ktkz6" (OuterVolumeSpecName: "kube-api-access-ktkz6") pod "ee296cec-5fae-45cd-9519-a490a805b307" (UID: "ee296cec-5fae-45cd-9519-a490a805b307"). InnerVolumeSpecName "kube-api-access-ktkz6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:30:38.026009 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:30:38.025921    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee296cec-5fae-45cd-9519-a490a805b307-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "ee296cec-5fae-45cd-9519-a490a805b307" (UID: "ee296cec-5fae-45cd-9519-a490a805b307"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:30:38.124512 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:30:38.124473    2559 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ee296cec-5fae-45cd-9519-a490a805b307-trusted-ca-bundle\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:30:38.124512 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:30:38.124508    2559 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ee296cec-5fae-45cd-9519-a490a805b307-service-ca\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:30:38.124512 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:30:38.124521    2559 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ee296cec-5fae-45cd-9519-a490a805b307-oauth-serving-cert\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:30:38.124780 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:30:38.124533    2559 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ee296cec-5fae-45cd-9519-a490a805b307-console-config\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:30:38.124780 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:30:38.124544    2559 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ee296cec-5fae-45cd-9519-a490a805b307-console-serving-cert\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:30:38.124780 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:30:38.124555    2559 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ee296cec-5fae-45cd-9519-a490a805b307-console-oauth-config\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:30:38.124780 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:30:38.124566    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktkz6\" (UniqueName: \"kubernetes.io/projected/ee296cec-5fae-45cd-9519-a490a805b307-kube-api-access-ktkz6\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:30:38.726914 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:30:38.726881    2559 generic.go:356] "Generic (PLEG): container finished" podID="ee296cec-5fae-45cd-9519-a490a805b307" containerID="d5f80c1ac9a08f7e86724a027ede6425d269eac99bc6ec7e3ea25f8039fbf7a5" exitCode=0
Sep 10 15:30:38.727086 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:30:38.726934    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8d4799894-zb5jb"
Sep 10 15:30:38.727086 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:30:38.726969    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8d4799894-zb5jb" event={"ID":"ee296cec-5fae-45cd-9519-a490a805b307","Type":"ContainerDied","Data":"d5f80c1ac9a08f7e86724a027ede6425d269eac99bc6ec7e3ea25f8039fbf7a5"}
Sep 10 15:30:38.727086 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:30:38.727029    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8d4799894-zb5jb" event={"ID":"ee296cec-5fae-45cd-9519-a490a805b307","Type":"ContainerDied","Data":"c5e0eadc2b5741c5959f5f78dfefaca933b03e8c0516eef7e587b37f8f84f0f9"}
Sep 10 15:30:38.727086 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:30:38.727049    2559 scope.go:117] "RemoveContainer" containerID="d5f80c1ac9a08f7e86724a027ede6425d269eac99bc6ec7e3ea25f8039fbf7a5"
Sep 10 15:30:38.735358 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:30:38.735215    2559 scope.go:117] "RemoveContainer" containerID="d5f80c1ac9a08f7e86724a027ede6425d269eac99bc6ec7e3ea25f8039fbf7a5"
Sep 10 15:30:38.735631 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:30:38.735490    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5f80c1ac9a08f7e86724a027ede6425d269eac99bc6ec7e3ea25f8039fbf7a5\": container with ID starting with d5f80c1ac9a08f7e86724a027ede6425d269eac99bc6ec7e3ea25f8039fbf7a5 not found: ID does not exist" containerID="d5f80c1ac9a08f7e86724a027ede6425d269eac99bc6ec7e3ea25f8039fbf7a5"
Sep 10 15:30:38.735631 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:30:38.735533    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5f80c1ac9a08f7e86724a027ede6425d269eac99bc6ec7e3ea25f8039fbf7a5"} err="failed to get container status \"d5f80c1ac9a08f7e86724a027ede6425d269eac99bc6ec7e3ea25f8039fbf7a5\": rpc error: code = NotFound desc = could not find container \"d5f80c1ac9a08f7e86724a027ede6425d269eac99bc6ec7e3ea25f8039fbf7a5\": container with ID starting with d5f80c1ac9a08f7e86724a027ede6425d269eac99bc6ec7e3ea25f8039fbf7a5 not found: ID does not exist"
Sep 10 15:30:38.748553 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:30:38.748529    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8d4799894-zb5jb"]
Sep 10 15:30:38.753513 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:30:38.753492    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-8d4799894-zb5jb"]
Sep 10 15:30:38.782915 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:30:38.782889    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee296cec-5fae-45cd-9519-a490a805b307" path="/var/lib/kubelet/pods/ee296cec-5fae-45cd-9519-a490a805b307/volumes"
Sep 10 15:31:10.103431 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:31:10.103393    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-685b4d8546-h2x7r"]
Sep 10 15:31:10.103915 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:31:10.103693    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee296cec-5fae-45cd-9519-a490a805b307" containerName="console"
Sep 10 15:31:10.103915 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:31:10.103706    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee296cec-5fae-45cd-9519-a490a805b307" containerName="console"
Sep 10 15:31:10.103915 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:31:10.103752    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee296cec-5fae-45cd-9519-a490a805b307" containerName="console"
Sep 10 15:31:10.107545 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:31:10.107530    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-685b4d8546-h2x7r"
Sep 10 15:31:10.109994 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:31:10.109969    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 10 15:31:10.110108 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:31:10.109999    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-57xrq\""
Sep 10 15:31:10.110209 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:31:10.110194    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 10 15:31:10.111290 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:31:10.111268    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 10 15:31:10.111417 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:31:10.111399    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 10 15:31:10.111508 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:31:10.111482    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 10 15:31:10.111642 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:31:10.111540    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 10 15:31:10.111701 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:31:10.111674    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 10 15:31:10.116536 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:31:10.116517    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 10 15:31:10.117627 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:31:10.117604    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-685b4d8546-h2x7r"]
Sep 10 15:31:10.151313 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:31:10.151285    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e2cacf38-f42a-4dcf-b350-2209b839209e-console-oauth-config\") pod \"console-685b4d8546-h2x7r\" (UID: \"e2cacf38-f42a-4dcf-b350-2209b839209e\") " pod="openshift-console/console-685b4d8546-h2x7r"
Sep 10 15:31:10.151313 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:31:10.151314    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e2cacf38-f42a-4dcf-b350-2209b839209e-oauth-serving-cert\") pod \"console-685b4d8546-h2x7r\" (UID: \"e2cacf38-f42a-4dcf-b350-2209b839209e\") " pod="openshift-console/console-685b4d8546-h2x7r"
Sep 10 15:31:10.151486 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:31:10.151333    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e2cacf38-f42a-4dcf-b350-2209b839209e-console-serving-cert\") pod \"console-685b4d8546-h2x7r\" (UID: \"e2cacf38-f42a-4dcf-b350-2209b839209e\") " pod="openshift-console/console-685b4d8546-h2x7r"
Sep 10 15:31:10.151486 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:31:10.151352    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2cacf38-f42a-4dcf-b350-2209b839209e-trusted-ca-bundle\") pod \"console-685b4d8546-h2x7r\" (UID: \"e2cacf38-f42a-4dcf-b350-2209b839209e\") " pod="openshift-console/console-685b4d8546-h2x7r"
Sep 10 15:31:10.151486 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:31:10.151401    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e2cacf38-f42a-4dcf-b350-2209b839209e-console-config\") pod \"console-685b4d8546-h2x7r\" (UID: \"e2cacf38-f42a-4dcf-b350-2209b839209e\") " pod="openshift-console/console-685b4d8546-h2x7r"
Sep 10 15:31:10.151486 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:31:10.151430    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e2cacf38-f42a-4dcf-b350-2209b839209e-service-ca\") pod \"console-685b4d8546-h2x7r\" (UID: \"e2cacf38-f42a-4dcf-b350-2209b839209e\") " pod="openshift-console/console-685b4d8546-h2x7r"
Sep 10 15:31:10.151486 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:31:10.151456    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fv96c\" (UniqueName: \"kubernetes.io/projected/e2cacf38-f42a-4dcf-b350-2209b839209e-kube-api-access-fv96c\") pod \"console-685b4d8546-h2x7r\" (UID: \"e2cacf38-f42a-4dcf-b350-2209b839209e\") " pod="openshift-console/console-685b4d8546-h2x7r"
Sep 10 15:31:10.252261 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:31:10.252217    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e2cacf38-f42a-4dcf-b350-2209b839209e-service-ca\") pod \"console-685b4d8546-h2x7r\" (UID: \"e2cacf38-f42a-4dcf-b350-2209b839209e\") " pod="openshift-console/console-685b4d8546-h2x7r"
Sep 10 15:31:10.252261 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:31:10.252262    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fv96c\" (UniqueName: \"kubernetes.io/projected/e2cacf38-f42a-4dcf-b350-2209b839209e-kube-api-access-fv96c\") pod \"console-685b4d8546-h2x7r\" (UID: \"e2cacf38-f42a-4dcf-b350-2209b839209e\") " pod="openshift-console/console-685b4d8546-h2x7r"
Sep 10 15:31:10.252498 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:31:10.252296    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e2cacf38-f42a-4dcf-b350-2209b839209e-console-oauth-config\") pod \"console-685b4d8546-h2x7r\" (UID: \"e2cacf38-f42a-4dcf-b350-2209b839209e\") " pod="openshift-console/console-685b4d8546-h2x7r"
Sep 10 15:31:10.252498 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:31:10.252313    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e2cacf38-f42a-4dcf-b350-2209b839209e-oauth-serving-cert\") pod \"console-685b4d8546-h2x7r\" (UID: \"e2cacf38-f42a-4dcf-b350-2209b839209e\") " pod="openshift-console/console-685b4d8546-h2x7r"
Sep 10 15:31:10.252498 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:31:10.252327    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e2cacf38-f42a-4dcf-b350-2209b839209e-console-serving-cert\") pod \"console-685b4d8546-h2x7r\" (UID: \"e2cacf38-f42a-4dcf-b350-2209b839209e\") " pod="openshift-console/console-685b4d8546-h2x7r"
Sep 10 15:31:10.252498 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:31:10.252346    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2cacf38-f42a-4dcf-b350-2209b839209e-trusted-ca-bundle\") pod \"console-685b4d8546-h2x7r\" (UID: \"e2cacf38-f42a-4dcf-b350-2209b839209e\") " pod="openshift-console/console-685b4d8546-h2x7r"
Sep 10 15:31:10.252498 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:31:10.252368    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e2cacf38-f42a-4dcf-b350-2209b839209e-console-config\") pod \"console-685b4d8546-h2x7r\" (UID: \"e2cacf38-f42a-4dcf-b350-2209b839209e\") " pod="openshift-console/console-685b4d8546-h2x7r"
Sep 10 15:31:10.253078 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:31:10.253054    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e2cacf38-f42a-4dcf-b350-2209b839209e-service-ca\") pod \"console-685b4d8546-h2x7r\" (UID: \"e2cacf38-f42a-4dcf-b350-2209b839209e\") " pod="openshift-console/console-685b4d8546-h2x7r"
Sep 10 15:31:10.253194 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:31:10.253095    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e2cacf38-f42a-4dcf-b350-2209b839209e-console-config\") pod \"console-685b4d8546-h2x7r\" (UID: \"e2cacf38-f42a-4dcf-b350-2209b839209e\") " pod="openshift-console/console-685b4d8546-h2x7r"
Sep 10 15:31:10.253194 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:31:10.253153    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e2cacf38-f42a-4dcf-b350-2209b839209e-oauth-serving-cert\") pod \"console-685b4d8546-h2x7r\" (UID: \"e2cacf38-f42a-4dcf-b350-2209b839209e\") " pod="openshift-console/console-685b4d8546-h2x7r"
Sep 10 15:31:10.253354 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:31:10.253332    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2cacf38-f42a-4dcf-b350-2209b839209e-trusted-ca-bundle\") pod \"console-685b4d8546-h2x7r\" (UID: \"e2cacf38-f42a-4dcf-b350-2209b839209e\") " pod="openshift-console/console-685b4d8546-h2x7r"
Sep 10 15:31:10.254920 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:31:10.254901    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e2cacf38-f42a-4dcf-b350-2209b839209e-console-oauth-config\") pod \"console-685b4d8546-h2x7r\" (UID: \"e2cacf38-f42a-4dcf-b350-2209b839209e\") " pod="openshift-console/console-685b4d8546-h2x7r"
Sep 10 15:31:10.255011 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:31:10.254960    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e2cacf38-f42a-4dcf-b350-2209b839209e-console-serving-cert\") pod \"console-685b4d8546-h2x7r\" (UID: \"e2cacf38-f42a-4dcf-b350-2209b839209e\") " pod="openshift-console/console-685b4d8546-h2x7r"
Sep 10 15:31:10.261550 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:31:10.261529    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fv96c\" (UniqueName: \"kubernetes.io/projected/e2cacf38-f42a-4dcf-b350-2209b839209e-kube-api-access-fv96c\") pod \"console-685b4d8546-h2x7r\" (UID: \"e2cacf38-f42a-4dcf-b350-2209b839209e\") " pod="openshift-console/console-685b4d8546-h2x7r"
Sep 10 15:31:10.418592 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:31:10.418545    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-685b4d8546-h2x7r"
Sep 10 15:31:10.689062 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:31:10.688986    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-685b4d8546-h2x7r"]
Sep 10 15:31:10.691857 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:31:10.691829    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2cacf38_f42a_4dcf_b350_2209b839209e.slice/crio-381316f4087c552558b17c321748362ff5009f3c1099542f3a4af3a5b15ed4b5 WatchSource:0}: Error finding container 381316f4087c552558b17c321748362ff5009f3c1099542f3a4af3a5b15ed4b5: Status 404 returned error can't find the container with id 381316f4087c552558b17c321748362ff5009f3c1099542f3a4af3a5b15ed4b5
Sep 10 15:31:10.838027 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:31:10.837986    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-685b4d8546-h2x7r" event={"ID":"e2cacf38-f42a-4dcf-b350-2209b839209e","Type":"ContainerStarted","Data":"66d50222b2402938a41201d83daaecfa698f1d923ca799515ddf70cbfbc92987"}
Sep 10 15:31:10.838196 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:31:10.838045    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-685b4d8546-h2x7r" event={"ID":"e2cacf38-f42a-4dcf-b350-2209b839209e","Type":"ContainerStarted","Data":"381316f4087c552558b17c321748362ff5009f3c1099542f3a4af3a5b15ed4b5"}
Sep 10 15:31:10.857211 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:31:10.857158    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-685b4d8546-h2x7r" podStartSLOduration=0.857143212 podStartE2EDuration="857.143212ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:31:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:31:10.856299875 +0000 UTC m=+246.598265001" watchObservedRunningTime="2026-09-10 15:31:10.857143212 +0000 UTC m=+246.599108367"
Sep 10 15:31:20.419302 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:31:20.419262    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-685b4d8546-h2x7r"
Sep 10 15:31:20.419719 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:31:20.419331    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-685b4d8546-h2x7r"
Sep 10 15:31:20.424074 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:31:20.424054    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-685b4d8546-h2x7r"
Sep 10 15:31:20.876627 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:31:20.876536    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-685b4d8546-h2x7r"
Sep 10 15:32:04.687693 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:32:04.687662    2559 kubelet.go:1648] "Image garbage collection succeeded"
Sep 10 15:32:38.976505 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:32:38.976471    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fz8j4h"]
Sep 10 15:32:38.979876 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:32:38.979856    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fz8j4h"
Sep 10 15:32:38.982749 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:32:38.982719    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 10 15:32:38.983756 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:32:38.983737    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 10 15:32:38.983756 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:32:38.983750    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-984bz\""
Sep 10 15:32:38.999009 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:32:38.998986    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fz8j4h"]
Sep 10 15:32:39.096518 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:32:39.096485    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmzt2\" (UniqueName: \"kubernetes.io/projected/bfd2ba34-6bc2-4d89-8fb6-d76a495e1160-kube-api-access-wmzt2\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fz8j4h\" (UID: \"bfd2ba34-6bc2-4d89-8fb6-d76a495e1160\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fz8j4h"
Sep 10 15:32:39.096737 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:32:39.096551    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bfd2ba34-6bc2-4d89-8fb6-d76a495e1160-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fz8j4h\" (UID: \"bfd2ba34-6bc2-4d89-8fb6-d76a495e1160\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fz8j4h"
Sep 10 15:32:39.096737 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:32:39.096657    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bfd2ba34-6bc2-4d89-8fb6-d76a495e1160-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fz8j4h\" (UID: \"bfd2ba34-6bc2-4d89-8fb6-d76a495e1160\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fz8j4h"
Sep 10 15:32:39.197381 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:32:39.197345    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bfd2ba34-6bc2-4d89-8fb6-d76a495e1160-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fz8j4h\" (UID: \"bfd2ba34-6bc2-4d89-8fb6-d76a495e1160\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fz8j4h"
Sep 10 15:32:39.197573 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:32:39.197400    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bfd2ba34-6bc2-4d89-8fb6-d76a495e1160-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fz8j4h\" (UID: \"bfd2ba34-6bc2-4d89-8fb6-d76a495e1160\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fz8j4h"
Sep 10 15:32:39.197573 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:32:39.197439    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmzt2\" (UniqueName: \"kubernetes.io/projected/bfd2ba34-6bc2-4d89-8fb6-d76a495e1160-kube-api-access-wmzt2\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fz8j4h\" (UID: \"bfd2ba34-6bc2-4d89-8fb6-d76a495e1160\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fz8j4h"
Sep 10 15:32:39.197840 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:32:39.197816    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bfd2ba34-6bc2-4d89-8fb6-d76a495e1160-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fz8j4h\" (UID: \"bfd2ba34-6bc2-4d89-8fb6-d76a495e1160\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fz8j4h"
Sep 10 15:32:39.197901 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:32:39.197841    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bfd2ba34-6bc2-4d89-8fb6-d76a495e1160-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fz8j4h\" (UID: \"bfd2ba34-6bc2-4d89-8fb6-d76a495e1160\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fz8j4h"
Sep 10 15:32:39.208112 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:32:39.208084    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmzt2\" (UniqueName: \"kubernetes.io/projected/bfd2ba34-6bc2-4d89-8fb6-d76a495e1160-kube-api-access-wmzt2\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fz8j4h\" (UID: \"bfd2ba34-6bc2-4d89-8fb6-d76a495e1160\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fz8j4h"
Sep 10 15:32:39.288753 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:32:39.288659    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fz8j4h"
Sep 10 15:32:39.577524 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:32:39.577438    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fz8j4h"]
Sep 10 15:32:39.580681 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:32:39.580650    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfd2ba34_6bc2_4d89_8fb6_d76a495e1160.slice/crio-9f9f214d5866b6ddf769c5eeaae3f6e0cfa1bdef1c64c71cc71dc224c998a1b4 WatchSource:0}: Error finding container 9f9f214d5866b6ddf769c5eeaae3f6e0cfa1bdef1c64c71cc71dc224c998a1b4: Status 404 returned error can't find the container with id 9f9f214d5866b6ddf769c5eeaae3f6e0cfa1bdef1c64c71cc71dc224c998a1b4
Sep 10 15:32:39.582663 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:32:39.582644    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 15:32:40.151013 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:32:40.150977    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fz8j4h" event={"ID":"bfd2ba34-6bc2-4d89-8fb6-d76a495e1160","Type":"ContainerStarted","Data":"9f9f214d5866b6ddf769c5eeaae3f6e0cfa1bdef1c64c71cc71dc224c998a1b4"}
Sep 10 15:32:46.177704 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:32:46.177666    2559 generic.go:356] "Generic (PLEG): container finished" podID="bfd2ba34-6bc2-4d89-8fb6-d76a495e1160" containerID="d0a29bbe0e9fbeb712cddeb8460b4f6d009c16dce175fd9608d3c91dbb6ef7d5" exitCode=0
Sep 10 15:32:46.178184 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:32:46.177722    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fz8j4h" event={"ID":"bfd2ba34-6bc2-4d89-8fb6-d76a495e1160","Type":"ContainerDied","Data":"d0a29bbe0e9fbeb712cddeb8460b4f6d009c16dce175fd9608d3c91dbb6ef7d5"}
Sep 10 15:32:49.191514 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:32:49.191473    2559 generic.go:356] "Generic (PLEG): container finished" podID="bfd2ba34-6bc2-4d89-8fb6-d76a495e1160" containerID="db59081152eb187940200373900dc3d80265e308af90ada5435132f7333dff7a" exitCode=0
Sep 10 15:32:49.191890 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:32:49.191548    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fz8j4h" event={"ID":"bfd2ba34-6bc2-4d89-8fb6-d76a495e1160","Type":"ContainerDied","Data":"db59081152eb187940200373900dc3d80265e308af90ada5435132f7333dff7a"}
Sep 10 15:32:55.219272 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:32:55.219188    2559 generic.go:356] "Generic (PLEG): container finished" podID="bfd2ba34-6bc2-4d89-8fb6-d76a495e1160" containerID="7ac0556cf289a08d646f0cafb8233e0036283d49614275291a0d14ddfac05422" exitCode=0
Sep 10 15:32:55.219641 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:32:55.219271    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fz8j4h" event={"ID":"bfd2ba34-6bc2-4d89-8fb6-d76a495e1160","Type":"ContainerDied","Data":"7ac0556cf289a08d646f0cafb8233e0036283d49614275291a0d14ddfac05422"}
Sep 10 15:32:56.340891 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:32:56.340861    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fz8j4h"
Sep 10 15:32:56.439269 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:32:56.439218    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wmzt2\" (UniqueName: \"kubernetes.io/projected/bfd2ba34-6bc2-4d89-8fb6-d76a495e1160-kube-api-access-wmzt2\") pod \"bfd2ba34-6bc2-4d89-8fb6-d76a495e1160\" (UID: \"bfd2ba34-6bc2-4d89-8fb6-d76a495e1160\") "
Sep 10 15:32:56.439443 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:32:56.439281    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bfd2ba34-6bc2-4d89-8fb6-d76a495e1160-util\") pod \"bfd2ba34-6bc2-4d89-8fb6-d76a495e1160\" (UID: \"bfd2ba34-6bc2-4d89-8fb6-d76a495e1160\") "
Sep 10 15:32:56.439443 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:32:56.439384    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bfd2ba34-6bc2-4d89-8fb6-d76a495e1160-bundle\") pod \"bfd2ba34-6bc2-4d89-8fb6-d76a495e1160\" (UID: \"bfd2ba34-6bc2-4d89-8fb6-d76a495e1160\") "
Sep 10 15:32:56.439758 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:32:56.439725    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfd2ba34-6bc2-4d89-8fb6-d76a495e1160-bundle" (OuterVolumeSpecName: "bundle") pod "bfd2ba34-6bc2-4d89-8fb6-d76a495e1160" (UID: "bfd2ba34-6bc2-4d89-8fb6-d76a495e1160"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:32:56.441544 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:32:56.441521    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfd2ba34-6bc2-4d89-8fb6-d76a495e1160-kube-api-access-wmzt2" (OuterVolumeSpecName: "kube-api-access-wmzt2") pod "bfd2ba34-6bc2-4d89-8fb6-d76a495e1160" (UID: "bfd2ba34-6bc2-4d89-8fb6-d76a495e1160"). InnerVolumeSpecName "kube-api-access-wmzt2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:32:56.443509 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:32:56.443488    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfd2ba34-6bc2-4d89-8fb6-d76a495e1160-util" (OuterVolumeSpecName: "util") pod "bfd2ba34-6bc2-4d89-8fb6-d76a495e1160" (UID: "bfd2ba34-6bc2-4d89-8fb6-d76a495e1160"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:32:56.540617 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:32:56.540513    2559 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bfd2ba34-6bc2-4d89-8fb6-d76a495e1160-bundle\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:32:56.540617 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:32:56.540540    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmzt2\" (UniqueName: \"kubernetes.io/projected/bfd2ba34-6bc2-4d89-8fb6-d76a495e1160-kube-api-access-wmzt2\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:32:56.540617 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:32:56.540549    2559 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bfd2ba34-6bc2-4d89-8fb6-d76a495e1160-util\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:32:57.229195 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:32:57.229158    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fz8j4h" event={"ID":"bfd2ba34-6bc2-4d89-8fb6-d76a495e1160","Type":"ContainerDied","Data":"9f9f214d5866b6ddf769c5eeaae3f6e0cfa1bdef1c64c71cc71dc224c998a1b4"}
Sep 10 15:32:57.229195 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:32:57.229203    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f9f214d5866b6ddf769c5eeaae3f6e0cfa1bdef1c64c71cc71dc224c998a1b4"
Sep 10 15:32:57.229398 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:32:57.229174    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fz8j4h"
Sep 10 15:33:02.826736 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:02.826699    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-v25lg"]
Sep 10 15:33:02.827122 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:02.826955    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfd2ba34-6bc2-4d89-8fb6-d76a495e1160" containerName="extract"
Sep 10 15:33:02.827122 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:02.826965    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfd2ba34-6bc2-4d89-8fb6-d76a495e1160" containerName="extract"
Sep 10 15:33:02.827122 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:02.826974    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfd2ba34-6bc2-4d89-8fb6-d76a495e1160" containerName="pull"
Sep 10 15:33:02.827122 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:02.826980    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfd2ba34-6bc2-4d89-8fb6-d76a495e1160" containerName="pull"
Sep 10 15:33:02.827122 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:02.826998    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfd2ba34-6bc2-4d89-8fb6-d76a495e1160" containerName="util"
Sep 10 15:33:02.827122 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:02.827003    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfd2ba34-6bc2-4d89-8fb6-d76a495e1160" containerName="util"
Sep 10 15:33:02.827122 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:02.827050    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfd2ba34-6bc2-4d89-8fb6-d76a495e1160" containerName="extract"
Sep 10 15:33:02.829840 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:02.829823    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-v25lg"
Sep 10 15:33:02.832450 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:02.832421    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 10 15:33:02.832450 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:02.832430    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-pts78\""
Sep 10 15:33:02.833485 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:02.833466    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 10 15:33:02.838244 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:02.838220    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-v25lg"]
Sep 10 15:33:02.990835 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:02.990795    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ea32fcf8-df19-4d72-82e9-9d48dcd4701d-tmp\") pod \"openshift-lws-operator-fd8ccff4c-v25lg\" (UID: \"ea32fcf8-df19-4d72-82e9-9d48dcd4701d\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-v25lg"
Sep 10 15:33:02.991001 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:02.990850    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wg5hq\" (UniqueName: \"kubernetes.io/projected/ea32fcf8-df19-4d72-82e9-9d48dcd4701d-kube-api-access-wg5hq\") pod \"openshift-lws-operator-fd8ccff4c-v25lg\" (UID: \"ea32fcf8-df19-4d72-82e9-9d48dcd4701d\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-v25lg"
Sep 10 15:33:03.092329 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:03.092240    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wg5hq\" (UniqueName: \"kubernetes.io/projected/ea32fcf8-df19-4d72-82e9-9d48dcd4701d-kube-api-access-wg5hq\") pod \"openshift-lws-operator-fd8ccff4c-v25lg\" (UID: \"ea32fcf8-df19-4d72-82e9-9d48dcd4701d\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-v25lg"
Sep 10 15:33:03.092494 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:03.092344    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ea32fcf8-df19-4d72-82e9-9d48dcd4701d-tmp\") pod \"openshift-lws-operator-fd8ccff4c-v25lg\" (UID: \"ea32fcf8-df19-4d72-82e9-9d48dcd4701d\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-v25lg"
Sep 10 15:33:03.092738 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:03.092717    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ea32fcf8-df19-4d72-82e9-9d48dcd4701d-tmp\") pod \"openshift-lws-operator-fd8ccff4c-v25lg\" (UID: \"ea32fcf8-df19-4d72-82e9-9d48dcd4701d\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-v25lg"
Sep 10 15:33:03.101748 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:03.101723    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wg5hq\" (UniqueName: \"kubernetes.io/projected/ea32fcf8-df19-4d72-82e9-9d48dcd4701d-kube-api-access-wg5hq\") pod \"openshift-lws-operator-fd8ccff4c-v25lg\" (UID: \"ea32fcf8-df19-4d72-82e9-9d48dcd4701d\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-v25lg"
Sep 10 15:33:03.139235 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:03.139204    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-v25lg"
Sep 10 15:33:03.412128 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:03.412094    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-v25lg"]
Sep 10 15:33:03.415149 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:33:03.415123    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podea32fcf8_df19_4d72_82e9_9d48dcd4701d.slice/crio-c6dec78ef1cba0c3f355063bdbb0240f51e5cebad353b8c98488b1c0fa9061e6 WatchSource:0}: Error finding container c6dec78ef1cba0c3f355063bdbb0240f51e5cebad353b8c98488b1c0fa9061e6: Status 404 returned error can't find the container with id c6dec78ef1cba0c3f355063bdbb0240f51e5cebad353b8c98488b1c0fa9061e6
Sep 10 15:33:04.256324 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:04.256279    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-v25lg" event={"ID":"ea32fcf8-df19-4d72-82e9-9d48dcd4701d","Type":"ContainerStarted","Data":"c6dec78ef1cba0c3f355063bdbb0240f51e5cebad353b8c98488b1c0fa9061e6"}
Sep 10 15:33:05.261515 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:05.261482    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-v25lg" event={"ID":"ea32fcf8-df19-4d72-82e9-9d48dcd4701d","Type":"ContainerStarted","Data":"bc7e095ba68406c0e87586ce23de782abe1c7e3ac41c5a9c75d80476a39dbf87"}
Sep 10 15:33:05.282893 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:05.282837    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-v25lg" podStartSLOduration=1.62862544 podStartE2EDuration="3.282820578s" totalImagesPullingTime="1.654195138s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:33:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:33:05.281843055 +0000 UTC m=+361.023808158" watchObservedRunningTime="2026-09-10 15:33:05.282820578 +0000 UTC m=+361.024785680"
Sep 10 15:33:19.407344 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:19.407303    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b08nxhl"]
Sep 10 15:33:19.409834 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:19.409817    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b08nxhl"
Sep 10 15:33:19.412390 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:19.412366    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 10 15:33:19.413369 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:19.413350    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 10 15:33:19.413471 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:19.413390    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-984bz\""
Sep 10 15:33:19.425113 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:19.425082    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b08nxhl"]
Sep 10 15:33:19.512850 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:19.512803    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbscj\" (UniqueName: \"kubernetes.io/projected/e713b52b-285a-455f-b2b3-4dfd494453e9-kube-api-access-kbscj\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b08nxhl\" (UID: \"e713b52b-285a-455f-b2b3-4dfd494453e9\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b08nxhl"
Sep 10 15:33:19.513027 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:19.512863    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e713b52b-285a-455f-b2b3-4dfd494453e9-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b08nxhl\" (UID: \"e713b52b-285a-455f-b2b3-4dfd494453e9\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b08nxhl"
Sep 10 15:33:19.513027 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:19.512944    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e713b52b-285a-455f-b2b3-4dfd494453e9-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b08nxhl\" (UID: \"e713b52b-285a-455f-b2b3-4dfd494453e9\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b08nxhl"
Sep 10 15:33:19.613545 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:19.613509    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e713b52b-285a-455f-b2b3-4dfd494453e9-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b08nxhl\" (UID: \"e713b52b-285a-455f-b2b3-4dfd494453e9\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b08nxhl"
Sep 10 15:33:19.613741 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:19.613559    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbscj\" (UniqueName: \"kubernetes.io/projected/e713b52b-285a-455f-b2b3-4dfd494453e9-kube-api-access-kbscj\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b08nxhl\" (UID: \"e713b52b-285a-455f-b2b3-4dfd494453e9\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b08nxhl"
Sep 10 15:33:19.613741 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:19.613617    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e713b52b-285a-455f-b2b3-4dfd494453e9-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b08nxhl\" (UID: \"e713b52b-285a-455f-b2b3-4dfd494453e9\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b08nxhl"
Sep 10 15:33:19.614017 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:19.613998    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e713b52b-285a-455f-b2b3-4dfd494453e9-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b08nxhl\" (UID: \"e713b52b-285a-455f-b2b3-4dfd494453e9\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b08nxhl"
Sep 10 15:33:19.614054 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:19.614001    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e713b52b-285a-455f-b2b3-4dfd494453e9-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b08nxhl\" (UID: \"e713b52b-285a-455f-b2b3-4dfd494453e9\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b08nxhl"
Sep 10 15:33:19.622557 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:19.622535    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbscj\" (UniqueName: \"kubernetes.io/projected/e713b52b-285a-455f-b2b3-4dfd494453e9-kube-api-access-kbscj\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b08nxhl\" (UID: \"e713b52b-285a-455f-b2b3-4dfd494453e9\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b08nxhl"
Sep 10 15:33:19.718365 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:19.718324    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b08nxhl"
Sep 10 15:33:19.992187 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:19.992114    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b08nxhl"]
Sep 10 15:33:19.995215 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:33:19.995183    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode713b52b_285a_455f_b2b3_4dfd494453e9.slice/crio-83ce48507a1b9e5454e4d2cd865f7bf79047f5948aac291fba3585066f621906 WatchSource:0}: Error finding container 83ce48507a1b9e5454e4d2cd865f7bf79047f5948aac291fba3585066f621906: Status 404 returned error can't find the container with id 83ce48507a1b9e5454e4d2cd865f7bf79047f5948aac291fba3585066f621906
Sep 10 15:33:20.318414 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:20.318315    2559 generic.go:356] "Generic (PLEG): container finished" podID="e713b52b-285a-455f-b2b3-4dfd494453e9" containerID="29ac89f6605aac1b9a773011ca98d36467c6ca24b725c86b9b685573a2032f59" exitCode=0
Sep 10 15:33:20.318550 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:20.318416    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b08nxhl" event={"ID":"e713b52b-285a-455f-b2b3-4dfd494453e9","Type":"ContainerDied","Data":"29ac89f6605aac1b9a773011ca98d36467c6ca24b725c86b9b685573a2032f59"}
Sep 10 15:33:20.318550 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:20.318463    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b08nxhl" event={"ID":"e713b52b-285a-455f-b2b3-4dfd494453e9","Type":"ContainerStarted","Data":"83ce48507a1b9e5454e4d2cd865f7bf79047f5948aac291fba3585066f621906"}
Sep 10 15:33:21.323143 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:21.323104    2559 generic.go:356] "Generic (PLEG): container finished" podID="e713b52b-285a-455f-b2b3-4dfd494453e9" containerID="d8295f96fb2a085c0d985e9f65d85fd219da302a6bceeb552784fe409895dabd" exitCode=0
Sep 10 15:33:21.323143 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:21.323148    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b08nxhl" event={"ID":"e713b52b-285a-455f-b2b3-4dfd494453e9","Type":"ContainerDied","Data":"d8295f96fb2a085c0d985e9f65d85fd219da302a6bceeb552784fe409895dabd"}
Sep 10 15:33:21.669744 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:21.669712    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-797ffb6d9c-6rqgv"]
Sep 10 15:33:21.671708 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:21.671693    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-797ffb6d9c-6rqgv"
Sep 10 15:33:21.674288 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:21.674261    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 10 15:33:21.674391 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:21.674286    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 10 15:33:21.674391 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:21.674380    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 10 15:33:21.674662 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:21.674644    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 10 15:33:21.674723 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:21.674674    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-f7nx8\""
Sep 10 15:33:21.690644 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:21.690622    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-797ffb6d9c-6rqgv"]
Sep 10 15:33:21.830151 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:21.830120    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/72e35df7-9069-4147-8025-408e7c8d08e1-webhook-cert\") pod \"opendatahub-operator-controller-manager-797ffb6d9c-6rqgv\" (UID: \"72e35df7-9069-4147-8025-408e7c8d08e1\") " pod="opendatahub/opendatahub-operator-controller-manager-797ffb6d9c-6rqgv"
Sep 10 15:33:21.830309 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:21.830158    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/72e35df7-9069-4147-8025-408e7c8d08e1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-797ffb6d9c-6rqgv\" (UID: \"72e35df7-9069-4147-8025-408e7c8d08e1\") " pod="opendatahub/opendatahub-operator-controller-manager-797ffb6d9c-6rqgv"
Sep 10 15:33:21.830309 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:21.830180    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/72e35df7-9069-4147-8025-408e7c8d08e1-manifests\") pod \"opendatahub-operator-controller-manager-797ffb6d9c-6rqgv\" (UID: \"72e35df7-9069-4147-8025-408e7c8d08e1\") " pod="opendatahub/opendatahub-operator-controller-manager-797ffb6d9c-6rqgv"
Sep 10 15:33:21.830309 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:21.830255    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/72e35df7-9069-4147-8025-408e7c8d08e1-tmp\") pod \"opendatahub-operator-controller-manager-797ffb6d9c-6rqgv\" (UID: \"72e35df7-9069-4147-8025-408e7c8d08e1\") " pod="opendatahub/opendatahub-operator-controller-manager-797ffb6d9c-6rqgv"
Sep 10 15:33:21.830309 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:21.830288    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vq844\" (UniqueName: \"kubernetes.io/projected/72e35df7-9069-4147-8025-408e7c8d08e1-kube-api-access-vq844\") pod \"opendatahub-operator-controller-manager-797ffb6d9c-6rqgv\" (UID: \"72e35df7-9069-4147-8025-408e7c8d08e1\") " pod="opendatahub/opendatahub-operator-controller-manager-797ffb6d9c-6rqgv"
Sep 10 15:33:21.930935 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:21.930852    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/72e35df7-9069-4147-8025-408e7c8d08e1-webhook-cert\") pod \"opendatahub-operator-controller-manager-797ffb6d9c-6rqgv\" (UID: \"72e35df7-9069-4147-8025-408e7c8d08e1\") " pod="opendatahub/opendatahub-operator-controller-manager-797ffb6d9c-6rqgv"
Sep 10 15:33:21.930935 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:21.930890    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/72e35df7-9069-4147-8025-408e7c8d08e1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-797ffb6d9c-6rqgv\" (UID: \"72e35df7-9069-4147-8025-408e7c8d08e1\") " pod="opendatahub/opendatahub-operator-controller-manager-797ffb6d9c-6rqgv"
Sep 10 15:33:21.930935 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:21.930912    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/72e35df7-9069-4147-8025-408e7c8d08e1-manifests\") pod \"opendatahub-operator-controller-manager-797ffb6d9c-6rqgv\" (UID: \"72e35df7-9069-4147-8025-408e7c8d08e1\") " pod="opendatahub/opendatahub-operator-controller-manager-797ffb6d9c-6rqgv"
Sep 10 15:33:21.931169 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:21.930949    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/72e35df7-9069-4147-8025-408e7c8d08e1-tmp\") pod \"opendatahub-operator-controller-manager-797ffb6d9c-6rqgv\" (UID: \"72e35df7-9069-4147-8025-408e7c8d08e1\") " pod="opendatahub/opendatahub-operator-controller-manager-797ffb6d9c-6rqgv"
Sep 10 15:33:21.931169 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:21.930973    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq844\" (UniqueName: \"kubernetes.io/projected/72e35df7-9069-4147-8025-408e7c8d08e1-kube-api-access-vq844\") pod \"opendatahub-operator-controller-manager-797ffb6d9c-6rqgv\" (UID: \"72e35df7-9069-4147-8025-408e7c8d08e1\") " pod="opendatahub/opendatahub-operator-controller-manager-797ffb6d9c-6rqgv"
Sep 10 15:33:21.931348 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:21.931327    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/72e35df7-9069-4147-8025-408e7c8d08e1-manifests\") pod \"opendatahub-operator-controller-manager-797ffb6d9c-6rqgv\" (UID: \"72e35df7-9069-4147-8025-408e7c8d08e1\") " pod="opendatahub/opendatahub-operator-controller-manager-797ffb6d9c-6rqgv"
Sep 10 15:33:21.933189 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:21.933166    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/72e35df7-9069-4147-8025-408e7c8d08e1-tmp\") pod \"opendatahub-operator-controller-manager-797ffb6d9c-6rqgv\" (UID: \"72e35df7-9069-4147-8025-408e7c8d08e1\") " pod="opendatahub/opendatahub-operator-controller-manager-797ffb6d9c-6rqgv"
Sep 10 15:33:21.933327 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:21.933308    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/72e35df7-9069-4147-8025-408e7c8d08e1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-797ffb6d9c-6rqgv\" (UID: \"72e35df7-9069-4147-8025-408e7c8d08e1\") " pod="opendatahub/opendatahub-operator-controller-manager-797ffb6d9c-6rqgv"
Sep 10 15:33:21.933374 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:21.933360    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/72e35df7-9069-4147-8025-408e7c8d08e1-webhook-cert\") pod \"opendatahub-operator-controller-manager-797ffb6d9c-6rqgv\" (UID: \"72e35df7-9069-4147-8025-408e7c8d08e1\") " pod="opendatahub/opendatahub-operator-controller-manager-797ffb6d9c-6rqgv"
Sep 10 15:33:21.945033 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:21.945006    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq844\" (UniqueName: \"kubernetes.io/projected/72e35df7-9069-4147-8025-408e7c8d08e1-kube-api-access-vq844\") pod \"opendatahub-operator-controller-manager-797ffb6d9c-6rqgv\" (UID: \"72e35df7-9069-4147-8025-408e7c8d08e1\") " pod="opendatahub/opendatahub-operator-controller-manager-797ffb6d9c-6rqgv"
Sep 10 15:33:21.981046 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:21.981015    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-797ffb6d9c-6rqgv"
Sep 10 15:33:22.261995 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:22.261970    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-797ffb6d9c-6rqgv"]
Sep 10 15:33:22.263554 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:33:22.263530    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72e35df7_9069_4147_8025_408e7c8d08e1.slice/crio-e2c2c7ffe021b1c7345e1ef532a8cfdc5632856ba6b051dba02a738d9a79c241 WatchSource:0}: Error finding container e2c2c7ffe021b1c7345e1ef532a8cfdc5632856ba6b051dba02a738d9a79c241: Status 404 returned error can't find the container with id e2c2c7ffe021b1c7345e1ef532a8cfdc5632856ba6b051dba02a738d9a79c241
Sep 10 15:33:22.329162 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:22.329124    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-797ffb6d9c-6rqgv" event={"ID":"72e35df7-9069-4147-8025-408e7c8d08e1","Type":"ContainerStarted","Data":"e2c2c7ffe021b1c7345e1ef532a8cfdc5632856ba6b051dba02a738d9a79c241"}
Sep 10 15:33:22.331168 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:22.331144    2559 generic.go:356] "Generic (PLEG): container finished" podID="e713b52b-285a-455f-b2b3-4dfd494453e9" containerID="60d6feab46b17276a14c7ec117125063ac81d9db2a9aa0da544f6a5c6d70423d" exitCode=0
Sep 10 15:33:22.331289 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:22.331208    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b08nxhl" event={"ID":"e713b52b-285a-455f-b2b3-4dfd494453e9","Type":"ContainerDied","Data":"60d6feab46b17276a14c7ec117125063ac81d9db2a9aa0da544f6a5c6d70423d"}
Sep 10 15:33:23.523952 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:23.523917    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b08nxhl"
Sep 10 15:33:23.645369 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:23.644798    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e713b52b-285a-455f-b2b3-4dfd494453e9-util\") pod \"e713b52b-285a-455f-b2b3-4dfd494453e9\" (UID: \"e713b52b-285a-455f-b2b3-4dfd494453e9\") "
Sep 10 15:33:23.645369 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:23.644872    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kbscj\" (UniqueName: \"kubernetes.io/projected/e713b52b-285a-455f-b2b3-4dfd494453e9-kube-api-access-kbscj\") pod \"e713b52b-285a-455f-b2b3-4dfd494453e9\" (UID: \"e713b52b-285a-455f-b2b3-4dfd494453e9\") "
Sep 10 15:33:23.645369 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:23.644905    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e713b52b-285a-455f-b2b3-4dfd494453e9-bundle\") pod \"e713b52b-285a-455f-b2b3-4dfd494453e9\" (UID: \"e713b52b-285a-455f-b2b3-4dfd494453e9\") "
Sep 10 15:33:23.646711 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:23.646665    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e713b52b-285a-455f-b2b3-4dfd494453e9-bundle" (OuterVolumeSpecName: "bundle") pod "e713b52b-285a-455f-b2b3-4dfd494453e9" (UID: "e713b52b-285a-455f-b2b3-4dfd494453e9"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:33:23.648602 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:23.648540    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e713b52b-285a-455f-b2b3-4dfd494453e9-kube-api-access-kbscj" (OuterVolumeSpecName: "kube-api-access-kbscj") pod "e713b52b-285a-455f-b2b3-4dfd494453e9" (UID: "e713b52b-285a-455f-b2b3-4dfd494453e9"). InnerVolumeSpecName "kube-api-access-kbscj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:33:23.653249 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:23.653221    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e713b52b-285a-455f-b2b3-4dfd494453e9-util" (OuterVolumeSpecName: "util") pod "e713b52b-285a-455f-b2b3-4dfd494453e9" (UID: "e713b52b-285a-455f-b2b3-4dfd494453e9"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:33:23.746402 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:23.746361    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbscj\" (UniqueName: \"kubernetes.io/projected/e713b52b-285a-455f-b2b3-4dfd494453e9-kube-api-access-kbscj\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:33:23.746402 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:23.746397    2559 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e713b52b-285a-455f-b2b3-4dfd494453e9-bundle\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:33:23.746402 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:23.746413    2559 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e713b52b-285a-455f-b2b3-4dfd494453e9-util\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:33:24.342567 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:24.342541    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b08nxhl"
Sep 10 15:33:24.342567 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:24.342546    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b08nxhl" event={"ID":"e713b52b-285a-455f-b2b3-4dfd494453e9","Type":"ContainerDied","Data":"83ce48507a1b9e5454e4d2cd865f7bf79047f5948aac291fba3585066f621906"}
Sep 10 15:33:24.342775 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:24.342602    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="83ce48507a1b9e5454e4d2cd865f7bf79047f5948aac291fba3585066f621906"
Sep 10 15:33:26.351822 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:26.351788    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-797ffb6d9c-6rqgv" event={"ID":"72e35df7-9069-4147-8025-408e7c8d08e1","Type":"ContainerStarted","Data":"6d96b712467d04ed0f6177c44cf93d45ccf7d6131ab9d2c759fc8ee823065bda"}
Sep 10 15:33:27.357605 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:27.357491    2559 generic.go:356] "Generic (PLEG): container finished" podID="72e35df7-9069-4147-8025-408e7c8d08e1" containerID="6d96b712467d04ed0f6177c44cf93d45ccf7d6131ab9d2c759fc8ee823065bda" exitCode=0
Sep 10 15:33:27.357605 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:27.357573    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-797ffb6d9c-6rqgv" event={"ID":"72e35df7-9069-4147-8025-408e7c8d08e1","Type":"ContainerDied","Data":"6d96b712467d04ed0f6177c44cf93d45ccf7d6131ab9d2c759fc8ee823065bda"}
Sep 10 15:33:28.363370 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:28.363331    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-797ffb6d9c-6rqgv" event={"ID":"72e35df7-9069-4147-8025-408e7c8d08e1","Type":"ContainerStarted","Data":"06ffef484129369ba051ac447aa85cf8764fd65b1bc368a25c2559c07b11b1f1"}
Sep 10 15:33:28.363785 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:28.363451    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-797ffb6d9c-6rqgv"
Sep 10 15:33:28.398327 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:28.398280    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-797ffb6d9c-6rqgv" podStartSLOduration=2.516991128 podStartE2EDuration="7.39826412s" totalImagesPullingTime="3.859638109s" totalInitContainerRuntime="1.021634883s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:33:21 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:33:28.396920042 +0000 UTC m=+384.138885146" watchObservedRunningTime="2026-09-10 15:33:28.39826412 +0000 UTC m=+384.140229223"
Sep 10 15:33:39.370049 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:39.370017    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-797ffb6d9c-6rqgv"
Sep 10 15:33:41.577908 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:41.577873    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-785ff899f9-wx2md"]
Sep 10 15:33:41.578289 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:41.578136    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e713b52b-285a-455f-b2b3-4dfd494453e9" containerName="pull"
Sep 10 15:33:41.578289 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:41.578146    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e713b52b-285a-455f-b2b3-4dfd494453e9" containerName="pull"
Sep 10 15:33:41.578289 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:41.578158    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e713b52b-285a-455f-b2b3-4dfd494453e9" containerName="extract"
Sep 10 15:33:41.578289 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:41.578163    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e713b52b-285a-455f-b2b3-4dfd494453e9" containerName="extract"
Sep 10 15:33:41.578289 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:41.578172    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e713b52b-285a-455f-b2b3-4dfd494453e9" containerName="util"
Sep 10 15:33:41.578289 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:41.578178    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e713b52b-285a-455f-b2b3-4dfd494453e9" containerName="util"
Sep 10 15:33:41.578289 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:41.578225    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="e713b52b-285a-455f-b2b3-4dfd494453e9" containerName="extract"
Sep 10 15:33:41.587515 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:41.587493    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-785ff899f9-wx2md"
Sep 10 15:33:41.591236 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:41.591209    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-dmj8w\""
Sep 10 15:33:41.591673 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:41.591653    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 10 15:33:41.592287 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:41.592261    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 10 15:33:41.592287 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:41.592263    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 10 15:33:41.592396 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:41.592352    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 10 15:33:41.606673 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:41.606650    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-785ff899f9-wx2md"]
Sep 10 15:33:41.679095 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:41.679057    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c775c3c-0dff-4fbd-bb68-67e4fcf59820-tls-certs\") pod \"kube-auth-proxy-785ff899f9-wx2md\" (UID: \"2c775c3c-0dff-4fbd-bb68-67e4fcf59820\") " pod="openshift-ingress/kube-auth-proxy-785ff899f9-wx2md"
Sep 10 15:33:41.679095 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:41.679099    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2c775c3c-0dff-4fbd-bb68-67e4fcf59820-tmp\") pod \"kube-auth-proxy-785ff899f9-wx2md\" (UID: \"2c775c3c-0dff-4fbd-bb68-67e4fcf59820\") " pod="openshift-ingress/kube-auth-proxy-785ff899f9-wx2md"
Sep 10 15:33:41.679292 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:41.679124    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjp4j\" (UniqueName: \"kubernetes.io/projected/2c775c3c-0dff-4fbd-bb68-67e4fcf59820-kube-api-access-qjp4j\") pod \"kube-auth-proxy-785ff899f9-wx2md\" (UID: \"2c775c3c-0dff-4fbd-bb68-67e4fcf59820\") " pod="openshift-ingress/kube-auth-proxy-785ff899f9-wx2md"
Sep 10 15:33:41.780352 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:41.780306    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c775c3c-0dff-4fbd-bb68-67e4fcf59820-tls-certs\") pod \"kube-auth-proxy-785ff899f9-wx2md\" (UID: \"2c775c3c-0dff-4fbd-bb68-67e4fcf59820\") " pod="openshift-ingress/kube-auth-proxy-785ff899f9-wx2md"
Sep 10 15:33:41.780493 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:41.780430    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2c775c3c-0dff-4fbd-bb68-67e4fcf59820-tmp\") pod \"kube-auth-proxy-785ff899f9-wx2md\" (UID: \"2c775c3c-0dff-4fbd-bb68-67e4fcf59820\") " pod="openshift-ingress/kube-auth-proxy-785ff899f9-wx2md"
Sep 10 15:33:41.780493 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:41.780483    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjp4j\" (UniqueName: \"kubernetes.io/projected/2c775c3c-0dff-4fbd-bb68-67e4fcf59820-kube-api-access-qjp4j\") pod \"kube-auth-proxy-785ff899f9-wx2md\" (UID: \"2c775c3c-0dff-4fbd-bb68-67e4fcf59820\") " pod="openshift-ingress/kube-auth-proxy-785ff899f9-wx2md"
Sep 10 15:33:41.782895 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:41.782868    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c775c3c-0dff-4fbd-bb68-67e4fcf59820-tls-certs\") pod \"kube-auth-proxy-785ff899f9-wx2md\" (UID: \"2c775c3c-0dff-4fbd-bb68-67e4fcf59820\") " pod="openshift-ingress/kube-auth-proxy-785ff899f9-wx2md"
Sep 10 15:33:41.783154 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:41.783136    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2c775c3c-0dff-4fbd-bb68-67e4fcf59820-tmp\") pod \"kube-auth-proxy-785ff899f9-wx2md\" (UID: \"2c775c3c-0dff-4fbd-bb68-67e4fcf59820\") " pod="openshift-ingress/kube-auth-proxy-785ff899f9-wx2md"
Sep 10 15:33:41.805389 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:41.805363    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjp4j\" (UniqueName: \"kubernetes.io/projected/2c775c3c-0dff-4fbd-bb68-67e4fcf59820-kube-api-access-qjp4j\") pod \"kube-auth-proxy-785ff899f9-wx2md\" (UID: \"2c775c3c-0dff-4fbd-bb68-67e4fcf59820\") " pod="openshift-ingress/kube-auth-proxy-785ff899f9-wx2md"
Sep 10 15:33:41.896845 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:41.896761    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-785ff899f9-wx2md"
Sep 10 15:33:42.178550 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:42.178524    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-785ff899f9-wx2md"]
Sep 10 15:33:42.181012 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:33:42.180986    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c775c3c_0dff_4fbd_bb68_67e4fcf59820.slice/crio-f2f22daeed5437826556f8d6a3b58515c42ed3d0059d404939fa93755ece206c WatchSource:0}: Error finding container f2f22daeed5437826556f8d6a3b58515c42ed3d0059d404939fa93755ece206c: Status 404 returned error can't find the container with id f2f22daeed5437826556f8d6a3b58515c42ed3d0059d404939fa93755ece206c
Sep 10 15:33:42.423793 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:42.423757    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-785ff899f9-wx2md" event={"ID":"2c775c3c-0dff-4fbd-bb68-67e4fcf59820","Type":"ContainerStarted","Data":"f2f22daeed5437826556f8d6a3b58515c42ed3d0059d404939fa93755ece206c"}
Sep 10 15:33:43.131765 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:43.131726    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bzt7g"]
Sep 10 15:33:43.135334 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:43.135311    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bzt7g"
Sep 10 15:33:43.138091 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:43.138068    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 10 15:33:43.138195 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:43.138108    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 10 15:33:43.138195 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:43.138136    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-4m6v8\""
Sep 10 15:33:43.138195 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:43.138161    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 10 15:33:43.138365 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:43.138069    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 10 15:33:43.152769 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:43.152743    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bzt7g"]
Sep 10 15:33:43.188603 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:43.188557    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a0282906-4494-4b48-821d-ebe85d8c12cb-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-bzt7g\" (UID: \"a0282906-4494-4b48-821d-ebe85d8c12cb\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bzt7g"
Sep 10 15:33:43.188603 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:43.188608    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a0282906-4494-4b48-821d-ebe85d8c12cb-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-bzt7g\" (UID: \"a0282906-4494-4b48-821d-ebe85d8c12cb\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bzt7g"
Sep 10 15:33:43.188815 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:43.188632    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0282906-4494-4b48-821d-ebe85d8c12cb-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-bzt7g\" (UID: \"a0282906-4494-4b48-821d-ebe85d8c12cb\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bzt7g"
Sep 10 15:33:43.188815 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:43.188674    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0282906-4494-4b48-821d-ebe85d8c12cb-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-bzt7g\" (UID: \"a0282906-4494-4b48-821d-ebe85d8c12cb\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bzt7g"
Sep 10 15:33:43.188815 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:43.188703    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a0282906-4494-4b48-821d-ebe85d8c12cb-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-bzt7g\" (UID: \"a0282906-4494-4b48-821d-ebe85d8c12cb\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bzt7g"
Sep 10 15:33:43.188815 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:43.188768    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a0282906-4494-4b48-821d-ebe85d8c12cb-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-bzt7g\" (UID: \"a0282906-4494-4b48-821d-ebe85d8c12cb\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bzt7g"
Sep 10 15:33:43.188815 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:43.188792    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zn2jj\" (UniqueName: \"kubernetes.io/projected/a0282906-4494-4b48-821d-ebe85d8c12cb-kube-api-access-zn2jj\") pod \"istiod-openshift-gateway-7b7995d5cd-bzt7g\" (UID: \"a0282906-4494-4b48-821d-ebe85d8c12cb\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bzt7g"
Sep 10 15:33:43.289302 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:43.289257    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0282906-4494-4b48-821d-ebe85d8c12cb-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-bzt7g\" (UID: \"a0282906-4494-4b48-821d-ebe85d8c12cb\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bzt7g"
Sep 10 15:33:43.289496 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:43.289319    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a0282906-4494-4b48-821d-ebe85d8c12cb-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-bzt7g\" (UID: \"a0282906-4494-4b48-821d-ebe85d8c12cb\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bzt7g"
Sep 10 15:33:43.289496 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:43.289362    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a0282906-4494-4b48-821d-ebe85d8c12cb-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-bzt7g\" (UID: \"a0282906-4494-4b48-821d-ebe85d8c12cb\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bzt7g"
Sep 10 15:33:43.289496 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:43.289379    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zn2jj\" (UniqueName: \"kubernetes.io/projected/a0282906-4494-4b48-821d-ebe85d8c12cb-kube-api-access-zn2jj\") pod \"istiod-openshift-gateway-7b7995d5cd-bzt7g\" (UID: \"a0282906-4494-4b48-821d-ebe85d8c12cb\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bzt7g"
Sep 10 15:33:43.289496 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:43.289444    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a0282906-4494-4b48-821d-ebe85d8c12cb-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-bzt7g\" (UID: \"a0282906-4494-4b48-821d-ebe85d8c12cb\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bzt7g"
Sep 10 15:33:43.289496 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:43.289467    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a0282906-4494-4b48-821d-ebe85d8c12cb-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-bzt7g\" (UID: \"a0282906-4494-4b48-821d-ebe85d8c12cb\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bzt7g"
Sep 10 15:33:43.289496 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:43.289491    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0282906-4494-4b48-821d-ebe85d8c12cb-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-bzt7g\" (UID: \"a0282906-4494-4b48-821d-ebe85d8c12cb\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bzt7g"
Sep 10 15:33:43.290345 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:43.290316    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a0282906-4494-4b48-821d-ebe85d8c12cb-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-bzt7g\" (UID: \"a0282906-4494-4b48-821d-ebe85d8c12cb\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bzt7g"
Sep 10 15:33:43.292848 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:43.292738    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a0282906-4494-4b48-821d-ebe85d8c12cb-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-bzt7g\" (UID: \"a0282906-4494-4b48-821d-ebe85d8c12cb\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bzt7g"
Sep 10 15:33:43.292970 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:43.292946    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0282906-4494-4b48-821d-ebe85d8c12cb-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-bzt7g\" (UID: \"a0282906-4494-4b48-821d-ebe85d8c12cb\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bzt7g"
Sep 10 15:33:43.293099 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:43.293028    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a0282906-4494-4b48-821d-ebe85d8c12cb-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-bzt7g\" (UID: \"a0282906-4494-4b48-821d-ebe85d8c12cb\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bzt7g"
Sep 10 15:33:43.293210 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:43.293190    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a0282906-4494-4b48-821d-ebe85d8c12cb-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-bzt7g\" (UID: \"a0282906-4494-4b48-821d-ebe85d8c12cb\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bzt7g"
Sep 10 15:33:43.301592 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:43.301555    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zn2jj\" (UniqueName: \"kubernetes.io/projected/a0282906-4494-4b48-821d-ebe85d8c12cb-kube-api-access-zn2jj\") pod \"istiod-openshift-gateway-7b7995d5cd-bzt7g\" (UID: \"a0282906-4494-4b48-821d-ebe85d8c12cb\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bzt7g"
Sep 10 15:33:43.301729 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:43.301709    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0282906-4494-4b48-821d-ebe85d8c12cb-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-bzt7g\" (UID: \"a0282906-4494-4b48-821d-ebe85d8c12cb\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bzt7g"
Sep 10 15:33:43.447669 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:43.447514    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bzt7g"
Sep 10 15:33:43.753098 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:43.753048    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bzt7g"]
Sep 10 15:33:44.989384 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:33:44.989345    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0282906_4494_4b48_821d_ebe85d8c12cb.slice/crio-e3185b04d3f6cb6197281043dbf8efa21c52abddcb16eabb97c73c0d9b08707a WatchSource:0}: Error finding container e3185b04d3f6cb6197281043dbf8efa21c52abddcb16eabb97c73c0d9b08707a: Status 404 returned error can't find the container with id e3185b04d3f6cb6197281043dbf8efa21c52abddcb16eabb97c73c0d9b08707a
Sep 10 15:33:45.440360 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:45.440326    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-785ff899f9-wx2md" event={"ID":"2c775c3c-0dff-4fbd-bb68-67e4fcf59820","Type":"ContainerStarted","Data":"5b127429be87c205825b10fed8a979d69c82f619564beaa6cd07e91ff4a8ad6d"}
Sep 10 15:33:45.441645 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:45.441625    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bzt7g" event={"ID":"a0282906-4494-4b48-821d-ebe85d8c12cb","Type":"ContainerStarted","Data":"e3185b04d3f6cb6197281043dbf8efa21c52abddcb16eabb97c73c0d9b08707a"}
Sep 10 15:33:45.460585 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:45.460534    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-785ff899f9-wx2md" podStartSLOduration=1.601755743 podStartE2EDuration="4.460517715s" totalImagesPullingTime="2.858761972s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:33:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:33:45.460383671 +0000 UTC m=+401.202348775" watchObservedRunningTime="2026-09-10 15:33:45.460517715 +0000 UTC m=+401.202482819"
Sep 10 15:33:48.372340 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:48.372299    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 15:33:48.372608 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:48.372381    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 15:33:48.458020 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:48.457990    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bzt7g" event={"ID":"a0282906-4494-4b48-821d-ebe85d8c12cb","Type":"ContainerStarted","Data":"1c6e89ff6907100c63ec978b548c80f5e4d31f9969080ce44e3e1d8d70cd927a"}
Sep 10 15:33:48.458173 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:48.458069    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bzt7g"
Sep 10 15:33:48.485698 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:48.485635    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bzt7g" podStartSLOduration=2.104979651 podStartE2EDuration="5.485614382s" totalImagesPullingTime="3.380634731s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:33:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:33:48.48260306 +0000 UTC m=+404.224568154" watchObservedRunningTime="2026-09-10 15:33:48.485614382 +0000 UTC m=+404.227579487"
Sep 10 15:33:49.464393 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:49.464358    2559 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-bzt7g container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 10 15:33:49.464814 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:49.464414    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bzt7g" podUID="a0282906-4494-4b48-821d-ebe85d8c12cb" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 10 15:33:52.464154 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:52.464119    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bzt7g"
Sep 10 15:33:52.935191 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:52.935157    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-wffnd"]
Sep 10 15:33:52.938542 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:52.938526    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-wffnd"
Sep 10 15:33:52.941102 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:52.941083    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Sep 10 15:33:52.953054 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:52.953029    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-wffnd"]
Sep 10 15:33:52.959721 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:52.959695    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gnhw\" (UniqueName: \"kubernetes.io/projected/fab62d93-c65f-4c84-92cf-8923ebd51112-kube-api-access-5gnhw\") pod \"model-serving-api-54d89bd5fc-wffnd\" (UID: \"fab62d93-c65f-4c84-92cf-8923ebd51112\") " pod="opendatahub/model-serving-api-54d89bd5fc-wffnd"
Sep 10 15:33:52.959805 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:52.959758    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fab62d93-c65f-4c84-92cf-8923ebd51112-tls-certs\") pod \"model-serving-api-54d89bd5fc-wffnd\" (UID: \"fab62d93-c65f-4c84-92cf-8923ebd51112\") " pod="opendatahub/model-serving-api-54d89bd5fc-wffnd"
Sep 10 15:33:53.060242 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:53.060209    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gnhw\" (UniqueName: \"kubernetes.io/projected/fab62d93-c65f-4c84-92cf-8923ebd51112-kube-api-access-5gnhw\") pod \"model-serving-api-54d89bd5fc-wffnd\" (UID: \"fab62d93-c65f-4c84-92cf-8923ebd51112\") " pod="opendatahub/model-serving-api-54d89bd5fc-wffnd"
Sep 10 15:33:53.060404 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:53.060262    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fab62d93-c65f-4c84-92cf-8923ebd51112-tls-certs\") pod \"model-serving-api-54d89bd5fc-wffnd\" (UID: \"fab62d93-c65f-4c84-92cf-8923ebd51112\") " pod="opendatahub/model-serving-api-54d89bd5fc-wffnd"
Sep 10 15:33:53.060404 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:33:53.060358    2559 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found
Sep 10 15:33:53.060477 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:33:53.060417    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fab62d93-c65f-4c84-92cf-8923ebd51112-tls-certs podName:fab62d93-c65f-4c84-92cf-8923ebd51112 nodeName:}" failed. No retries permitted until 2026-09-10 15:33:53.560397749 +0000 UTC m=+409.302362843 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/fab62d93-c65f-4c84-92cf-8923ebd51112-tls-certs") pod "model-serving-api-54d89bd5fc-wffnd" (UID: "fab62d93-c65f-4c84-92cf-8923ebd51112") : secret "model-serving-api-tls" not found
Sep 10 15:33:53.070509 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:53.070480    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gnhw\" (UniqueName: \"kubernetes.io/projected/fab62d93-c65f-4c84-92cf-8923ebd51112-kube-api-access-5gnhw\") pod \"model-serving-api-54d89bd5fc-wffnd\" (UID: \"fab62d93-c65f-4c84-92cf-8923ebd51112\") " pod="opendatahub/model-serving-api-54d89bd5fc-wffnd"
Sep 10 15:33:53.563401 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:53.563367    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fab62d93-c65f-4c84-92cf-8923ebd51112-tls-certs\") pod \"model-serving-api-54d89bd5fc-wffnd\" (UID: \"fab62d93-c65f-4c84-92cf-8923ebd51112\") " pod="opendatahub/model-serving-api-54d89bd5fc-wffnd"
Sep 10 15:33:53.565800 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:53.565776    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fab62d93-c65f-4c84-92cf-8923ebd51112-tls-certs\") pod \"model-serving-api-54d89bd5fc-wffnd\" (UID: \"fab62d93-c65f-4c84-92cf-8923ebd51112\") " pod="opendatahub/model-serving-api-54d89bd5fc-wffnd"
Sep 10 15:33:53.848738 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:53.848652    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-wffnd"
Sep 10 15:33:54.124485 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:54.124400    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-wffnd"]
Sep 10 15:33:54.127227 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:33:54.127200    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfab62d93_c65f_4c84_92cf_8923ebd51112.slice/crio-7fa4448a41fa0774d535caf113f54e53713633ac7795df7f18cd74b67faf4cc0 WatchSource:0}: Error finding container 7fa4448a41fa0774d535caf113f54e53713633ac7795df7f18cd74b67faf4cc0: Status 404 returned error can't find the container with id 7fa4448a41fa0774d535caf113f54e53713633ac7795df7f18cd74b67faf4cc0
Sep 10 15:33:54.483620 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:54.483566    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-wffnd" event={"ID":"fab62d93-c65f-4c84-92cf-8923ebd51112","Type":"ContainerStarted","Data":"7fa4448a41fa0774d535caf113f54e53713633ac7795df7f18cd74b67faf4cc0"}
Sep 10 15:33:56.494059 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:56.494022    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-wffnd" event={"ID":"fab62d93-c65f-4c84-92cf-8923ebd51112","Type":"ContainerStarted","Data":"5eff76bae850bae23d77c937d93bef50b15c51cc8564d92309921f8247893207"}
Sep 10 15:33:56.494462 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:56.494139    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-wffnd"
Sep 10 15:33:56.515656 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:33:56.515608    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-wffnd" podStartSLOduration=2.985936704 podStartE2EDuration="4.515593302s" totalImagesPullingTime="1.529656598s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:33:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:33:56.514701827 +0000 UTC m=+412.256666941" watchObservedRunningTime="2026-09-10 15:33:56.515593302 +0000 UTC m=+412.257558396"
Sep 10 15:34:01.945773 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:01.945736    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-pw7kv"]
Sep 10 15:34:01.952061 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:01.952040    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pw7kv"
Sep 10 15:34:01.955212 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:01.955183    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Sep 10 15:34:01.958778 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:01.958357    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-s2djd\""
Sep 10 15:34:01.977644 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:01.977615    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-pw7kv"]
Sep 10 15:34:02.018188 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:02.018126    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1315ff10-e055-458c-9e83-7199ef2c576a-cert\") pod \"kserve-controller-manager-d6f4b65b8-pw7kv\" (UID: \"1315ff10-e055-458c-9e83-7199ef2c576a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pw7kv"
Sep 10 15:34:02.018395 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:02.018268    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vr6gl\" (UniqueName: \"kubernetes.io/projected/1315ff10-e055-458c-9e83-7199ef2c576a-kube-api-access-vr6gl\") pod \"kserve-controller-manager-d6f4b65b8-pw7kv\" (UID: \"1315ff10-e055-458c-9e83-7199ef2c576a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pw7kv"
Sep 10 15:34:02.039366 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:02.039329    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-6w8hg"]
Sep 10 15:34:02.041664 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:02.041647    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6w8hg"
Sep 10 15:34:02.044886 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:02.044868    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-kxf4v\""
Sep 10 15:34:02.044993 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:02.044886    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Sep 10 15:34:02.058500 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:02.058469    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-6w8hg"]
Sep 10 15:34:02.119519 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:02.119478    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vr6gl\" (UniqueName: \"kubernetes.io/projected/1315ff10-e055-458c-9e83-7199ef2c576a-kube-api-access-vr6gl\") pod \"kserve-controller-manager-d6f4b65b8-pw7kv\" (UID: \"1315ff10-e055-458c-9e83-7199ef2c576a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pw7kv"
Sep 10 15:34:02.119717 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:02.119540    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1315ff10-e055-458c-9e83-7199ef2c576a-cert\") pod \"kserve-controller-manager-d6f4b65b8-pw7kv\" (UID: \"1315ff10-e055-458c-9e83-7199ef2c576a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pw7kv"
Sep 10 15:34:02.119717 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:02.119616    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/49ace7c7-c5ee-46fc-8f39-3334100ea261-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-6w8hg\" (UID: \"49ace7c7-c5ee-46fc-8f39-3334100ea261\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6w8hg"
Sep 10 15:34:02.119717 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:02.119650    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwv8l\" (UniqueName: \"kubernetes.io/projected/49ace7c7-c5ee-46fc-8f39-3334100ea261-kube-api-access-dwv8l\") pod \"llmisvc-controller-manager-777c6bd7d5-6w8hg\" (UID: \"49ace7c7-c5ee-46fc-8f39-3334100ea261\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6w8hg"
Sep 10 15:34:02.119830 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:34:02.119758    2559 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Sep 10 15:34:02.119830 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:34:02.119814    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1315ff10-e055-458c-9e83-7199ef2c576a-cert podName:1315ff10-e055-458c-9e83-7199ef2c576a nodeName:}" failed. No retries permitted until 2026-09-10 15:34:02.619796463 +0000 UTC m=+418.361761549 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/1315ff10-e055-458c-9e83-7199ef2c576a-cert") pod "kserve-controller-manager-d6f4b65b8-pw7kv" (UID: "1315ff10-e055-458c-9e83-7199ef2c576a") : secret "kserve-webhook-server-cert" not found
Sep 10 15:34:02.135999 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:02.135972    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vr6gl\" (UniqueName: \"kubernetes.io/projected/1315ff10-e055-458c-9e83-7199ef2c576a-kube-api-access-vr6gl\") pod \"kserve-controller-manager-d6f4b65b8-pw7kv\" (UID: \"1315ff10-e055-458c-9e83-7199ef2c576a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pw7kv"
Sep 10 15:34:02.220734 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:02.220642    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/49ace7c7-c5ee-46fc-8f39-3334100ea261-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-6w8hg\" (UID: \"49ace7c7-c5ee-46fc-8f39-3334100ea261\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6w8hg"
Sep 10 15:34:02.220734 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:02.220685    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwv8l\" (UniqueName: \"kubernetes.io/projected/49ace7c7-c5ee-46fc-8f39-3334100ea261-kube-api-access-dwv8l\") pod \"llmisvc-controller-manager-777c6bd7d5-6w8hg\" (UID: \"49ace7c7-c5ee-46fc-8f39-3334100ea261\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6w8hg"
Sep 10 15:34:02.223048 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:02.223024    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/49ace7c7-c5ee-46fc-8f39-3334100ea261-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-6w8hg\" (UID: \"49ace7c7-c5ee-46fc-8f39-3334100ea261\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6w8hg"
Sep 10 15:34:02.230888 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:02.230859    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwv8l\" (UniqueName: \"kubernetes.io/projected/49ace7c7-c5ee-46fc-8f39-3334100ea261-kube-api-access-dwv8l\") pod \"llmisvc-controller-manager-777c6bd7d5-6w8hg\" (UID: \"49ace7c7-c5ee-46fc-8f39-3334100ea261\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6w8hg"
Sep 10 15:34:02.350965 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:02.350927    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6w8hg"
Sep 10 15:34:02.624773 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:02.624658    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1315ff10-e055-458c-9e83-7199ef2c576a-cert\") pod \"kserve-controller-manager-d6f4b65b8-pw7kv\" (UID: \"1315ff10-e055-458c-9e83-7199ef2c576a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pw7kv"
Sep 10 15:34:02.627677 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:02.627658    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1315ff10-e055-458c-9e83-7199ef2c576a-cert\") pod \"kserve-controller-manager-d6f4b65b8-pw7kv\" (UID: \"1315ff10-e055-458c-9e83-7199ef2c576a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pw7kv"
Sep 10 15:34:02.627935 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:02.627913    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-6w8hg"]
Sep 10 15:34:02.629539 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:34:02.629521    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod49ace7c7_c5ee_46fc_8f39_3334100ea261.slice/crio-d4b54bbdb5c593bbd094e755657aec944a46e42d1681dbf4b68075a4be9d8137 WatchSource:0}: Error finding container d4b54bbdb5c593bbd094e755657aec944a46e42d1681dbf4b68075a4be9d8137: Status 404 returned error can't find the container with id d4b54bbdb5c593bbd094e755657aec944a46e42d1681dbf4b68075a4be9d8137
Sep 10 15:34:02.861890 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:02.861851    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pw7kv"
Sep 10 15:34:03.142110 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:03.142081    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-pw7kv"]
Sep 10 15:34:03.143331 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:34:03.143305    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1315ff10_e055_458c_9e83_7199ef2c576a.slice/crio-554f6b8191d52ee8d387e5407b94e57e8877e6b5eddbb1a8b7dca060aaf368fb WatchSource:0}: Error finding container 554f6b8191d52ee8d387e5407b94e57e8877e6b5eddbb1a8b7dca060aaf368fb: Status 404 returned error can't find the container with id 554f6b8191d52ee8d387e5407b94e57e8877e6b5eddbb1a8b7dca060aaf368fb
Sep 10 15:34:03.525480 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:03.525442    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6w8hg" event={"ID":"49ace7c7-c5ee-46fc-8f39-3334100ea261","Type":"ContainerStarted","Data":"d4b54bbdb5c593bbd094e755657aec944a46e42d1681dbf4b68075a4be9d8137"}
Sep 10 15:34:03.526636 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:03.526609    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pw7kv" event={"ID":"1315ff10-e055-458c-9e83-7199ef2c576a","Type":"ContainerStarted","Data":"554f6b8191d52ee8d387e5407b94e57e8877e6b5eddbb1a8b7dca060aaf368fb"}
Sep 10 15:34:03.791355 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:03.791203    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bzt7g"]
Sep 10 15:34:03.791516 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:03.791435    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bzt7g" podUID="a0282906-4494-4b48-821d-ebe85d8c12cb" containerName="discovery" containerID="cri-o://1c6e89ff6907100c63ec978b548c80f5e4d31f9969080ce44e3e1d8d70cd927a" gracePeriod=30
Sep 10 15:34:04.153269 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:04.153000    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bzt7g"
Sep 10 15:34:04.238126 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:04.238086    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a0282906-4494-4b48-821d-ebe85d8c12cb-cacerts\") pod \"a0282906-4494-4b48-821d-ebe85d8c12cb\" (UID: \"a0282906-4494-4b48-821d-ebe85d8c12cb\") "
Sep 10 15:34:04.238302 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:04.238136    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zn2jj\" (UniqueName: \"kubernetes.io/projected/a0282906-4494-4b48-821d-ebe85d8c12cb-kube-api-access-zn2jj\") pod \"a0282906-4494-4b48-821d-ebe85d8c12cb\" (UID: \"a0282906-4494-4b48-821d-ebe85d8c12cb\") "
Sep 10 15:34:04.238302 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:04.238181    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0282906-4494-4b48-821d-ebe85d8c12cb-istio-token\") pod \"a0282906-4494-4b48-821d-ebe85d8c12cb\" (UID: \"a0282906-4494-4b48-821d-ebe85d8c12cb\") "
Sep 10 15:34:04.238302 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:04.238236    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a0282906-4494-4b48-821d-ebe85d8c12cb-istio-csr-ca-configmap\") pod \"a0282906-4494-4b48-821d-ebe85d8c12cb\" (UID: \"a0282906-4494-4b48-821d-ebe85d8c12cb\") "
Sep 10 15:34:04.238302 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:04.238266    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a0282906-4494-4b48-821d-ebe85d8c12cb-istio-kubeconfig\") pod \"a0282906-4494-4b48-821d-ebe85d8c12cb\" (UID: \"a0282906-4494-4b48-821d-ebe85d8c12cb\") "
Sep 10 15:34:04.238302 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:04.238294    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a0282906-4494-4b48-821d-ebe85d8c12cb-istio-csr-dns-cert\") pod \"a0282906-4494-4b48-821d-ebe85d8c12cb\" (UID: \"a0282906-4494-4b48-821d-ebe85d8c12cb\") "
Sep 10 15:34:04.238569 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:04.238323    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0282906-4494-4b48-821d-ebe85d8c12cb-local-certs\") pod \"a0282906-4494-4b48-821d-ebe85d8c12cb\" (UID: \"a0282906-4494-4b48-821d-ebe85d8c12cb\") "
Sep 10 15:34:04.238844 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:04.238799    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0282906-4494-4b48-821d-ebe85d8c12cb-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "a0282906-4494-4b48-821d-ebe85d8c12cb" (UID: "a0282906-4494-4b48-821d-ebe85d8c12cb"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:34:04.241164 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:04.241003    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0282906-4494-4b48-821d-ebe85d8c12cb-istio-token" (OuterVolumeSpecName: "istio-token") pod "a0282906-4494-4b48-821d-ebe85d8c12cb" (UID: "a0282906-4494-4b48-821d-ebe85d8c12cb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:34:04.241623 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:04.241570    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0282906-4494-4b48-821d-ebe85d8c12cb-cacerts" (OuterVolumeSpecName: "cacerts") pod "a0282906-4494-4b48-821d-ebe85d8c12cb" (UID: "a0282906-4494-4b48-821d-ebe85d8c12cb"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:34:04.241758 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:04.241733    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0282906-4494-4b48-821d-ebe85d8c12cb-kube-api-access-zn2jj" (OuterVolumeSpecName: "kube-api-access-zn2jj") pod "a0282906-4494-4b48-821d-ebe85d8c12cb" (UID: "a0282906-4494-4b48-821d-ebe85d8c12cb"). InnerVolumeSpecName "kube-api-access-zn2jj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:34:04.242083 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:04.242042    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0282906-4494-4b48-821d-ebe85d8c12cb-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "a0282906-4494-4b48-821d-ebe85d8c12cb" (UID: "a0282906-4494-4b48-821d-ebe85d8c12cb"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:34:04.242172 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:04.242081    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0282906-4494-4b48-821d-ebe85d8c12cb-local-certs" (OuterVolumeSpecName: "local-certs") pod "a0282906-4494-4b48-821d-ebe85d8c12cb" (UID: "a0282906-4494-4b48-821d-ebe85d8c12cb"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:34:04.242484 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:04.242459    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0282906-4494-4b48-821d-ebe85d8c12cb-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "a0282906-4494-4b48-821d-ebe85d8c12cb" (UID: "a0282906-4494-4b48-821d-ebe85d8c12cb"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:34:04.339806 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:04.339768    2559 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a0282906-4494-4b48-821d-ebe85d8c12cb-cacerts\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:34:04.339806 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:04.339807    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zn2jj\" (UniqueName: \"kubernetes.io/projected/a0282906-4494-4b48-821d-ebe85d8c12cb-kube-api-access-zn2jj\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:34:04.340106 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:04.339823    2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0282906-4494-4b48-821d-ebe85d8c12cb-istio-token\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:34:04.340106 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:04.339835    2559 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a0282906-4494-4b48-821d-ebe85d8c12cb-istio-csr-ca-configmap\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:34:04.340106 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:04.339848    2559 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a0282906-4494-4b48-821d-ebe85d8c12cb-istio-kubeconfig\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:34:04.340106 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:04.339859    2559 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a0282906-4494-4b48-821d-ebe85d8c12cb-istio-csr-dns-cert\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:34:04.340106 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:04.339869    2559 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0282906-4494-4b48-821d-ebe85d8c12cb-local-certs\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:34:04.533984 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:04.533943    2559 generic.go:356] "Generic (PLEG): container finished" podID="a0282906-4494-4b48-821d-ebe85d8c12cb" containerID="1c6e89ff6907100c63ec978b548c80f5e4d31f9969080ce44e3e1d8d70cd927a" exitCode=0
Sep 10 15:34:04.534206 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:04.534004    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bzt7g"
Sep 10 15:34:04.534206 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:04.534044    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bzt7g" event={"ID":"a0282906-4494-4b48-821d-ebe85d8c12cb","Type":"ContainerDied","Data":"1c6e89ff6907100c63ec978b548c80f5e4d31f9969080ce44e3e1d8d70cd927a"}
Sep 10 15:34:04.534206 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:04.534106    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bzt7g" event={"ID":"a0282906-4494-4b48-821d-ebe85d8c12cb","Type":"ContainerDied","Data":"e3185b04d3f6cb6197281043dbf8efa21c52abddcb16eabb97c73c0d9b08707a"}
Sep 10 15:34:04.534206 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:04.534129    2559 scope.go:117] "RemoveContainer" containerID="1c6e89ff6907100c63ec978b548c80f5e4d31f9969080ce44e3e1d8d70cd927a"
Sep 10 15:34:04.544284 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:04.544112    2559 scope.go:117] "RemoveContainer" containerID="1c6e89ff6907100c63ec978b548c80f5e4d31f9969080ce44e3e1d8d70cd927a"
Sep 10 15:34:04.544720 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:34:04.544694    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c6e89ff6907100c63ec978b548c80f5e4d31f9969080ce44e3e1d8d70cd927a\": container with ID starting with 1c6e89ff6907100c63ec978b548c80f5e4d31f9969080ce44e3e1d8d70cd927a not found: ID does not exist" containerID="1c6e89ff6907100c63ec978b548c80f5e4d31f9969080ce44e3e1d8d70cd927a"
Sep 10 15:34:04.544828 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:04.544730    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c6e89ff6907100c63ec978b548c80f5e4d31f9969080ce44e3e1d8d70cd927a"} err="failed to get container status \"1c6e89ff6907100c63ec978b548c80f5e4d31f9969080ce44e3e1d8d70cd927a\": rpc error: code = NotFound desc = could not find container \"1c6e89ff6907100c63ec978b548c80f5e4d31f9969080ce44e3e1d8d70cd927a\": container with ID starting with 1c6e89ff6907100c63ec978b548c80f5e4d31f9969080ce44e3e1d8d70cd927a not found: ID does not exist"
Sep 10 15:34:04.561188 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:04.561160    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bzt7g"]
Sep 10 15:34:04.565701 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:04.565675    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bzt7g"]
Sep 10 15:34:04.784920 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:04.784852    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0282906-4494-4b48-821d-ebe85d8c12cb" path="/var/lib/kubelet/pods/a0282906-4494-4b48-821d-ebe85d8c12cb/volumes"
Sep 10 15:34:06.545769 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:06.545729    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pw7kv" event={"ID":"1315ff10-e055-458c-9e83-7199ef2c576a","Type":"ContainerStarted","Data":"1c07eb6a3d58c6f56cbfb0fb260cdf457256bf317a2db33e0ed17bc7cc103f9b"}
Sep 10 15:34:06.546259 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:06.545835    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pw7kv"
Sep 10 15:34:06.547447 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:06.547418    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6w8hg" event={"ID":"49ace7c7-c5ee-46fc-8f39-3334100ea261","Type":"ContainerStarted","Data":"371d109eae131d28684130a68e285950c372884f73da3d367f32f8dede1e92aa"}
Sep 10 15:34:06.547571 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:06.547556    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6w8hg"
Sep 10 15:34:06.568879 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:06.568831    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pw7kv" podStartSLOduration=2.6559653819999998 podStartE2EDuration="5.568817334s" totalImagesPullingTime="2.912851952s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:34:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:34:06.568042511 +0000 UTC m=+422.310007627" watchObservedRunningTime="2026-09-10 15:34:06.568817334 +0000 UTC m=+422.310782436"
Sep 10 15:34:06.593069 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:06.593011    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6w8hg" podStartSLOduration=1.167948259 podStartE2EDuration="4.592994954s" totalImagesPullingTime="3.425046695s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:34:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:34:06.592052814 +0000 UTC m=+422.334017920" watchObservedRunningTime="2026-09-10 15:34:06.592994954 +0000 UTC m=+422.334960058"
Sep 10 15:34:07.502385 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:07.502347    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-wffnd"
Sep 10 15:34:37.554466 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:37.554379    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6w8hg"
Sep 10 15:34:37.557264 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:37.557245    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pw7kv"
Sep 10 15:34:57.850218 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:57.850179    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5qjzb"]
Sep 10 15:34:57.850706 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:57.850501    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0282906-4494-4b48-821d-ebe85d8c12cb" containerName="discovery"
Sep 10 15:34:57.850706 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:57.850514    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0282906-4494-4b48-821d-ebe85d8c12cb" containerName="discovery"
Sep 10 15:34:57.850706 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:57.850563    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0282906-4494-4b48-821d-ebe85d8c12cb" containerName="discovery"
Sep 10 15:34:57.855905 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:57.855888    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5qjzb"
Sep 10 15:34:57.858411 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:57.858392    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 10 15:34:57.858518 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:57.858434    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-984bz\""
Sep 10 15:34:57.859484 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:57.859463    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 10 15:34:57.867724 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:57.867703    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5qjzb"]
Sep 10 15:34:57.947911 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:57.947874    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mpl9r"]
Sep 10 15:34:57.951183 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:57.951165    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mpl9r"
Sep 10 15:34:57.962304 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:57.962278    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mpl9r"]
Sep 10 15:34:57.965180 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:57.965153    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8bf32747-053b-4aa7-94f3-173b4b4a2992-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5qjzb\" (UID: \"8bf32747-053b-4aa7-94f3-173b4b4a2992\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5qjzb"
Sep 10 15:34:57.965353 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:57.965208    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8bf32747-053b-4aa7-94f3-173b4b4a2992-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5qjzb\" (UID: \"8bf32747-053b-4aa7-94f3-173b4b4a2992\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5qjzb"
Sep 10 15:34:57.965353 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:57.965242    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfvws\" (UniqueName: \"kubernetes.io/projected/8bf32747-053b-4aa7-94f3-173b4b4a2992-kube-api-access-rfvws\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5qjzb\" (UID: \"8bf32747-053b-4aa7-94f3-173b4b4a2992\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5qjzb"
Sep 10 15:34:58.039978 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:58.039947    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jfnbl"]
Sep 10 15:34:58.043231 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:58.043214    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jfnbl"
Sep 10 15:34:58.057634 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:58.057610    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jfnbl"]
Sep 10 15:34:58.066267 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:58.066243    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lp6z\" (UniqueName: \"kubernetes.io/projected/44b86695-746d-4ac6-9733-de366037ccc4-kube-api-access-5lp6z\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mpl9r\" (UID: \"44b86695-746d-4ac6-9733-de366037ccc4\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mpl9r"
Sep 10 15:34:58.066383 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:58.066276    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/44b86695-746d-4ac6-9733-de366037ccc4-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mpl9r\" (UID: \"44b86695-746d-4ac6-9733-de366037ccc4\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mpl9r"
Sep 10 15:34:58.066383 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:58.066335    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8bf32747-053b-4aa7-94f3-173b4b4a2992-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5qjzb\" (UID: \"8bf32747-053b-4aa7-94f3-173b4b4a2992\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5qjzb"
Sep 10 15:34:58.066383 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:58.066366    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8bf32747-053b-4aa7-94f3-173b4b4a2992-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5qjzb\" (UID: \"8bf32747-053b-4aa7-94f3-173b4b4a2992\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5qjzb"
Sep 10 15:34:58.066508 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:58.066393    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfvws\" (UniqueName: \"kubernetes.io/projected/8bf32747-053b-4aa7-94f3-173b4b4a2992-kube-api-access-rfvws\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5qjzb\" (UID: \"8bf32747-053b-4aa7-94f3-173b4b4a2992\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5qjzb"
Sep 10 15:34:58.066508 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:58.066438    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/44b86695-746d-4ac6-9733-de366037ccc4-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mpl9r\" (UID: \"44b86695-746d-4ac6-9733-de366037ccc4\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mpl9r"
Sep 10 15:34:58.066771 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:58.066751    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8bf32747-053b-4aa7-94f3-173b4b4a2992-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5qjzb\" (UID: \"8bf32747-053b-4aa7-94f3-173b4b4a2992\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5qjzb"
Sep 10 15:34:58.066812 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:58.066764    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8bf32747-053b-4aa7-94f3-173b4b4a2992-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5qjzb\" (UID: \"8bf32747-053b-4aa7-94f3-173b4b4a2992\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5qjzb"
Sep 10 15:34:58.077398 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:58.077366    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfvws\" (UniqueName: \"kubernetes.io/projected/8bf32747-053b-4aa7-94f3-173b4b4a2992-kube-api-access-rfvws\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5qjzb\" (UID: \"8bf32747-053b-4aa7-94f3-173b4b4a2992\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5qjzb"
Sep 10 15:34:58.145084 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:58.145005    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4qbq7"]
Sep 10 15:34:58.148505 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:58.148487    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4qbq7"
Sep 10 15:34:58.164489 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:58.164426    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5qjzb"
Sep 10 15:34:58.165104 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:58.165079    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4qbq7"]
Sep 10 15:34:58.167254 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:58.167227    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lp6z\" (UniqueName: \"kubernetes.io/projected/44b86695-746d-4ac6-9733-de366037ccc4-kube-api-access-5lp6z\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mpl9r\" (UID: \"44b86695-746d-4ac6-9733-de366037ccc4\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mpl9r"
Sep 10 15:34:58.167348 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:58.167259    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/44b86695-746d-4ac6-9733-de366037ccc4-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mpl9r\" (UID: \"44b86695-746d-4ac6-9733-de366037ccc4\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mpl9r"
Sep 10 15:34:58.167348 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:58.167311    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1a1b47b0-5654-41dc-8cde-6e0f0d7f9847-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jfnbl\" (UID: \"1a1b47b0-5654-41dc-8cde-6e0f0d7f9847\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jfnbl"
Sep 10 15:34:58.167348 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:58.167341    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1a1b47b0-5654-41dc-8cde-6e0f0d7f9847-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jfnbl\" (UID: \"1a1b47b0-5654-41dc-8cde-6e0f0d7f9847\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jfnbl"
Sep 10 15:34:58.167820 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:58.167374    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/44b86695-746d-4ac6-9733-de366037ccc4-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mpl9r\" (UID: \"44b86695-746d-4ac6-9733-de366037ccc4\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mpl9r"
Sep 10 15:34:58.167820 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:58.167401    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xk25\" (UniqueName: \"kubernetes.io/projected/1a1b47b0-5654-41dc-8cde-6e0f0d7f9847-kube-api-access-2xk25\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jfnbl\" (UID: \"1a1b47b0-5654-41dc-8cde-6e0f0d7f9847\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jfnbl"
Sep 10 15:34:58.167820 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:58.167619    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/44b86695-746d-4ac6-9733-de366037ccc4-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mpl9r\" (UID: \"44b86695-746d-4ac6-9733-de366037ccc4\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mpl9r"
Sep 10 15:34:58.167820 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:58.167713    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/44b86695-746d-4ac6-9733-de366037ccc4-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mpl9r\" (UID: \"44b86695-746d-4ac6-9733-de366037ccc4\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mpl9r"
Sep 10 15:34:58.188205 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:58.188173    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lp6z\" (UniqueName: \"kubernetes.io/projected/44b86695-746d-4ac6-9733-de366037ccc4-kube-api-access-5lp6z\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mpl9r\" (UID: \"44b86695-746d-4ac6-9733-de366037ccc4\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mpl9r"
Sep 10 15:34:58.259946 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:58.259912    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mpl9r"
Sep 10 15:34:58.268022 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:58.267999    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xk25\" (UniqueName: \"kubernetes.io/projected/1a1b47b0-5654-41dc-8cde-6e0f0d7f9847-kube-api-access-2xk25\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jfnbl\" (UID: \"1a1b47b0-5654-41dc-8cde-6e0f0d7f9847\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jfnbl"
Sep 10 15:34:58.268149 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:58.268049    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/51ee3f1f-1632-4ef5-85b6-43a8a8b2e9bf-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4qbq7\" (UID: \"51ee3f1f-1632-4ef5-85b6-43a8a8b2e9bf\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4qbq7"
Sep 10 15:34:58.268149 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:58.268086    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1a1b47b0-5654-41dc-8cde-6e0f0d7f9847-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jfnbl\" (UID: \"1a1b47b0-5654-41dc-8cde-6e0f0d7f9847\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jfnbl"
Sep 10 15:34:58.268226 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:58.268155    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1a1b47b0-5654-41dc-8cde-6e0f0d7f9847-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jfnbl\" (UID: \"1a1b47b0-5654-41dc-8cde-6e0f0d7f9847\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jfnbl"
Sep 10 15:34:58.268226 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:58.268200    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/51ee3f1f-1632-4ef5-85b6-43a8a8b2e9bf-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4qbq7\" (UID: \"51ee3f1f-1632-4ef5-85b6-43a8a8b2e9bf\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4qbq7"
Sep 10 15:34:58.268314 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:58.268236    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2bdd\" (UniqueName: \"kubernetes.io/projected/51ee3f1f-1632-4ef5-85b6-43a8a8b2e9bf-kube-api-access-z2bdd\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4qbq7\" (UID: \"51ee3f1f-1632-4ef5-85b6-43a8a8b2e9bf\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4qbq7"
Sep 10 15:34:58.268425 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:58.268404    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1a1b47b0-5654-41dc-8cde-6e0f0d7f9847-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jfnbl\" (UID: \"1a1b47b0-5654-41dc-8cde-6e0f0d7f9847\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jfnbl"
Sep 10 15:34:58.268506 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:58.268492    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1a1b47b0-5654-41dc-8cde-6e0f0d7f9847-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jfnbl\" (UID: \"1a1b47b0-5654-41dc-8cde-6e0f0d7f9847\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jfnbl"
Sep 10 15:34:58.280676 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:58.280650    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xk25\" (UniqueName: \"kubernetes.io/projected/1a1b47b0-5654-41dc-8cde-6e0f0d7f9847-kube-api-access-2xk25\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jfnbl\" (UID: \"1a1b47b0-5654-41dc-8cde-6e0f0d7f9847\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jfnbl"
Sep 10 15:34:58.352622 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:58.352561    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jfnbl"
Sep 10 15:34:58.368848 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:58.368821    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/51ee3f1f-1632-4ef5-85b6-43a8a8b2e9bf-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4qbq7\" (UID: \"51ee3f1f-1632-4ef5-85b6-43a8a8b2e9bf\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4qbq7"
Sep 10 15:34:58.368973 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:58.368856    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2bdd\" (UniqueName: \"kubernetes.io/projected/51ee3f1f-1632-4ef5-85b6-43a8a8b2e9bf-kube-api-access-z2bdd\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4qbq7\" (UID: \"51ee3f1f-1632-4ef5-85b6-43a8a8b2e9bf\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4qbq7"
Sep 10 15:34:58.368973 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:58.368919    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/51ee3f1f-1632-4ef5-85b6-43a8a8b2e9bf-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4qbq7\" (UID: \"51ee3f1f-1632-4ef5-85b6-43a8a8b2e9bf\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4qbq7"
Sep 10 15:34:58.369248 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:58.369226    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/51ee3f1f-1632-4ef5-85b6-43a8a8b2e9bf-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4qbq7\" (UID: \"51ee3f1f-1632-4ef5-85b6-43a8a8b2e9bf\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4qbq7"
Sep 10 15:34:58.369248 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:58.369246    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/51ee3f1f-1632-4ef5-85b6-43a8a8b2e9bf-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4qbq7\" (UID: \"51ee3f1f-1632-4ef5-85b6-43a8a8b2e9bf\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4qbq7"
Sep 10 15:34:58.380270 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:58.380241    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2bdd\" (UniqueName: \"kubernetes.io/projected/51ee3f1f-1632-4ef5-85b6-43a8a8b2e9bf-kube-api-access-z2bdd\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4qbq7\" (UID: \"51ee3f1f-1632-4ef5-85b6-43a8a8b2e9bf\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4qbq7"
Sep 10 15:34:58.441458 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:58.441433    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5qjzb"]
Sep 10 15:34:58.443700 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:34:58.443674    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bf32747_053b_4aa7_94f3_173b4b4a2992.slice/crio-08b74cd3ae3cadb4552a003659b79027f052339e6c9d977f13c793253bebcdb8 WatchSource:0}: Error finding container 08b74cd3ae3cadb4552a003659b79027f052339e6c9d977f13c793253bebcdb8: Status 404 returned error can't find the container with id 08b74cd3ae3cadb4552a003659b79027f052339e6c9d977f13c793253bebcdb8
Sep 10 15:34:58.459970 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:58.459947    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4qbq7"
Sep 10 15:34:58.536536 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:58.536503    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mpl9r"]
Sep 10 15:34:58.539542 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:34:58.539511    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44b86695_746d_4ac6_9733_de366037ccc4.slice/crio-98439d8a5909fc0ada79884efdc1e4bb46520fdb2224697300904521f7548014 WatchSource:0}: Error finding container 98439d8a5909fc0ada79884efdc1e4bb46520fdb2224697300904521f7548014: Status 404 returned error can't find the container with id 98439d8a5909fc0ada79884efdc1e4bb46520fdb2224697300904521f7548014
Sep 10 15:34:58.627566 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:58.627543    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jfnbl"]
Sep 10 15:34:58.630105 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:34:58.630082    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a1b47b0_5654_41dc_8cde_6e0f0d7f9847.slice/crio-ffc4d97bbefb554c94c7c4b6334097eb2ccf12c75edf4561369c706fa11f646a WatchSource:0}: Error finding container ffc4d97bbefb554c94c7c4b6334097eb2ccf12c75edf4561369c706fa11f646a: Status 404 returned error can't find the container with id ffc4d97bbefb554c94c7c4b6334097eb2ccf12c75edf4561369c706fa11f646a
Sep 10 15:34:58.740205 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:58.740173    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4qbq7"]
Sep 10 15:34:58.774163 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:58.774135    2559 generic.go:356] "Generic (PLEG): container finished" podID="44b86695-746d-4ac6-9733-de366037ccc4" containerID="bd2ed2b3d021ce680dbb7035e9a8de96b14f3f031e8badb5edaa4bd7a4d3ab76" exitCode=0
Sep 10 15:34:58.774291 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:58.774209    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mpl9r" event={"ID":"44b86695-746d-4ac6-9733-de366037ccc4","Type":"ContainerDied","Data":"bd2ed2b3d021ce680dbb7035e9a8de96b14f3f031e8badb5edaa4bd7a4d3ab76"}
Sep 10 15:34:58.774291 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:58.774251    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mpl9r" event={"ID":"44b86695-746d-4ac6-9733-de366037ccc4","Type":"ContainerStarted","Data":"98439d8a5909fc0ada79884efdc1e4bb46520fdb2224697300904521f7548014"}
Sep 10 15:34:58.775891 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:58.775790    2559 generic.go:356] "Generic (PLEG): container finished" podID="1a1b47b0-5654-41dc-8cde-6e0f0d7f9847" containerID="eb108f94d09fab5d7ffd8feb01fb83de4feab21c4fa687aca564c80cdd8d2023" exitCode=0
Sep 10 15:34:58.775891 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:58.775829    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jfnbl" event={"ID":"1a1b47b0-5654-41dc-8cde-6e0f0d7f9847","Type":"ContainerDied","Data":"eb108f94d09fab5d7ffd8feb01fb83de4feab21c4fa687aca564c80cdd8d2023"}
Sep 10 15:34:58.775891 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:58.775887    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jfnbl" event={"ID":"1a1b47b0-5654-41dc-8cde-6e0f0d7f9847","Type":"ContainerStarted","Data":"ffc4d97bbefb554c94c7c4b6334097eb2ccf12c75edf4561369c706fa11f646a"}
Sep 10 15:34:58.777474 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:58.777453    2559 generic.go:356] "Generic (PLEG): container finished" podID="8bf32747-053b-4aa7-94f3-173b4b4a2992" containerID="bc8ef849649da3c860e1d238f150186363249c31cb814a57149728001a10b44c" exitCode=0
Sep 10 15:34:58.777560 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:58.777525    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5qjzb" event={"ID":"8bf32747-053b-4aa7-94f3-173b4b4a2992","Type":"ContainerDied","Data":"bc8ef849649da3c860e1d238f150186363249c31cb814a57149728001a10b44c"}
Sep 10 15:34:58.777619 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:58.777558    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5qjzb" event={"ID":"8bf32747-053b-4aa7-94f3-173b4b4a2992","Type":"ContainerStarted","Data":"08b74cd3ae3cadb4552a003659b79027f052339e6c9d977f13c793253bebcdb8"}
Sep 10 15:34:58.782716 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:34:58.782693    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51ee3f1f_1632_4ef5_85b6_43a8a8b2e9bf.slice/crio-ec5ace6b71ea3d9fde9c4005333e4fd5f099002d642c3115fb66f7a5b7795a0e WatchSource:0}: Error finding container ec5ace6b71ea3d9fde9c4005333e4fd5f099002d642c3115fb66f7a5b7795a0e: Status 404 returned error can't find the container with id ec5ace6b71ea3d9fde9c4005333e4fd5f099002d642c3115fb66f7a5b7795a0e
Sep 10 15:34:59.784079 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:59.784044    2559 generic.go:356] "Generic (PLEG): container finished" podID="1a1b47b0-5654-41dc-8cde-6e0f0d7f9847" containerID="ba1bff2423dc77af979900eef07ffe5a823dfeb81a4d8c0d5dbbce5b7f62eaa7" exitCode=0
Sep 10 15:34:59.784543 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:59.784129    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jfnbl" event={"ID":"1a1b47b0-5654-41dc-8cde-6e0f0d7f9847","Type":"ContainerDied","Data":"ba1bff2423dc77af979900eef07ffe5a823dfeb81a4d8c0d5dbbce5b7f62eaa7"}
Sep 10 15:34:59.786181 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:59.786161    2559 generic.go:356] "Generic (PLEG): container finished" podID="8bf32747-053b-4aa7-94f3-173b4b4a2992" containerID="81654e3bbc9570bd2d42656706c198d65d0c484c3ae8e40e11b21eadbd085bf2" exitCode=0
Sep 10 15:34:59.786266 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:59.786191    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5qjzb" event={"ID":"8bf32747-053b-4aa7-94f3-173b4b4a2992","Type":"ContainerDied","Data":"81654e3bbc9570bd2d42656706c198d65d0c484c3ae8e40e11b21eadbd085bf2"}
Sep 10 15:34:59.787794 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:59.787773    2559 generic.go:356] "Generic (PLEG): container finished" podID="51ee3f1f-1632-4ef5-85b6-43a8a8b2e9bf" containerID="d6cf41822a9addb9d637644321092807afebd467db53d2c077a9d8ee7c217003" exitCode=0
Sep 10 15:34:59.787864 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:59.787805    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4qbq7" event={"ID":"51ee3f1f-1632-4ef5-85b6-43a8a8b2e9bf","Type":"ContainerDied","Data":"d6cf41822a9addb9d637644321092807afebd467db53d2c077a9d8ee7c217003"}
Sep 10 15:34:59.787864 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:34:59.787834    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4qbq7" event={"ID":"51ee3f1f-1632-4ef5-85b6-43a8a8b2e9bf","Type":"ContainerStarted","Data":"ec5ace6b71ea3d9fde9c4005333e4fd5f099002d642c3115fb66f7a5b7795a0e"}
Sep 10 15:35:00.799109 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:00.799071    2559 generic.go:356] "Generic (PLEG): container finished" podID="51ee3f1f-1632-4ef5-85b6-43a8a8b2e9bf" containerID="c3ba1b80aeef647c1cb21882f93b922296e41b4c122bb3c181b9f3d48aad34cb" exitCode=0
Sep 10 15:35:00.799796 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:00.799109    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4qbq7" event={"ID":"51ee3f1f-1632-4ef5-85b6-43a8a8b2e9bf","Type":"ContainerDied","Data":"c3ba1b80aeef647c1cb21882f93b922296e41b4c122bb3c181b9f3d48aad34cb"}
Sep 10 15:35:00.801393 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:00.801369    2559 generic.go:356] "Generic (PLEG): container finished" podID="1a1b47b0-5654-41dc-8cde-6e0f0d7f9847" containerID="af6eb01fbdd3cf76a1b663af222a4a2e13cf752df836d11fc1dcae5b0fcf5ded" exitCode=0
Sep 10 15:35:00.801507 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:00.801452    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jfnbl" event={"ID":"1a1b47b0-5654-41dc-8cde-6e0f0d7f9847","Type":"ContainerDied","Data":"af6eb01fbdd3cf76a1b663af222a4a2e13cf752df836d11fc1dcae5b0fcf5ded"}
Sep 10 15:35:00.803517 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:00.803496    2559 generic.go:356] "Generic (PLEG): container finished" podID="8bf32747-053b-4aa7-94f3-173b4b4a2992" containerID="8192b728c685ae2cb7e803969b7d22dcba29992576839fe9e2f02f0df342584b" exitCode=0
Sep 10 15:35:00.803637 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:00.803556    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5qjzb" event={"ID":"8bf32747-053b-4aa7-94f3-173b4b4a2992","Type":"ContainerDied","Data":"8192b728c685ae2cb7e803969b7d22dcba29992576839fe9e2f02f0df342584b"}
Sep 10 15:35:01.810749 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:01.810714    2559 generic.go:356] "Generic (PLEG): container finished" podID="51ee3f1f-1632-4ef5-85b6-43a8a8b2e9bf" containerID="3a9b94c18b6bd0259023e0724291170b3a6c0f34149615b2626e164e81381131" exitCode=0
Sep 10 15:35:01.811123 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:01.810801    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4qbq7" event={"ID":"51ee3f1f-1632-4ef5-85b6-43a8a8b2e9bf","Type":"ContainerDied","Data":"3a9b94c18b6bd0259023e0724291170b3a6c0f34149615b2626e164e81381131"}
Sep 10 15:35:01.961824 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:01.961800    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5qjzb"
Sep 10 15:35:01.964717 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:01.964695    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jfnbl"
Sep 10 15:35:02.003686 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:02.003654    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1a1b47b0-5654-41dc-8cde-6e0f0d7f9847-util\") pod \"1a1b47b0-5654-41dc-8cde-6e0f0d7f9847\" (UID: \"1a1b47b0-5654-41dc-8cde-6e0f0d7f9847\") "
Sep 10 15:35:02.003831 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:02.003703    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2xk25\" (UniqueName: \"kubernetes.io/projected/1a1b47b0-5654-41dc-8cde-6e0f0d7f9847-kube-api-access-2xk25\") pod \"1a1b47b0-5654-41dc-8cde-6e0f0d7f9847\" (UID: \"1a1b47b0-5654-41dc-8cde-6e0f0d7f9847\") "
Sep 10 15:35:02.003831 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:02.003733    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rfvws\" (UniqueName: \"kubernetes.io/projected/8bf32747-053b-4aa7-94f3-173b4b4a2992-kube-api-access-rfvws\") pod \"8bf32747-053b-4aa7-94f3-173b4b4a2992\" (UID: \"8bf32747-053b-4aa7-94f3-173b4b4a2992\") "
Sep 10 15:35:02.003831 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:02.003759    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8bf32747-053b-4aa7-94f3-173b4b4a2992-bundle\") pod \"8bf32747-053b-4aa7-94f3-173b4b4a2992\" (UID: \"8bf32747-053b-4aa7-94f3-173b4b4a2992\") "
Sep 10 15:35:02.003831 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:02.003777    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1a1b47b0-5654-41dc-8cde-6e0f0d7f9847-bundle\") pod \"1a1b47b0-5654-41dc-8cde-6e0f0d7f9847\" (UID: \"1a1b47b0-5654-41dc-8cde-6e0f0d7f9847\") "
Sep 10 15:35:02.003831 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:02.003807    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8bf32747-053b-4aa7-94f3-173b4b4a2992-util\") pod \"8bf32747-053b-4aa7-94f3-173b4b4a2992\" (UID: \"8bf32747-053b-4aa7-94f3-173b4b4a2992\") "
Sep 10 15:35:02.004427 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:02.004364    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a1b47b0-5654-41dc-8cde-6e0f0d7f9847-bundle" (OuterVolumeSpecName: "bundle") pod "1a1b47b0-5654-41dc-8cde-6e0f0d7f9847" (UID: "1a1b47b0-5654-41dc-8cde-6e0f0d7f9847"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:35:02.004618 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:02.004589    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bf32747-053b-4aa7-94f3-173b4b4a2992-bundle" (OuterVolumeSpecName: "bundle") pod "8bf32747-053b-4aa7-94f3-173b4b4a2992" (UID: "8bf32747-053b-4aa7-94f3-173b4b4a2992"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:35:02.005948 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:02.005923    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a1b47b0-5654-41dc-8cde-6e0f0d7f9847-kube-api-access-2xk25" (OuterVolumeSpecName: "kube-api-access-2xk25") pod "1a1b47b0-5654-41dc-8cde-6e0f0d7f9847" (UID: "1a1b47b0-5654-41dc-8cde-6e0f0d7f9847"). InnerVolumeSpecName "kube-api-access-2xk25". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:35:02.006042 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:02.005945    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bf32747-053b-4aa7-94f3-173b4b4a2992-kube-api-access-rfvws" (OuterVolumeSpecName: "kube-api-access-rfvws") pod "8bf32747-053b-4aa7-94f3-173b4b4a2992" (UID: "8bf32747-053b-4aa7-94f3-173b4b4a2992"). InnerVolumeSpecName "kube-api-access-rfvws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:35:02.010879 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:02.010850    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bf32747-053b-4aa7-94f3-173b4b4a2992-util" (OuterVolumeSpecName: "util") pod "8bf32747-053b-4aa7-94f3-173b4b4a2992" (UID: "8bf32747-053b-4aa7-94f3-173b4b4a2992"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:35:02.012029 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:02.011984    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a1b47b0-5654-41dc-8cde-6e0f0d7f9847-util" (OuterVolumeSpecName: "util") pod "1a1b47b0-5654-41dc-8cde-6e0f0d7f9847" (UID: "1a1b47b0-5654-41dc-8cde-6e0f0d7f9847"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:35:02.104941 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:02.104852    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfvws\" (UniqueName: \"kubernetes.io/projected/8bf32747-053b-4aa7-94f3-173b4b4a2992-kube-api-access-rfvws\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:35:02.104941 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:02.104884    2559 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8bf32747-053b-4aa7-94f3-173b4b4a2992-bundle\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:35:02.104941 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:02.104894    2559 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1a1b47b0-5654-41dc-8cde-6e0f0d7f9847-bundle\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:35:02.104941 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:02.104901    2559 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8bf32747-053b-4aa7-94f3-173b4b4a2992-util\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:35:02.104941 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:02.104913    2559 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1a1b47b0-5654-41dc-8cde-6e0f0d7f9847-util\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:35:02.104941 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:02.104921    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xk25\" (UniqueName: \"kubernetes.io/projected/1a1b47b0-5654-41dc-8cde-6e0f0d7f9847-kube-api-access-2xk25\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:35:02.818923 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:02.818892    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jfnbl"
Sep 10 15:35:02.818923 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:02.818906    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jfnbl" event={"ID":"1a1b47b0-5654-41dc-8cde-6e0f0d7f9847","Type":"ContainerDied","Data":"ffc4d97bbefb554c94c7c4b6334097eb2ccf12c75edf4561369c706fa11f646a"}
Sep 10 15:35:02.819440 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:02.818969    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ffc4d97bbefb554c94c7c4b6334097eb2ccf12c75edf4561369c706fa11f646a"
Sep 10 15:35:02.821237 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:02.821214    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5qjzb"
Sep 10 15:35:02.821352 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:02.821242    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a5qjzb" event={"ID":"8bf32747-053b-4aa7-94f3-173b4b4a2992","Type":"ContainerDied","Data":"08b74cd3ae3cadb4552a003659b79027f052339e6c9d977f13c793253bebcdb8"}
Sep 10 15:35:02.821352 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:02.821298    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="08b74cd3ae3cadb4552a003659b79027f052339e6c9d977f13c793253bebcdb8"
Sep 10 15:35:02.944274 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:02.944249    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4qbq7"
Sep 10 15:35:03.009642 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:03.009611    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z2bdd\" (UniqueName: \"kubernetes.io/projected/51ee3f1f-1632-4ef5-85b6-43a8a8b2e9bf-kube-api-access-z2bdd\") pod \"51ee3f1f-1632-4ef5-85b6-43a8a8b2e9bf\" (UID: \"51ee3f1f-1632-4ef5-85b6-43a8a8b2e9bf\") "
Sep 10 15:35:03.009761 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:03.009652    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/51ee3f1f-1632-4ef5-85b6-43a8a8b2e9bf-util\") pod \"51ee3f1f-1632-4ef5-85b6-43a8a8b2e9bf\" (UID: \"51ee3f1f-1632-4ef5-85b6-43a8a8b2e9bf\") "
Sep 10 15:35:03.009761 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:03.009680    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/51ee3f1f-1632-4ef5-85b6-43a8a8b2e9bf-bundle\") pod \"51ee3f1f-1632-4ef5-85b6-43a8a8b2e9bf\" (UID: \"51ee3f1f-1632-4ef5-85b6-43a8a8b2e9bf\") "
Sep 10 15:35:03.010209 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:03.010181    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51ee3f1f-1632-4ef5-85b6-43a8a8b2e9bf-bundle" (OuterVolumeSpecName: "bundle") pod "51ee3f1f-1632-4ef5-85b6-43a8a8b2e9bf" (UID: "51ee3f1f-1632-4ef5-85b6-43a8a8b2e9bf"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:35:03.011911 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:03.011866    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51ee3f1f-1632-4ef5-85b6-43a8a8b2e9bf-kube-api-access-z2bdd" (OuterVolumeSpecName: "kube-api-access-z2bdd") pod "51ee3f1f-1632-4ef5-85b6-43a8a8b2e9bf" (UID: "51ee3f1f-1632-4ef5-85b6-43a8a8b2e9bf"). InnerVolumeSpecName "kube-api-access-z2bdd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:35:03.014736 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:03.014715    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51ee3f1f-1632-4ef5-85b6-43a8a8b2e9bf-util" (OuterVolumeSpecName: "util") pod "51ee3f1f-1632-4ef5-85b6-43a8a8b2e9bf" (UID: "51ee3f1f-1632-4ef5-85b6-43a8a8b2e9bf"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:35:03.110444 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:03.110385    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2bdd\" (UniqueName: \"kubernetes.io/projected/51ee3f1f-1632-4ef5-85b6-43a8a8b2e9bf-kube-api-access-z2bdd\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:35:03.110444 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:03.110440    2559 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/51ee3f1f-1632-4ef5-85b6-43a8a8b2e9bf-util\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:35:03.110444 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:03.110451    2559 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/51ee3f1f-1632-4ef5-85b6-43a8a8b2e9bf-bundle\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:35:03.827881 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:03.827853    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4qbq7"
Sep 10 15:35:03.828292 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:03.827855    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4qbq7" event={"ID":"51ee3f1f-1632-4ef5-85b6-43a8a8b2e9bf","Type":"ContainerDied","Data":"ec5ace6b71ea3d9fde9c4005333e4fd5f099002d642c3115fb66f7a5b7795a0e"}
Sep 10 15:35:03.828292 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:03.827978    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec5ace6b71ea3d9fde9c4005333e4fd5f099002d642c3115fb66f7a5b7795a0e"
Sep 10 15:35:03.829742 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:03.829719    2559 generic.go:356] "Generic (PLEG): container finished" podID="44b86695-746d-4ac6-9733-de366037ccc4" containerID="a40f265fe09c07b53e6676f2cc538197be29415aef997fe9ad406b2e09a45fa0" exitCode=0
Sep 10 15:35:03.829861 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:03.829798    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mpl9r" event={"ID":"44b86695-746d-4ac6-9733-de366037ccc4","Type":"ContainerDied","Data":"a40f265fe09c07b53e6676f2cc538197be29415aef997fe9ad406b2e09a45fa0"}
Sep 10 15:35:04.836314 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:04.836279    2559 generic.go:356] "Generic (PLEG): container finished" podID="44b86695-746d-4ac6-9733-de366037ccc4" containerID="ee23a48bf118e5f3af3998cc022210b599f5d9b9220b467556954fb1715fd92e" exitCode=0
Sep 10 15:35:04.836747 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:04.836322    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mpl9r" event={"ID":"44b86695-746d-4ac6-9733-de366037ccc4","Type":"ContainerDied","Data":"ee23a48bf118e5f3af3998cc022210b599f5d9b9220b467556954fb1715fd92e"}
Sep 10 15:35:05.964068 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:05.964043    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mpl9r"
Sep 10 15:35:06.029469 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:06.029432    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/44b86695-746d-4ac6-9733-de366037ccc4-util\") pod \"44b86695-746d-4ac6-9733-de366037ccc4\" (UID: \"44b86695-746d-4ac6-9733-de366037ccc4\") "
Sep 10 15:35:06.029655 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:06.029495    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5lp6z\" (UniqueName: \"kubernetes.io/projected/44b86695-746d-4ac6-9733-de366037ccc4-kube-api-access-5lp6z\") pod \"44b86695-746d-4ac6-9733-de366037ccc4\" (UID: \"44b86695-746d-4ac6-9733-de366037ccc4\") "
Sep 10 15:35:06.031544 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:06.031522    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44b86695-746d-4ac6-9733-de366037ccc4-kube-api-access-5lp6z" (OuterVolumeSpecName: "kube-api-access-5lp6z") pod "44b86695-746d-4ac6-9733-de366037ccc4" (UID: "44b86695-746d-4ac6-9733-de366037ccc4"). InnerVolumeSpecName "kube-api-access-5lp6z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:35:06.035655 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:06.035628    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44b86695-746d-4ac6-9733-de366037ccc4-util" (OuterVolumeSpecName: "util") pod "44b86695-746d-4ac6-9733-de366037ccc4" (UID: "44b86695-746d-4ac6-9733-de366037ccc4"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:35:06.130185 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:06.130092    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/44b86695-746d-4ac6-9733-de366037ccc4-bundle\") pod \"44b86695-746d-4ac6-9733-de366037ccc4\" (UID: \"44b86695-746d-4ac6-9733-de366037ccc4\") "
Sep 10 15:35:06.130342 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:06.130253    2559 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/44b86695-746d-4ac6-9733-de366037ccc4-util\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:35:06.130342 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:06.130265    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lp6z\" (UniqueName: \"kubernetes.io/projected/44b86695-746d-4ac6-9733-de366037ccc4-kube-api-access-5lp6z\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:35:06.130522 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:06.130498    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44b86695-746d-4ac6-9733-de366037ccc4-bundle" (OuterVolumeSpecName: "bundle") pod "44b86695-746d-4ac6-9733-de366037ccc4" (UID: "44b86695-746d-4ac6-9733-de366037ccc4"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:35:06.230680 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:06.230629    2559 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/44b86695-746d-4ac6-9733-de366037ccc4-bundle\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:35:06.847159 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:06.847127    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mpl9r"
Sep 10 15:35:06.847294 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:06.847127    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mpl9r" event={"ID":"44b86695-746d-4ac6-9733-de366037ccc4","Type":"ContainerDied","Data":"98439d8a5909fc0ada79884efdc1e4bb46520fdb2224697300904521f7548014"}
Sep 10 15:35:06.847294 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:06.847247    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="98439d8a5909fc0ada79884efdc1e4bb46520fdb2224697300904521f7548014"
Sep 10 15:35:13.835275 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:13.835239    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-f5x5m"]
Sep 10 15:35:13.835713 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:13.835523    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51ee3f1f-1632-4ef5-85b6-43a8a8b2e9bf" containerName="pull"
Sep 10 15:35:13.835713 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:13.835534    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="51ee3f1f-1632-4ef5-85b6-43a8a8b2e9bf" containerName="pull"
Sep 10 15:35:13.835713 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:13.835547    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bf32747-053b-4aa7-94f3-173b4b4a2992" containerName="util"
Sep 10 15:35:13.835713 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:13.835553    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bf32747-053b-4aa7-94f3-173b4b4a2992" containerName="util"
Sep 10 15:35:13.835713 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:13.835558    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44b86695-746d-4ac6-9733-de366037ccc4" containerName="util"
Sep 10 15:35:13.835713 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:13.835564    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="44b86695-746d-4ac6-9733-de366037ccc4" containerName="util"
Sep 10 15:35:13.835713 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:13.835569    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bf32747-053b-4aa7-94f3-173b4b4a2992" containerName="pull"
Sep 10 15:35:13.835713 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:13.835588    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bf32747-053b-4aa7-94f3-173b4b4a2992" containerName="pull"
Sep 10 15:35:13.835713 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:13.835596    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51ee3f1f-1632-4ef5-85b6-43a8a8b2e9bf" containerName="extract"
Sep 10 15:35:13.835713 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:13.835601    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="51ee3f1f-1632-4ef5-85b6-43a8a8b2e9bf" containerName="extract"
Sep 10 15:35:13.835713 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:13.835614    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a1b47b0-5654-41dc-8cde-6e0f0d7f9847" containerName="pull"
Sep 10 15:35:13.835713 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:13.835621    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a1b47b0-5654-41dc-8cde-6e0f0d7f9847" containerName="pull"
Sep 10 15:35:13.835713 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:13.835628    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44b86695-746d-4ac6-9733-de366037ccc4" containerName="pull"
Sep 10 15:35:13.835713 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:13.835633    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="44b86695-746d-4ac6-9733-de366037ccc4" containerName="pull"
Sep 10 15:35:13.835713 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:13.835639    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44b86695-746d-4ac6-9733-de366037ccc4" containerName="extract"
Sep 10 15:35:13.835713 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:13.835646    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="44b86695-746d-4ac6-9733-de366037ccc4" containerName="extract"
Sep 10 15:35:13.835713 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:13.835652    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a1b47b0-5654-41dc-8cde-6e0f0d7f9847" containerName="extract"
Sep 10 15:35:13.835713 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:13.835656    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a1b47b0-5654-41dc-8cde-6e0f0d7f9847" containerName="extract"
Sep 10 15:35:13.835713 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:13.835662    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bf32747-053b-4aa7-94f3-173b4b4a2992" containerName="extract"
Sep 10 15:35:13.835713 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:13.835667    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bf32747-053b-4aa7-94f3-173b4b4a2992" containerName="extract"
Sep 10 15:35:13.835713 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:13.835674    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a1b47b0-5654-41dc-8cde-6e0f0d7f9847" containerName="util"
Sep 10 15:35:13.835713 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:13.835679    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a1b47b0-5654-41dc-8cde-6e0f0d7f9847" containerName="util"
Sep 10 15:35:13.835713 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:13.835686    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51ee3f1f-1632-4ef5-85b6-43a8a8b2e9bf" containerName="util"
Sep 10 15:35:13.835713 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:13.835691    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="51ee3f1f-1632-4ef5-85b6-43a8a8b2e9bf" containerName="util"
Sep 10 15:35:13.836362 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:13.835738    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bf32747-053b-4aa7-94f3-173b4b4a2992" containerName="extract"
Sep 10 15:35:13.836362 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:13.835748    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="44b86695-746d-4ac6-9733-de366037ccc4" containerName="extract"
Sep 10 15:35:13.836362 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:13.835753    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="51ee3f1f-1632-4ef5-85b6-43a8a8b2e9bf" containerName="extract"
Sep 10 15:35:13.836362 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:13.835759    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a1b47b0-5654-41dc-8cde-6e0f0d7f9847" containerName="extract"
Sep 10 15:35:13.839315 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:13.839298    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-f5x5m"
Sep 10 15:35:13.841934 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:13.841904    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 10 15:35:13.842043 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:13.841983    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 10 15:35:13.842043 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:13.842028    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-rk9mn\""
Sep 10 15:35:13.853975 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:13.853951    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-f5x5m"]
Sep 10 15:35:13.885373 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:13.885342    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8ba013fc-07b7-46aa-9019-07f73324923d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-f5x5m\" (UID: \"8ba013fc-07b7-46aa-9019-07f73324923d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-f5x5m"
Sep 10 15:35:13.885487 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:13.885378    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bp6bc\" (UniqueName: \"kubernetes.io/projected/8ba013fc-07b7-46aa-9019-07f73324923d-kube-api-access-bp6bc\") pod \"kuadrant-operator-controller-manager-76f8f686bf-f5x5m\" (UID: \"8ba013fc-07b7-46aa-9019-07f73324923d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-f5x5m"
Sep 10 15:35:13.986129 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:13.986091    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8ba013fc-07b7-46aa-9019-07f73324923d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-f5x5m\" (UID: \"8ba013fc-07b7-46aa-9019-07f73324923d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-f5x5m"
Sep 10 15:35:13.986283 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:13.986145    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bp6bc\" (UniqueName: \"kubernetes.io/projected/8ba013fc-07b7-46aa-9019-07f73324923d-kube-api-access-bp6bc\") pod \"kuadrant-operator-controller-manager-76f8f686bf-f5x5m\" (UID: \"8ba013fc-07b7-46aa-9019-07f73324923d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-f5x5m"
Sep 10 15:35:13.986560 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:13.986541    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8ba013fc-07b7-46aa-9019-07f73324923d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-f5x5m\" (UID: \"8ba013fc-07b7-46aa-9019-07f73324923d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-f5x5m"
Sep 10 15:35:13.998727 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:13.998700    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bp6bc\" (UniqueName: \"kubernetes.io/projected/8ba013fc-07b7-46aa-9019-07f73324923d-kube-api-access-bp6bc\") pod \"kuadrant-operator-controller-manager-76f8f686bf-f5x5m\" (UID: \"8ba013fc-07b7-46aa-9019-07f73324923d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-f5x5m"
Sep 10 15:35:14.149001 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:14.148889    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-f5x5m"
Sep 10 15:35:14.432253 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:14.432116    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-f5x5m"]
Sep 10 15:35:14.434987 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:35:14.434956    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ba013fc_07b7_46aa_9019_07f73324923d.slice/crio-bc63eb3b7ddcb92e8842c67341c34f8c6ad9cde8b74517108ab75943d3c0e4cf WatchSource:0}: Error finding container bc63eb3b7ddcb92e8842c67341c34f8c6ad9cde8b74517108ab75943d3c0e4cf: Status 404 returned error can't find the container with id bc63eb3b7ddcb92e8842c67341c34f8c6ad9cde8b74517108ab75943d3c0e4cf
Sep 10 15:35:14.884981 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:14.884895    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-f5x5m" event={"ID":"8ba013fc-07b7-46aa-9019-07f73324923d","Type":"ContainerStarted","Data":"bc63eb3b7ddcb92e8842c67341c34f8c6ad9cde8b74517108ab75943d3c0e4cf"}
Sep 10 15:35:15.818548 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:15.818507    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-hhvct"]
Sep 10 15:35:15.822254 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:15.822226    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-hhvct"
Sep 10 15:35:15.824823 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:15.824779    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-vz95f\""
Sep 10 15:35:15.833986 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:15.833958    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-hhvct"]
Sep 10 15:35:15.904100 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:15.904061    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpf68\" (UniqueName: \"kubernetes.io/projected/90a0cf60-5e03-4372-ac92-6b3e59c5054a-kube-api-access-gpf68\") pod \"limitador-operator-controller-manager-74495c69f8-hhvct\" (UID: \"90a0cf60-5e03-4372-ac92-6b3e59c5054a\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-hhvct"
Sep 10 15:35:16.005524 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:16.005474    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpf68\" (UniqueName: \"kubernetes.io/projected/90a0cf60-5e03-4372-ac92-6b3e59c5054a-kube-api-access-gpf68\") pod \"limitador-operator-controller-manager-74495c69f8-hhvct\" (UID: \"90a0cf60-5e03-4372-ac92-6b3e59c5054a\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-hhvct"
Sep 10 15:35:16.020299 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:16.020261    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpf68\" (UniqueName: \"kubernetes.io/projected/90a0cf60-5e03-4372-ac92-6b3e59c5054a-kube-api-access-gpf68\") pod \"limitador-operator-controller-manager-74495c69f8-hhvct\" (UID: \"90a0cf60-5e03-4372-ac92-6b3e59c5054a\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-hhvct"
Sep 10 15:35:16.136728 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:16.136641    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-hhvct"
Sep 10 15:35:17.064185 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:17.064105    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-hhvct"]
Sep 10 15:35:17.067779 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:35:17.067752    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90a0cf60_5e03_4372_ac92_6b3e59c5054a.slice/crio-6cc8c2dcc47de4bbbd84eb7de092d3aa4cd6b866883093f665003b82ec19ac89 WatchSource:0}: Error finding container 6cc8c2dcc47de4bbbd84eb7de092d3aa4cd6b866883093f665003b82ec19ac89: Status 404 returned error can't find the container with id 6cc8c2dcc47de4bbbd84eb7de092d3aa4cd6b866883093f665003b82ec19ac89
Sep 10 15:35:17.903370 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:17.903331    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-hhvct" event={"ID":"90a0cf60-5e03-4372-ac92-6b3e59c5054a","Type":"ContainerStarted","Data":"6cc8c2dcc47de4bbbd84eb7de092d3aa4cd6b866883093f665003b82ec19ac89"}
Sep 10 15:35:17.966699 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:17.966659    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-vc2w4"]
Sep 10 15:35:17.970739 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:17.970714    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-vc2w4"
Sep 10 15:35:17.973215 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:17.973193    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 10 15:35:17.973343 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:17.973258    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-wwsrd\""
Sep 10 15:35:17.984059 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:17.984032    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-vc2w4"]
Sep 10 15:35:18.024181 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:18.024146    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbdz9\" (UniqueName: \"kubernetes.io/projected/24787bdd-6822-40e8-b6b4-c701ce34f0dc-kube-api-access-zbdz9\") pod \"dns-operator-controller-manager-7b84cd7c9c-vc2w4\" (UID: \"24787bdd-6822-40e8-b6b4-c701ce34f0dc\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-vc2w4"
Sep 10 15:35:18.124776 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:18.124730    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbdz9\" (UniqueName: \"kubernetes.io/projected/24787bdd-6822-40e8-b6b4-c701ce34f0dc-kube-api-access-zbdz9\") pod \"dns-operator-controller-manager-7b84cd7c9c-vc2w4\" (UID: \"24787bdd-6822-40e8-b6b4-c701ce34f0dc\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-vc2w4"
Sep 10 15:35:18.136294 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:18.136263    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbdz9\" (UniqueName: \"kubernetes.io/projected/24787bdd-6822-40e8-b6b4-c701ce34f0dc-kube-api-access-zbdz9\") pod \"dns-operator-controller-manager-7b84cd7c9c-vc2w4\" (UID: \"24787bdd-6822-40e8-b6b4-c701ce34f0dc\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-vc2w4"
Sep 10 15:35:18.283233 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:18.283193    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-vc2w4"
Sep 10 15:35:18.695870 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:18.695844    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-vc2w4"]
Sep 10 15:35:18.697790 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:35:18.697762    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24787bdd_6822_40e8_b6b4_c701ce34f0dc.slice/crio-ac4b1db2151769ed86e4d9df924d76711d0fb2977b088727650d57663da8b0c8 WatchSource:0}: Error finding container ac4b1db2151769ed86e4d9df924d76711d0fb2977b088727650d57663da8b0c8: Status 404 returned error can't find the container with id ac4b1db2151769ed86e4d9df924d76711d0fb2977b088727650d57663da8b0c8
Sep 10 15:35:18.910658 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:18.910619    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-f5x5m" event={"ID":"8ba013fc-07b7-46aa-9019-07f73324923d","Type":"ContainerStarted","Data":"fcd777368532b9961d2b86cc4c67ba59457f8dee691533dce27c15960431b363"}
Sep 10 15:35:18.910839 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:18.910690    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-f5x5m"
Sep 10 15:35:18.911953 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:18.911929    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-vc2w4" event={"ID":"24787bdd-6822-40e8-b6b4-c701ce34f0dc","Type":"ContainerStarted","Data":"ac4b1db2151769ed86e4d9df924d76711d0fb2977b088727650d57663da8b0c8"}
Sep 10 15:35:18.942281 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:18.942216    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-f5x5m" podStartSLOduration=1.888799821 podStartE2EDuration="5.942194214s" totalImagesPullingTime="4.053394393s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:35:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:35:18.939572814 +0000 UTC m=+494.681537917" watchObservedRunningTime="2026-09-10 15:35:18.942194214 +0000 UTC m=+494.684159319"
Sep 10 15:35:21.081364 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:21.081328    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-zl8r9"]
Sep 10 15:35:21.084830 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:21.084815    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-zl8r9"
Sep 10 15:35:21.087260 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:21.087238    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-wmkpd\""
Sep 10 15:35:21.097132 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:21.097112    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-zl8r9"]
Sep 10 15:35:21.148610 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:21.148561    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzg5q\" (UniqueName: \"kubernetes.io/projected/b57d4eb5-1764-4f58-8fe7-137e9b02a2a2-kube-api-access-hzg5q\") pod \"authorino-operator-76b478bd78-zl8r9\" (UID: \"b57d4eb5-1764-4f58-8fe7-137e9b02a2a2\") " pod="kuadrant-system/authorino-operator-76b478bd78-zl8r9"
Sep 10 15:35:21.249992 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:21.249956    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzg5q\" (UniqueName: \"kubernetes.io/projected/b57d4eb5-1764-4f58-8fe7-137e9b02a2a2-kube-api-access-hzg5q\") pod \"authorino-operator-76b478bd78-zl8r9\" (UID: \"b57d4eb5-1764-4f58-8fe7-137e9b02a2a2\") " pod="kuadrant-system/authorino-operator-76b478bd78-zl8r9"
Sep 10 15:35:21.260142 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:21.260109    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzg5q\" (UniqueName: \"kubernetes.io/projected/b57d4eb5-1764-4f58-8fe7-137e9b02a2a2-kube-api-access-hzg5q\") pod \"authorino-operator-76b478bd78-zl8r9\" (UID: \"b57d4eb5-1764-4f58-8fe7-137e9b02a2a2\") " pod="kuadrant-system/authorino-operator-76b478bd78-zl8r9"
Sep 10 15:35:21.396068 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:21.395976    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-zl8r9"
Sep 10 15:35:21.452361 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:21.451900    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-m6j75"]
Sep 10 15:35:21.457902 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:21.457099    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-m6j75"
Sep 10 15:35:21.460511 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:21.460486    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 10 15:35:21.460667 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:21.460636    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-bgbdk\""
Sep 10 15:35:21.460787 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:21.460689    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 10 15:35:21.467809 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:21.467788    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-m6j75"]
Sep 10 15:35:21.553503 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:21.553463    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/173ab4ec-9a00-40f5-b459-a6b8ac84779a-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-m6j75\" (UID: \"173ab4ec-9a00-40f5-b459-a6b8ac84779a\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-m6j75"
Sep 10 15:35:21.553683 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:21.553517    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pc7tw\" (UniqueName: \"kubernetes.io/projected/173ab4ec-9a00-40f5-b459-a6b8ac84779a-kube-api-access-pc7tw\") pod \"kuadrant-console-plugin-fbdd4f445-m6j75\" (UID: \"173ab4ec-9a00-40f5-b459-a6b8ac84779a\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-m6j75"
Sep 10 15:35:21.553683 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:21.553569    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/173ab4ec-9a00-40f5-b459-a6b8ac84779a-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-m6j75\" (UID: \"173ab4ec-9a00-40f5-b459-a6b8ac84779a\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-m6j75"
Sep 10 15:35:21.655154 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:21.655074    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pc7tw\" (UniqueName: \"kubernetes.io/projected/173ab4ec-9a00-40f5-b459-a6b8ac84779a-kube-api-access-pc7tw\") pod \"kuadrant-console-plugin-fbdd4f445-m6j75\" (UID: \"173ab4ec-9a00-40f5-b459-a6b8ac84779a\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-m6j75"
Sep 10 15:35:21.655154 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:21.655117    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/173ab4ec-9a00-40f5-b459-a6b8ac84779a-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-m6j75\" (UID: \"173ab4ec-9a00-40f5-b459-a6b8ac84779a\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-m6j75"
Sep 10 15:35:21.655154 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:21.655154    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/173ab4ec-9a00-40f5-b459-a6b8ac84779a-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-m6j75\" (UID: \"173ab4ec-9a00-40f5-b459-a6b8ac84779a\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-m6j75"
Sep 10 15:35:21.655761 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:21.655739    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/173ab4ec-9a00-40f5-b459-a6b8ac84779a-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-m6j75\" (UID: \"173ab4ec-9a00-40f5-b459-a6b8ac84779a\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-m6j75"
Sep 10 15:35:21.657468 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:21.657446    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/173ab4ec-9a00-40f5-b459-a6b8ac84779a-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-m6j75\" (UID: \"173ab4ec-9a00-40f5-b459-a6b8ac84779a\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-m6j75"
Sep 10 15:35:21.668094 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:21.668065    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pc7tw\" (UniqueName: \"kubernetes.io/projected/173ab4ec-9a00-40f5-b459-a6b8ac84779a-kube-api-access-pc7tw\") pod \"kuadrant-console-plugin-fbdd4f445-m6j75\" (UID: \"173ab4ec-9a00-40f5-b459-a6b8ac84779a\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-m6j75"
Sep 10 15:35:21.685695 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:21.685673    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-zl8r9"]
Sep 10 15:35:21.687207 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:35:21.687178    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb57d4eb5_1764_4f58_8fe7_137e9b02a2a2.slice/crio-e0a8a12d41b6297e0bd863310e456067f0745fb3ed28bc10b355e21b434dbf04 WatchSource:0}: Error finding container e0a8a12d41b6297e0bd863310e456067f0745fb3ed28bc10b355e21b434dbf04: Status 404 returned error can't find the container with id e0a8a12d41b6297e0bd863310e456067f0745fb3ed28bc10b355e21b434dbf04
Sep 10 15:35:21.768172 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:21.768147    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-m6j75"
Sep 10 15:35:21.927642 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:21.927608    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-zl8r9" event={"ID":"b57d4eb5-1764-4f58-8fe7-137e9b02a2a2","Type":"ContainerStarted","Data":"e0a8a12d41b6297e0bd863310e456067f0745fb3ed28bc10b355e21b434dbf04"}
Sep 10 15:35:22.050531 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:22.050504    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-m6j75"]
Sep 10 15:35:22.051421 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:35:22.051396    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod173ab4ec_9a00_40f5_b459_a6b8ac84779a.slice/crio-0d005928929a57a413e12999004018f5864b160d23416f70669236b4f9ca1b44 WatchSource:0}: Error finding container 0d005928929a57a413e12999004018f5864b160d23416f70669236b4f9ca1b44: Status 404 returned error can't find the container with id 0d005928929a57a413e12999004018f5864b160d23416f70669236b4f9ca1b44
Sep 10 15:35:22.942817 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:22.942746    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-m6j75" event={"ID":"173ab4ec-9a00-40f5-b459-a6b8ac84779a","Type":"ContainerStarted","Data":"0d005928929a57a413e12999004018f5864b160d23416f70669236b4f9ca1b44"}
Sep 10 15:35:23.951391 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:23.951345    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-vc2w4" event={"ID":"24787bdd-6822-40e8-b6b4-c701ce34f0dc","Type":"ContainerStarted","Data":"21bab8ee204413c95a3c67966ac5965deed3a8bf32256bcf4842ab079ad58347"}
Sep 10 15:35:23.951887 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:23.951445    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-vc2w4"
Sep 10 15:35:23.980975 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:23.980918    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-vc2w4" podStartSLOduration=2.244010708 podStartE2EDuration="6.980899714s" totalImagesPullingTime="4.736889006s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:35:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:35:23.977393002 +0000 UTC m=+499.719358107" watchObservedRunningTime="2026-09-10 15:35:23.980899714 +0000 UTC m=+499.722864821"
Sep 10 15:35:24.958915 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:24.958875    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-zl8r9" event={"ID":"b57d4eb5-1764-4f58-8fe7-137e9b02a2a2","Type":"ContainerStarted","Data":"662b7df7e8d3860b675d712bd0a101bd23b9fcd05b5c2de68618bfdee63055c4"}
Sep 10 15:35:24.959433 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:24.958975    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-zl8r9"
Sep 10 15:35:24.992160 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:24.992103    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-76b478bd78-zl8r9" podStartSLOduration=1.433296552 podStartE2EDuration="3.99208452s" totalImagesPullingTime="2.558787968s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:35:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:35:24.990560074 +0000 UTC m=+500.732525181" watchObservedRunningTime="2026-09-10 15:35:24.99208452 +0000 UTC m=+500.734049622"
Sep 10 15:35:27.976908 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:27.976866    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-hhvct" event={"ID":"90a0cf60-5e03-4372-ac92-6b3e59c5054a","Type":"ContainerStarted","Data":"c10070230c21071ca7bafa7f54da170163ed88e3ec6f4072c937eb981fd1ac39"}
Sep 10 15:35:27.977352 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:27.976981    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-hhvct"
Sep 10 15:35:27.978505 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:27.978478    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-m6j75" event={"ID":"173ab4ec-9a00-40f5-b459-a6b8ac84779a","Type":"ContainerStarted","Data":"1c85d652c07f2cdfac86ed3df13746e2332fd76a7d6077d173a21f16c092e982"}
Sep 10 15:35:28.002716 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:28.002671    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-hhvct" podStartSLOduration=2.881169523 podStartE2EDuration="13.002659437s" totalImagesPullingTime="10.121489914s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:35:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:35:27.998964196 +0000 UTC m=+503.740929299" watchObservedRunningTime="2026-09-10 15:35:28.002659437 +0000 UTC m=+503.744624541"
Sep 10 15:35:28.027001 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:28.026956    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-m6j75" podStartSLOduration=1.935604064 podStartE2EDuration="7.026943644s" totalImagesPullingTime="5.09133958s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:35:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:35:28.024537958 +0000 UTC m=+503.766503072" watchObservedRunningTime="2026-09-10 15:35:28.026943644 +0000 UTC m=+503.768908771"
Sep 10 15:35:29.918432 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:29.918401    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-f5x5m"
Sep 10 15:35:32.293967 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:32.293925    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-f5x5m"]
Sep 10 15:35:32.294418 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:32.294148    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-f5x5m" podUID="8ba013fc-07b7-46aa-9019-07f73324923d" containerName="manager" containerID="cri-o://fcd777368532b9961d2b86cc4c67ba59457f8dee691533dce27c15960431b363" gracePeriod=2
Sep 10 15:35:32.299208 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:32.299177    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-f5x5m"]
Sep 10 15:35:32.317129 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:32.317099    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-hhvct"]
Sep 10 15:35:32.317329 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:32.317282    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-hhvct" podUID="90a0cf60-5e03-4372-ac92-6b3e59c5054a" containerName="manager" containerID="cri-o://c10070230c21071ca7bafa7f54da170163ed88e3ec6f4072c937eb981fd1ac39" gracePeriod=2
Sep 10 15:35:32.319365 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:32.319340    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-hhvct"
Sep 10 15:35:32.320180 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:35:32.320148    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-f5x5m\" is forbidden: User \"system:node:ip-10-0-130-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-30.ec2.internal' and this object" podUID="8ba013fc-07b7-46aa-9019-07f73324923d" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-f5x5m"
Sep 10 15:35:32.329822 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:32.329798    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-hhvct"]
Sep 10 15:35:32.334145 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:35:32.334107    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-f5x5m\" is forbidden: User \"system:node:ip-10-0-130-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-30.ec2.internal' and this object" podUID="8ba013fc-07b7-46aa-9019-07f73324923d" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-f5x5m"
Sep 10 15:35:32.339927 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:35:32.339901    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-hhvct\" is forbidden: User \"system:node:ip-10-0-130-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-30.ec2.internal' and this object" podUID="90a0cf60-5e03-4372-ac92-6b3e59c5054a" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-hhvct"
Sep 10 15:35:32.361792 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:32.361764    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-7gjkr"]
Sep 10 15:35:32.362126 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:32.362109    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ba013fc-07b7-46aa-9019-07f73324923d" containerName="manager"
Sep 10 15:35:32.362204 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:32.362129    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ba013fc-07b7-46aa-9019-07f73324923d" containerName="manager"
Sep 10 15:35:32.362204 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:32.362161    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90a0cf60-5e03-4372-ac92-6b3e59c5054a" containerName="manager"
Sep 10 15:35:32.362204 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:32.362169    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="90a0cf60-5e03-4372-ac92-6b3e59c5054a" containerName="manager"
Sep 10 15:35:32.362346 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:32.362253    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ba013fc-07b7-46aa-9019-07f73324923d" containerName="manager"
Sep 10 15:35:32.362346 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:32.362265    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="90a0cf60-5e03-4372-ac92-6b3e59c5054a" containerName="manager"
Sep 10 15:35:32.365180 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:32.365160    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7gjkr"
Sep 10 15:35:32.386666 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:32.386631    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-7gjkr"]
Sep 10 15:35:32.423552 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:35:32.423517    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-f5x5m\" is forbidden: User \"system:node:ip-10-0-130-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-30.ec2.internal' and this object" podUID="8ba013fc-07b7-46aa-9019-07f73324923d" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-f5x5m"
Sep 10 15:35:32.425751 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:35:32.425718    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-hhvct\" is forbidden: User \"system:node:ip-10-0-130-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-30.ec2.internal' and this object" podUID="90a0cf60-5e03-4372-ac92-6b3e59c5054a" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-hhvct"
Sep 10 15:35:32.447618 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:32.447565    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhlbb\" (UniqueName: \"kubernetes.io/projected/1d313e05-146a-428f-8e9c-8709bd11293c-kube-api-access-fhlbb\") pod \"limitador-operator-controller-manager-74495c69f8-7gjkr\" (UID: \"1d313e05-146a-428f-8e9c-8709bd11293c\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7gjkr"
Sep 10 15:35:32.548238 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:32.548180    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhlbb\" (UniqueName: \"kubernetes.io/projected/1d313e05-146a-428f-8e9c-8709bd11293c-kube-api-access-fhlbb\") pod \"limitador-operator-controller-manager-74495c69f8-7gjkr\" (UID: \"1d313e05-146a-428f-8e9c-8709bd11293c\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7gjkr"
Sep 10 15:35:32.551948 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:32.551925    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-f5x5m"
Sep 10 15:35:32.554338 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:35:32.554310    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-f5x5m\" is forbidden: User \"system:node:ip-10-0-130-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-30.ec2.internal' and this object" podUID="8ba013fc-07b7-46aa-9019-07f73324923d" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-f5x5m"
Sep 10 15:35:32.554824 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:32.554804    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-hhvct"
Sep 10 15:35:32.556609 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:35:32.556571    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-hhvct\" is forbidden: User \"system:node:ip-10-0-130-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-30.ec2.internal' and this object" podUID="90a0cf60-5e03-4372-ac92-6b3e59c5054a" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-hhvct"
Sep 10 15:35:32.558653 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:35:32.558631    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-f5x5m\" is forbidden: User \"system:node:ip-10-0-130-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-30.ec2.internal' and this object" podUID="8ba013fc-07b7-46aa-9019-07f73324923d" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-f5x5m"
Sep 10 15:35:32.561008 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:35:32.560984    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-hhvct\" is forbidden: User \"system:node:ip-10-0-130-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-30.ec2.internal' and this object" podUID="90a0cf60-5e03-4372-ac92-6b3e59c5054a" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-hhvct"
Sep 10 15:35:32.571904 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:32.571882    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhlbb\" (UniqueName: \"kubernetes.io/projected/1d313e05-146a-428f-8e9c-8709bd11293c-kube-api-access-fhlbb\") pod \"limitador-operator-controller-manager-74495c69f8-7gjkr\" (UID: \"1d313e05-146a-428f-8e9c-8709bd11293c\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7gjkr"
Sep 10 15:35:32.648640 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:32.648600    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gpf68\" (UniqueName: \"kubernetes.io/projected/90a0cf60-5e03-4372-ac92-6b3e59c5054a-kube-api-access-gpf68\") pod \"90a0cf60-5e03-4372-ac92-6b3e59c5054a\" (UID: \"90a0cf60-5e03-4372-ac92-6b3e59c5054a\") "
Sep 10 15:35:32.648829 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:32.648663    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8ba013fc-07b7-46aa-9019-07f73324923d-extensions-socket-volume\") pod \"8ba013fc-07b7-46aa-9019-07f73324923d\" (UID: \"8ba013fc-07b7-46aa-9019-07f73324923d\") "
Sep 10 15:35:32.648829 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:32.648691    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bp6bc\" (UniqueName: \"kubernetes.io/projected/8ba013fc-07b7-46aa-9019-07f73324923d-kube-api-access-bp6bc\") pod \"8ba013fc-07b7-46aa-9019-07f73324923d\" (UID: \"8ba013fc-07b7-46aa-9019-07f73324923d\") "
Sep 10 15:35:32.649162 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:32.649135    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ba013fc-07b7-46aa-9019-07f73324923d-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "8ba013fc-07b7-46aa-9019-07f73324923d" (UID: "8ba013fc-07b7-46aa-9019-07f73324923d"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:35:32.650866 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:32.650840    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90a0cf60-5e03-4372-ac92-6b3e59c5054a-kube-api-access-gpf68" (OuterVolumeSpecName: "kube-api-access-gpf68") pod "90a0cf60-5e03-4372-ac92-6b3e59c5054a" (UID: "90a0cf60-5e03-4372-ac92-6b3e59c5054a"). InnerVolumeSpecName "kube-api-access-gpf68". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:35:32.650932 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:32.650876    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ba013fc-07b7-46aa-9019-07f73324923d-kube-api-access-bp6bc" (OuterVolumeSpecName: "kube-api-access-bp6bc") pod "8ba013fc-07b7-46aa-9019-07f73324923d" (UID: "8ba013fc-07b7-46aa-9019-07f73324923d"). InnerVolumeSpecName "kube-api-access-bp6bc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:35:32.721770 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:32.721734    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7gjkr"
Sep 10 15:35:32.750313 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:32.750269    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpf68\" (UniqueName: \"kubernetes.io/projected/90a0cf60-5e03-4372-ac92-6b3e59c5054a-kube-api-access-gpf68\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:35:32.750431 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:32.750316    2559 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8ba013fc-07b7-46aa-9019-07f73324923d-extensions-socket-volume\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:35:32.750431 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:32.750332    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bp6bc\" (UniqueName: \"kubernetes.io/projected/8ba013fc-07b7-46aa-9019-07f73324923d-kube-api-access-bp6bc\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:35:32.792984 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:32.792948    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ba013fc-07b7-46aa-9019-07f73324923d" path="/var/lib/kubelet/pods/8ba013fc-07b7-46aa-9019-07f73324923d/volumes"
Sep 10 15:35:32.793483 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:32.793460    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90a0cf60-5e03-4372-ac92-6b3e59c5054a" path="/var/lib/kubelet/pods/90a0cf60-5e03-4372-ac92-6b3e59c5054a/volumes"
Sep 10 15:35:33.004648 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:33.004478    2559 generic.go:356] "Generic (PLEG): container finished" podID="8ba013fc-07b7-46aa-9019-07f73324923d" containerID="fcd777368532b9961d2b86cc4c67ba59457f8dee691533dce27c15960431b363" exitCode=0
Sep 10 15:35:33.004648 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:33.004559    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-f5x5m"
Sep 10 15:35:33.004648 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:33.004632    2559 scope.go:117] "RemoveContainer" containerID="fcd777368532b9961d2b86cc4c67ba59457f8dee691533dce27c15960431b363"
Sep 10 15:35:33.006561 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:33.006533    2559 generic.go:356] "Generic (PLEG): container finished" podID="90a0cf60-5e03-4372-ac92-6b3e59c5054a" containerID="c10070230c21071ca7bafa7f54da170163ed88e3ec6f4072c937eb981fd1ac39" exitCode=0
Sep 10 15:35:33.006701 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:33.006652    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-hhvct"
Sep 10 15:35:33.008480 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:33.008451    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-7gjkr"]
Sep 10 15:35:33.009103 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:35:33.009081    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d313e05_146a_428f_8e9c_8709bd11293c.slice/crio-d4b8785fb332463717ed7b387a1c1fdfba8f0d86cf0b580389f837d09d3d6d2f WatchSource:0}: Error finding container d4b8785fb332463717ed7b387a1c1fdfba8f0d86cf0b580389f837d09d3d6d2f: Status 404 returned error can't find the container with id d4b8785fb332463717ed7b387a1c1fdfba8f0d86cf0b580389f837d09d3d6d2f
Sep 10 15:35:33.014102 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:33.014083    2559 scope.go:117] "RemoveContainer" containerID="fcd777368532b9961d2b86cc4c67ba59457f8dee691533dce27c15960431b363"
Sep 10 15:35:33.014361 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:35:33.014344    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcd777368532b9961d2b86cc4c67ba59457f8dee691533dce27c15960431b363\": container with ID starting with fcd777368532b9961d2b86cc4c67ba59457f8dee691533dce27c15960431b363 not found: ID does not exist" containerID="fcd777368532b9961d2b86cc4c67ba59457f8dee691533dce27c15960431b363"
Sep 10 15:35:33.014415 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:33.014369    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcd777368532b9961d2b86cc4c67ba59457f8dee691533dce27c15960431b363"} err="failed to get container status \"fcd777368532b9961d2b86cc4c67ba59457f8dee691533dce27c15960431b363\": rpc error: code = NotFound desc = could not find container \"fcd777368532b9961d2b86cc4c67ba59457f8dee691533dce27c15960431b363\": container with ID starting with fcd777368532b9961d2b86cc4c67ba59457f8dee691533dce27c15960431b363 not found: ID does not exist"
Sep 10 15:35:33.014415 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:33.014386    2559 scope.go:117] "RemoveContainer" containerID="c10070230c21071ca7bafa7f54da170163ed88e3ec6f4072c937eb981fd1ac39"
Sep 10 15:35:33.022467 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:33.022452    2559 scope.go:117] "RemoveContainer" containerID="c10070230c21071ca7bafa7f54da170163ed88e3ec6f4072c937eb981fd1ac39"
Sep 10 15:35:33.022765 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:35:33.022739    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c10070230c21071ca7bafa7f54da170163ed88e3ec6f4072c937eb981fd1ac39\": container with ID starting with c10070230c21071ca7bafa7f54da170163ed88e3ec6f4072c937eb981fd1ac39 not found: ID does not exist" containerID="c10070230c21071ca7bafa7f54da170163ed88e3ec6f4072c937eb981fd1ac39"
Sep 10 15:35:33.022819 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:33.022774    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c10070230c21071ca7bafa7f54da170163ed88e3ec6f4072c937eb981fd1ac39"} err="failed to get container status \"c10070230c21071ca7bafa7f54da170163ed88e3ec6f4072c937eb981fd1ac39\": rpc error: code = NotFound desc = could not find container \"c10070230c21071ca7bafa7f54da170163ed88e3ec6f4072c937eb981fd1ac39\": container with ID starting with c10070230c21071ca7bafa7f54da170163ed88e3ec6f4072c937eb981fd1ac39 not found: ID does not exist"
Sep 10 15:35:33.553415 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:33.553380    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7c7ft"]
Sep 10 15:35:33.556833 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:33.556815    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7c7ft"
Sep 10 15:35:33.560448 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:33.560428    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-rk9mn\""
Sep 10 15:35:33.579566 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:33.579510    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7c7ft"]
Sep 10 15:35:33.657809 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:33.657778    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8f20ab32-7d9d-4055-a1f0-0e07cf692e5e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-7c7ft\" (UID: \"8f20ab32-7d9d-4055-a1f0-0e07cf692e5e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7c7ft"
Sep 10 15:35:33.657992 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:33.657878    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rl79x\" (UniqueName: \"kubernetes.io/projected/8f20ab32-7d9d-4055-a1f0-0e07cf692e5e-kube-api-access-rl79x\") pod \"kuadrant-operator-controller-manager-5769f99d9f-7c7ft\" (UID: \"8f20ab32-7d9d-4055-a1f0-0e07cf692e5e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7c7ft"
Sep 10 15:35:33.758871 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:33.758838    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8f20ab32-7d9d-4055-a1f0-0e07cf692e5e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-7c7ft\" (UID: \"8f20ab32-7d9d-4055-a1f0-0e07cf692e5e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7c7ft"
Sep 10 15:35:33.759036 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:33.758889    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rl79x\" (UniqueName: \"kubernetes.io/projected/8f20ab32-7d9d-4055-a1f0-0e07cf692e5e-kube-api-access-rl79x\") pod \"kuadrant-operator-controller-manager-5769f99d9f-7c7ft\" (UID: \"8f20ab32-7d9d-4055-a1f0-0e07cf692e5e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7c7ft"
Sep 10 15:35:33.759249 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:33.759226    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8f20ab32-7d9d-4055-a1f0-0e07cf692e5e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-7c7ft\" (UID: \"8f20ab32-7d9d-4055-a1f0-0e07cf692e5e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7c7ft"
Sep 10 15:35:33.772999 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:33.772973    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rl79x\" (UniqueName: \"kubernetes.io/projected/8f20ab32-7d9d-4055-a1f0-0e07cf692e5e-kube-api-access-rl79x\") pod \"kuadrant-operator-controller-manager-5769f99d9f-7c7ft\" (UID: \"8f20ab32-7d9d-4055-a1f0-0e07cf692e5e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7c7ft"
Sep 10 15:35:33.866811 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:33.866739    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7c7ft"
Sep 10 15:35:34.015654 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:34.015620    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7gjkr" event={"ID":"1d313e05-146a-428f-8e9c-8709bd11293c","Type":"ContainerStarted","Data":"4ac8a72871b748b0a2577c5dd428992a4156c9cbf5c86a17f2e8ada31f79b57a"}
Sep 10 15:35:34.015819 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:34.015667    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7gjkr" event={"ID":"1d313e05-146a-428f-8e9c-8709bd11293c","Type":"ContainerStarted","Data":"d4b8785fb332463717ed7b387a1c1fdfba8f0d86cf0b580389f837d09d3d6d2f"}
Sep 10 15:35:34.015819 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:34.015733    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7gjkr"
Sep 10 15:35:34.037016 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:34.036967    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7gjkr" podStartSLOduration=2.036950383 podStartE2EDuration="2.036950383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:35:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:35:34.034542242 +0000 UTC m=+509.776507345" watchObservedRunningTime="2026-09-10 15:35:34.036950383 +0000 UTC m=+509.778915486"
Sep 10 15:35:34.147972 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:34.147943    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7c7ft"]
Sep 10 15:35:34.149557 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:35:34.149533    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f20ab32_7d9d_4055_a1f0_0e07cf692e5e.slice/crio-0c060a35c417c122f8fc22930046959a2daecc256d70532808fa0646130f9693 WatchSource:0}: Error finding container 0c060a35c417c122f8fc22930046959a2daecc256d70532808fa0646130f9693: Status 404 returned error can't find the container with id 0c060a35c417c122f8fc22930046959a2daecc256d70532808fa0646130f9693
Sep 10 15:35:34.961133 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:34.961100    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-vc2w4"
Sep 10 15:35:35.031501 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:35.031462    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7c7ft" event={"ID":"8f20ab32-7d9d-4055-a1f0-0e07cf692e5e","Type":"ContainerStarted","Data":"f385fa6a38e5450745bb0c0bb6c698b26ad410b06341075ffb3d7cb13ad6be90"}
Sep 10 15:35:35.031677 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:35.031565    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7c7ft" event={"ID":"8f20ab32-7d9d-4055-a1f0-0e07cf692e5e","Type":"ContainerStarted","Data":"0c060a35c417c122f8fc22930046959a2daecc256d70532808fa0646130f9693"}
Sep 10 15:35:35.031677 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:35.031656    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7c7ft"
Sep 10 15:35:35.088318 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:35.088267    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7c7ft" podStartSLOduration=2.088252231 podStartE2EDuration="2.088252231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:35:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:35:35.084812229 +0000 UTC m=+510.826777341" watchObservedRunningTime="2026-09-10 15:35:35.088252231 +0000 UTC m=+510.830217333"
Sep 10 15:35:35.966881 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:35.966845    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-zl8r9"
Sep 10 15:35:45.033161 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:45.033130    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7gjkr"
Sep 10 15:35:47.049142 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:35:47.049114    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7c7ft"
Sep 10 15:36:07.252350 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:36:07.252314    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bs7hd"]
Sep 10 15:36:07.259373 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:36:07.259353    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-bs7hd"
Sep 10 15:36:07.262868 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:36:07.262851    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 10 15:36:07.276702 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:36:07.276677    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bs7hd"]
Sep 10 15:36:07.331156 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:36:07.331120    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bs7hd"]
Sep 10 15:36:07.339296 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:36:07.339265    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3f6d47db-a6d6-49ee-88a8-0239dbdee32f-config-file\") pod \"limitador-limitador-84994f7ffc-bs7hd\" (UID: \"3f6d47db-a6d6-49ee-88a8-0239dbdee32f\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-bs7hd"
Sep 10 15:36:07.339436 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:36:07.339299    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvrq8\" (UniqueName: \"kubernetes.io/projected/3f6d47db-a6d6-49ee-88a8-0239dbdee32f-kube-api-access-gvrq8\") pod \"limitador-limitador-84994f7ffc-bs7hd\" (UID: \"3f6d47db-a6d6-49ee-88a8-0239dbdee32f\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-bs7hd"
Sep 10 15:36:07.440335 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:36:07.440300    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3f6d47db-a6d6-49ee-88a8-0239dbdee32f-config-file\") pod \"limitador-limitador-84994f7ffc-bs7hd\" (UID: \"3f6d47db-a6d6-49ee-88a8-0239dbdee32f\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-bs7hd"
Sep 10 15:36:07.440335 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:36:07.440336    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvrq8\" (UniqueName: \"kubernetes.io/projected/3f6d47db-a6d6-49ee-88a8-0239dbdee32f-kube-api-access-gvrq8\") pod \"limitador-limitador-84994f7ffc-bs7hd\" (UID: \"3f6d47db-a6d6-49ee-88a8-0239dbdee32f\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-bs7hd"
Sep 10 15:36:07.440982 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:36:07.440961    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3f6d47db-a6d6-49ee-88a8-0239dbdee32f-config-file\") pod \"limitador-limitador-84994f7ffc-bs7hd\" (UID: \"3f6d47db-a6d6-49ee-88a8-0239dbdee32f\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-bs7hd"
Sep 10 15:36:07.451505 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:36:07.451475    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvrq8\" (UniqueName: \"kubernetes.io/projected/3f6d47db-a6d6-49ee-88a8-0239dbdee32f-kube-api-access-gvrq8\") pod \"limitador-limitador-84994f7ffc-bs7hd\" (UID: \"3f6d47db-a6d6-49ee-88a8-0239dbdee32f\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-bs7hd"
Sep 10 15:36:07.569336 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:36:07.569246    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-bs7hd"
Sep 10 15:36:07.847225 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:36:07.847201    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bs7hd"]
Sep 10 15:36:07.848925 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:36:07.848901    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f6d47db_a6d6_49ee_88a8_0239dbdee32f.slice/crio-e01d75e2e049fbfeec31229b8e50f25c9aaba612d2f830de1382d593dc0c70db WatchSource:0}: Error finding container e01d75e2e049fbfeec31229b8e50f25c9aaba612d2f830de1382d593dc0c70db: Status 404 returned error can't find the container with id e01d75e2e049fbfeec31229b8e50f25c9aaba612d2f830de1382d593dc0c70db
Sep 10 15:36:08.205537 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:36:08.205503    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-bs7hd" event={"ID":"3f6d47db-a6d6-49ee-88a8-0239dbdee32f","Type":"ContainerStarted","Data":"e01d75e2e049fbfeec31229b8e50f25c9aaba612d2f830de1382d593dc0c70db"}
Sep 10 15:36:09.212716 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:36:09.212679    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-bs7hd" event={"ID":"3f6d47db-a6d6-49ee-88a8-0239dbdee32f","Type":"ContainerStarted","Data":"157cf56366161c0bf5579709e4711b1499227e8ff1cbfe5fc84c65fb0024a420"}
Sep 10 15:36:09.213100 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:36:09.212742    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-bs7hd"
Sep 10 15:36:09.240274 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:36:09.240216    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-bs7hd" podStartSLOduration=1.245629125 podStartE2EDuration="2.240200656s" totalImagesPullingTime="994.571531ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:36:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:36:09.238119847 +0000 UTC m=+544.980084961" watchObservedRunningTime="2026-09-10 15:36:09.240200656 +0000 UTC m=+544.982165793"
Sep 10 15:36:20.219277 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:36:20.219244    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-bs7hd"
Sep 10 15:37:20.949747 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:37:20.949711    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bs7hd"]
Sep 10 15:37:47.615491 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:37:47.615455    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bs7hd"]
Sep 10 15:38:21.605865 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:38:21.605826    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bs7hd"]
Sep 10 15:38:25.697788 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:38:25.697751    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8nw6r"]
Sep 10 15:38:25.701301 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:38:25.701274    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8nw6r"
Sep 10 15:38:25.705004 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:38:25.704973    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 10 15:38:25.705150 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:38:25.705017    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-vwb6m\""
Sep 10 15:38:25.705150 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:38:25.704985    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Sep 10 15:38:25.705150 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:38:25.705055    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 10 15:38:25.713229 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:38:25.713204    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8nw6r"]
Sep 10 15:38:25.779515 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:38:25.779475    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vf9t4\" (UniqueName: \"kubernetes.io/projected/6ce9541a-aa1f-417b-a31a-263a328bae29-kube-api-access-vf9t4\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8nw6r\" (UID: \"6ce9541a-aa1f-417b-a31a-263a328bae29\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8nw6r"
Sep 10 15:38:25.779515 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:38:25.779519    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ce9541a-aa1f-417b-a31a-263a328bae29-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8nw6r\" (UID: \"6ce9541a-aa1f-417b-a31a-263a328bae29\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8nw6r"
Sep 10 15:38:25.779763 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:38:25.779541    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ce9541a-aa1f-417b-a31a-263a328bae29-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8nw6r\" (UID: \"6ce9541a-aa1f-417b-a31a-263a328bae29\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8nw6r"
Sep 10 15:38:25.779763 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:38:25.779670    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ce9541a-aa1f-417b-a31a-263a328bae29-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8nw6r\" (UID: \"6ce9541a-aa1f-417b-a31a-263a328bae29\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8nw6r"
Sep 10 15:38:25.779763 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:38:25.779706    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ce9541a-aa1f-417b-a31a-263a328bae29-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8nw6r\" (UID: \"6ce9541a-aa1f-417b-a31a-263a328bae29\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8nw6r"
Sep 10 15:38:25.779763 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:38:25.779725    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ce9541a-aa1f-417b-a31a-263a328bae29-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8nw6r\" (UID: \"6ce9541a-aa1f-417b-a31a-263a328bae29\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8nw6r"
Sep 10 15:38:25.880301 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:38:25.880268    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vf9t4\" (UniqueName: \"kubernetes.io/projected/6ce9541a-aa1f-417b-a31a-263a328bae29-kube-api-access-vf9t4\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8nw6r\" (UID: \"6ce9541a-aa1f-417b-a31a-263a328bae29\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8nw6r"
Sep 10 15:38:25.880301 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:38:25.880309    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ce9541a-aa1f-417b-a31a-263a328bae29-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8nw6r\" (UID: \"6ce9541a-aa1f-417b-a31a-263a328bae29\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8nw6r"
Sep 10 15:38:25.880554 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:38:25.880339    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ce9541a-aa1f-417b-a31a-263a328bae29-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8nw6r\" (UID: \"6ce9541a-aa1f-417b-a31a-263a328bae29\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8nw6r"
Sep 10 15:38:25.880554 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:38:25.880365    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ce9541a-aa1f-417b-a31a-263a328bae29-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8nw6r\" (UID: \"6ce9541a-aa1f-417b-a31a-263a328bae29\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8nw6r"
Sep 10 15:38:25.880554 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:38:25.880381    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ce9541a-aa1f-417b-a31a-263a328bae29-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8nw6r\" (UID: \"6ce9541a-aa1f-417b-a31a-263a328bae29\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8nw6r"
Sep 10 15:38:25.880554 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:38:25.880396    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ce9541a-aa1f-417b-a31a-263a328bae29-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8nw6r\" (UID: \"6ce9541a-aa1f-417b-a31a-263a328bae29\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8nw6r"
Sep 10 15:38:25.880855 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:38:25.880831    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ce9541a-aa1f-417b-a31a-263a328bae29-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8nw6r\" (UID: \"6ce9541a-aa1f-417b-a31a-263a328bae29\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8nw6r"
Sep 10 15:38:25.880911 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:38:25.880875    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ce9541a-aa1f-417b-a31a-263a328bae29-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8nw6r\" (UID: \"6ce9541a-aa1f-417b-a31a-263a328bae29\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8nw6r"
Sep 10 15:38:25.880911 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:38:25.880884    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ce9541a-aa1f-417b-a31a-263a328bae29-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8nw6r\" (UID: \"6ce9541a-aa1f-417b-a31a-263a328bae29\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8nw6r"
Sep 10 15:38:25.882547 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:38:25.882524    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ce9541a-aa1f-417b-a31a-263a328bae29-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8nw6r\" (UID: \"6ce9541a-aa1f-417b-a31a-263a328bae29\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8nw6r"
Sep 10 15:38:25.882888 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:38:25.882870    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ce9541a-aa1f-417b-a31a-263a328bae29-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8nw6r\" (UID: \"6ce9541a-aa1f-417b-a31a-263a328bae29\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8nw6r"
Sep 10 15:38:25.889517 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:38:25.889495    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vf9t4\" (UniqueName: \"kubernetes.io/projected/6ce9541a-aa1f-417b-a31a-263a328bae29-kube-api-access-vf9t4\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-8nw6r\" (UID: \"6ce9541a-aa1f-417b-a31a-263a328bae29\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8nw6r"
Sep 10 15:38:26.013671 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:38:26.013552    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8nw6r"
Sep 10 15:38:26.289837 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:38:26.289510    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8nw6r"]
Sep 10 15:38:26.292034 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:38:26.292003    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ce9541a_aa1f_417b_a31a_263a328bae29.slice/crio-bb2a6ecda40c48c1a999e6dcc8a67d40a7bb264b3c2cdee3c6aada01cbd8eb47 WatchSource:0}: Error finding container bb2a6ecda40c48c1a999e6dcc8a67d40a7bb264b3c2cdee3c6aada01cbd8eb47: Status 404 returned error can't find the container with id bb2a6ecda40c48c1a999e6dcc8a67d40a7bb264b3c2cdee3c6aada01cbd8eb47
Sep 10 15:38:26.293778 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:38:26.293758    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 15:38:26.813186 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:38:26.813146    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bs7hd"]
Sep 10 15:38:26.901003 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:38:26.900966    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8nw6r" event={"ID":"6ce9541a-aa1f-417b-a31a-263a328bae29","Type":"ContainerStarted","Data":"bb2a6ecda40c48c1a999e6dcc8a67d40a7bb264b3c2cdee3c6aada01cbd8eb47"}
Sep 10 15:38:27.907671 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:38:27.907556    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8nw6r" event={"ID":"6ce9541a-aa1f-417b-a31a-263a328bae29","Type":"ContainerStarted","Data":"46f0dd5efd50b2e599977321b9491eac050dd153db0133d11f9bd2968d10d636"}
Sep 10 15:38:27.927963 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:38:27.927907    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8nw6r" podStartSLOduration=1.564634872 podStartE2EDuration="2.927891659s" totalImagesPullingTime="1.363256787s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:38:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:38:27.92617999 +0000 UTC m=+683.668145093" watchObservedRunningTime="2026-09-10 15:38:27.927891659 +0000 UTC m=+683.669856762"
Sep 10 15:38:36.013754 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:38:36.013715    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8nw6r"
Sep 10 15:38:36.014152 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:38:36.013780    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8nw6r"
Sep 10 15:38:36.026719 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:38:36.026689    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8nw6r"
Sep 10 15:38:36.968657 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:38:36.968628    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-8nw6r"
Sep 10 15:38:40.313115 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:38:40.313071    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bs7hd"]
Sep 10 15:38:52.215611 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:38:52.215561    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwcp7h"]
Sep 10 15:38:52.220470 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:38:52.220453    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwcp7h"
Sep 10 15:38:52.223178 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:38:52.223156    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Sep 10 15:38:52.228882 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:38:52.228858    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwcp7h"]
Sep 10 15:38:52.305658 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:38:52.305629    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f5629ad-b3bb-4286-bd99-97ea74ff324c-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwcp7h\" (UID: \"1f5629ad-b3bb-4286-bd99-97ea74ff324c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwcp7h"
Sep 10 15:38:52.305658 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:38:52.305662    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f5629ad-b3bb-4286-bd99-97ea74ff324c-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwcp7h\" (UID: \"1f5629ad-b3bb-4286-bd99-97ea74ff324c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwcp7h"
Sep 10 15:38:52.305873 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:38:52.305730    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f5629ad-b3bb-4286-bd99-97ea74ff324c-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwcp7h\" (UID: \"1f5629ad-b3bb-4286-bd99-97ea74ff324c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwcp7h"
Sep 10 15:38:52.305873 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:38:52.305802    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f5629ad-b3bb-4286-bd99-97ea74ff324c-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwcp7h\" (UID: \"1f5629ad-b3bb-4286-bd99-97ea74ff324c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwcp7h"
Sep 10 15:38:52.305873 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:38:52.305845    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2ln2\" (UniqueName: \"kubernetes.io/projected/1f5629ad-b3bb-4286-bd99-97ea74ff324c-kube-api-access-l2ln2\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwcp7h\" (UID: \"1f5629ad-b3bb-4286-bd99-97ea74ff324c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwcp7h"
Sep 10 15:38:52.305873 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:38:52.305871    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f5629ad-b3bb-4286-bd99-97ea74ff324c-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwcp7h\" (UID: \"1f5629ad-b3bb-4286-bd99-97ea74ff324c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwcp7h"
Sep 10 15:38:52.406628 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:38:52.406552    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f5629ad-b3bb-4286-bd99-97ea74ff324c-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwcp7h\" (UID: \"1f5629ad-b3bb-4286-bd99-97ea74ff324c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwcp7h"
Sep 10 15:38:52.406874 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:38:52.406652    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2ln2\" (UniqueName: \"kubernetes.io/projected/1f5629ad-b3bb-4286-bd99-97ea74ff324c-kube-api-access-l2ln2\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwcp7h\" (UID: \"1f5629ad-b3bb-4286-bd99-97ea74ff324c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwcp7h"
Sep 10 15:38:52.406874 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:38:52.406695    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f5629ad-b3bb-4286-bd99-97ea74ff324c-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwcp7h\" (UID: \"1f5629ad-b3bb-4286-bd99-97ea74ff324c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwcp7h"
Sep 10 15:38:52.406874 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:38:52.406839    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f5629ad-b3bb-4286-bd99-97ea74ff324c-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwcp7h\" (UID: \"1f5629ad-b3bb-4286-bd99-97ea74ff324c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwcp7h"
Sep 10 15:38:52.407041 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:38:52.406893    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f5629ad-b3bb-4286-bd99-97ea74ff324c-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwcp7h\" (UID: \"1f5629ad-b3bb-4286-bd99-97ea74ff324c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwcp7h"
Sep 10 15:38:52.407041 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:38:52.406962    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f5629ad-b3bb-4286-bd99-97ea74ff324c-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwcp7h\" (UID: \"1f5629ad-b3bb-4286-bd99-97ea74ff324c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwcp7h"
Sep 10 15:38:52.407041 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:38:52.407034    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f5629ad-b3bb-4286-bd99-97ea74ff324c-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwcp7h\" (UID: \"1f5629ad-b3bb-4286-bd99-97ea74ff324c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwcp7h"
Sep 10 15:38:52.407238 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:38:52.407209    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f5629ad-b3bb-4286-bd99-97ea74ff324c-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwcp7h\" (UID: \"1f5629ad-b3bb-4286-bd99-97ea74ff324c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwcp7h"
Sep 10 15:38:52.407300 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:38:52.407269    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f5629ad-b3bb-4286-bd99-97ea74ff324c-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwcp7h\" (UID: \"1f5629ad-b3bb-4286-bd99-97ea74ff324c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwcp7h"
Sep 10 15:38:52.409005 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:38:52.408982    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f5629ad-b3bb-4286-bd99-97ea74ff324c-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwcp7h\" (UID: \"1f5629ad-b3bb-4286-bd99-97ea74ff324c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwcp7h"
Sep 10 15:38:52.409410 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:38:52.409364    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f5629ad-b3bb-4286-bd99-97ea74ff324c-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwcp7h\" (UID: \"1f5629ad-b3bb-4286-bd99-97ea74ff324c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwcp7h"
Sep 10 15:38:52.415703 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:38:52.415682    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2ln2\" (UniqueName: \"kubernetes.io/projected/1f5629ad-b3bb-4286-bd99-97ea74ff324c-kube-api-access-l2ln2\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwcp7h\" (UID: \"1f5629ad-b3bb-4286-bd99-97ea74ff324c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwcp7h"
Sep 10 15:38:52.531110 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:38:52.531003    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwcp7h"
Sep 10 15:38:52.807002 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:38:52.806977    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwcp7h"]
Sep 10 15:38:52.808967 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:38:52.808940    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f5629ad_b3bb_4286_bd99_97ea74ff324c.slice/crio-7e4ec048682de8a12ec6bb7888aeb2631b31684074bccffafd85c8ed08c1231b WatchSource:0}: Error finding container 7e4ec048682de8a12ec6bb7888aeb2631b31684074bccffafd85c8ed08c1231b: Status 404 returned error can't find the container with id 7e4ec048682de8a12ec6bb7888aeb2631b31684074bccffafd85c8ed08c1231b
Sep 10 15:38:53.042911 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:38:53.042870    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwcp7h" event={"ID":"1f5629ad-b3bb-4286-bd99-97ea74ff324c","Type":"ContainerStarted","Data":"7e4ec048682de8a12ec6bb7888aeb2631b31684074bccffafd85c8ed08c1231b"}
Sep 10 15:38:54.051278 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:38:54.051242    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwcp7h" event={"ID":"1f5629ad-b3bb-4286-bd99-97ea74ff324c","Type":"ContainerStarted","Data":"f9036afda7cc7f78ae1a0478309daf45936df39372bf2801e31832ecdf248168"}
Sep 10 15:38:54.070603 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:38:54.070533    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwcp7h" podStartSLOduration=1.8774890499999999 podStartE2EDuration="2.070517776s" totalImagesPullingTime="193.028726ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:38:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:38:54.069954111 +0000 UTC m=+709.811919215" watchObservedRunningTime="2026-09-10 15:38:54.070517776 +0000 UTC m=+709.812482880"
Sep 10 15:38:54.099460 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:38:54.099421    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bs7hd"]
Sep 10 15:39:02.531928 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:39:02.531883    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwcp7h"
Sep 10 15:39:02.532326 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:39:02.531957    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwcp7h"
Sep 10 15:39:02.544425 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:39:02.544392    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwcp7h"
Sep 10 15:39:03.106270 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:39:03.106239    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwcp7h"
Sep 10 15:39:13.206972 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:39:13.206938    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bs7hd"]
Sep 10 15:39:20.296932 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:39:20.296897    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-wmk4q"]
Sep 10 15:39:20.300685 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:39:20.300662    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wmk4q"
Sep 10 15:39:20.303203 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:39:20.303177    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 10 15:39:20.315972 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:39:20.315947    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-wmk4q"]
Sep 10 15:39:20.341444 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:39:20.341414    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f6fc26b-0880-4957-a0fd-199b0b7aa389-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-wmk4q\" (UID: \"1f6fc26b-0880-4957-a0fd-199b0b7aa389\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wmk4q"
Sep 10 15:39:20.341594 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:39:20.341469    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f6fc26b-0880-4957-a0fd-199b0b7aa389-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-wmk4q\" (UID: \"1f6fc26b-0880-4957-a0fd-199b0b7aa389\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wmk4q"
Sep 10 15:39:20.341594 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:39:20.341508    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f6fc26b-0880-4957-a0fd-199b0b7aa389-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-wmk4q\" (UID: \"1f6fc26b-0880-4957-a0fd-199b0b7aa389\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wmk4q"
Sep 10 15:39:20.341594 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:39:20.341563    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f6fc26b-0880-4957-a0fd-199b0b7aa389-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-wmk4q\" (UID: \"1f6fc26b-0880-4957-a0fd-199b0b7aa389\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wmk4q"
Sep 10 15:39:20.341758 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:39:20.341601    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f6fc26b-0880-4957-a0fd-199b0b7aa389-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-wmk4q\" (UID: \"1f6fc26b-0880-4957-a0fd-199b0b7aa389\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wmk4q"
Sep 10 15:39:20.341758 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:39:20.341657    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xk52s\" (UniqueName: \"kubernetes.io/projected/1f6fc26b-0880-4957-a0fd-199b0b7aa389-kube-api-access-xk52s\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-wmk4q\" (UID: \"1f6fc26b-0880-4957-a0fd-199b0b7aa389\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wmk4q"
Sep 10 15:39:20.442727 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:39:20.442692    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f6fc26b-0880-4957-a0fd-199b0b7aa389-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-wmk4q\" (UID: \"1f6fc26b-0880-4957-a0fd-199b0b7aa389\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wmk4q"
Sep 10 15:39:20.442727 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:39:20.442728    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f6fc26b-0880-4957-a0fd-199b0b7aa389-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-wmk4q\" (UID: \"1f6fc26b-0880-4957-a0fd-199b0b7aa389\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wmk4q"
Sep 10 15:39:20.442970 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:39:20.442769    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xk52s\" (UniqueName: \"kubernetes.io/projected/1f6fc26b-0880-4957-a0fd-199b0b7aa389-kube-api-access-xk52s\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-wmk4q\" (UID: \"1f6fc26b-0880-4957-a0fd-199b0b7aa389\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wmk4q"
Sep 10 15:39:20.442970 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:39:20.442809    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f6fc26b-0880-4957-a0fd-199b0b7aa389-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-wmk4q\" (UID: \"1f6fc26b-0880-4957-a0fd-199b0b7aa389\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wmk4q"
Sep 10 15:39:20.442970 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:39:20.442859    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f6fc26b-0880-4957-a0fd-199b0b7aa389-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-wmk4q\" (UID: \"1f6fc26b-0880-4957-a0fd-199b0b7aa389\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wmk4q"
Sep 10 15:39:20.442970 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:39:20.442894    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f6fc26b-0880-4957-a0fd-199b0b7aa389-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-wmk4q\" (UID: \"1f6fc26b-0880-4957-a0fd-199b0b7aa389\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wmk4q"
Sep 10 15:39:20.443301 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:39:20.443275    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f6fc26b-0880-4957-a0fd-199b0b7aa389-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-wmk4q\" (UID: \"1f6fc26b-0880-4957-a0fd-199b0b7aa389\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wmk4q"
Sep 10 15:39:20.443406 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:39:20.443329    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f6fc26b-0880-4957-a0fd-199b0b7aa389-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-wmk4q\" (UID: \"1f6fc26b-0880-4957-a0fd-199b0b7aa389\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wmk4q"
Sep 10 15:39:20.443406 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:39:20.443348    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f6fc26b-0880-4957-a0fd-199b0b7aa389-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-wmk4q\" (UID: \"1f6fc26b-0880-4957-a0fd-199b0b7aa389\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wmk4q"
Sep 10 15:39:20.445346 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:39:20.445326    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f6fc26b-0880-4957-a0fd-199b0b7aa389-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-wmk4q\" (UID: \"1f6fc26b-0880-4957-a0fd-199b0b7aa389\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wmk4q"
Sep 10 15:39:20.445518 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:39:20.445499    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f6fc26b-0880-4957-a0fd-199b0b7aa389-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-wmk4q\" (UID: \"1f6fc26b-0880-4957-a0fd-199b0b7aa389\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wmk4q"
Sep 10 15:39:20.452751 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:39:20.452726    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xk52s\" (UniqueName: \"kubernetes.io/projected/1f6fc26b-0880-4957-a0fd-199b0b7aa389-kube-api-access-xk52s\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-wmk4q\" (UID: \"1f6fc26b-0880-4957-a0fd-199b0b7aa389\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wmk4q"
Sep 10 15:39:20.609204 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:39:20.609120    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wmk4q"
Sep 10 15:39:20.880992 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:39:20.880968    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-wmk4q"]
Sep 10 15:39:20.882717 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:39:20.882678    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1f6fc26b_0880_4957_a0fd_199b0b7aa389.slice/crio-c716ba68f341cda130a1a89969da5147c87be809667e358f517b594ec3d012f4 WatchSource:0}: Error finding container c716ba68f341cda130a1a89969da5147c87be809667e358f517b594ec3d012f4: Status 404 returned error can't find the container with id c716ba68f341cda130a1a89969da5147c87be809667e358f517b594ec3d012f4
Sep 10 15:39:21.192176 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:39:21.192141    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wmk4q" event={"ID":"1f6fc26b-0880-4957-a0fd-199b0b7aa389","Type":"ContainerStarted","Data":"2fb3b625968d9683a5fc0c75d09c62b6c0b4a53635e101369915e6eb9a5ddf56"}
Sep 10 15:39:21.192325 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:39:21.192186    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wmk4q" event={"ID":"1f6fc26b-0880-4957-a0fd-199b0b7aa389","Type":"ContainerStarted","Data":"c716ba68f341cda130a1a89969da5147c87be809667e358f517b594ec3d012f4"}
Sep 10 15:39:21.212376 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:39:21.212324    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wmk4q" podStartSLOduration=1.009552414 podStartE2EDuration="1.212305072s" totalImagesPullingTime="202.752658ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:39:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:39:21.211170401 +0000 UTC m=+736.953135505" watchObservedRunningTime="2026-09-10 15:39:21.212305072 +0000 UTC m=+736.954270181"
Sep 10 15:39:22.607539 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:39:22.607502    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bs7hd"]
Sep 10 15:39:30.609452 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:39:30.609415    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wmk4q"
Sep 10 15:39:30.610043 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:39:30.609494    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wmk4q"
Sep 10 15:39:30.621956 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:39:30.621934    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wmk4q"
Sep 10 15:39:31.254620 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:39:31.254593    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wmk4q"
Sep 10 15:41:29.458143 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:29.458099    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7761-8a46de"]
Sep 10 15:41:29.462220 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:29.462184    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7761-8a46de"
Sep 10 15:41:29.465167 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:29.465144    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-wtngs\""
Sep 10 15:41:29.465344 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:29.465322    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7761-8a46de"]
Sep 10 15:41:29.557178 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:29.557141    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fc9ch\" (UniqueName: \"kubernetes.io/projected/1c05f6e0-4227-441a-a1ed-7d29c9be0ea5-kube-api-access-fc9ch\") pod \"test-curl-7761-8a46de\" (UID: \"1c05f6e0-4227-441a-a1ed-7d29c9be0ea5\") " pod="opendatahub/test-curl-7761-8a46de"
Sep 10 15:41:29.658454 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:29.658417    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fc9ch\" (UniqueName: \"kubernetes.io/projected/1c05f6e0-4227-441a-a1ed-7d29c9be0ea5-kube-api-access-fc9ch\") pod \"test-curl-7761-8a46de\" (UID: \"1c05f6e0-4227-441a-a1ed-7d29c9be0ea5\") " pod="opendatahub/test-curl-7761-8a46de"
Sep 10 15:41:29.668774 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:29.668745    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fc9ch\" (UniqueName: \"kubernetes.io/projected/1c05f6e0-4227-441a-a1ed-7d29c9be0ea5-kube-api-access-fc9ch\") pod \"test-curl-7761-8a46de\" (UID: \"1c05f6e0-4227-441a-a1ed-7d29c9be0ea5\") " pod="opendatahub/test-curl-7761-8a46de"
Sep 10 15:41:29.772909 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:29.772820    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7761-8a46de"
Sep 10 15:41:30.047018 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:30.046986    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7761-8a46de"]
Sep 10 15:41:30.048285 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:41:30.048259    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1c05f6e0_4227_441a_a1ed_7d29c9be0ea5.slice/crio-ee7f1f054deb6afcb58b8053b02a63d4e4834df4d5686549638e243671500fbd WatchSource:0}: Error finding container ee7f1f054deb6afcb58b8053b02a63d4e4834df4d5686549638e243671500fbd: Status 404 returned error can't find the container with id ee7f1f054deb6afcb58b8053b02a63d4e4834df4d5686549638e243671500fbd
Sep 10 15:41:30.861073 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:30.861038    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7761-8a46de" event={"ID":"1c05f6e0-4227-441a-a1ed-7d29c9be0ea5","Type":"ContainerStarted","Data":"ee7f1f054deb6afcb58b8053b02a63d4e4834df4d5686549638e243671500fbd"}
Sep 10 15:41:31.405694 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:31.405650    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bs7hd"]
Sep 10 15:41:33.879570 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:33.879467    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7761-8a46de" event={"ID":"1c05f6e0-4227-441a-a1ed-7d29c9be0ea5","Type":"ContainerStarted","Data":"c1b8829a3fd0a46115a4f26d394ccbd32fa9c48be4a4faf6f7ea9de12310c07e"}
Sep 10 15:41:33.897010 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:33.896953    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7761-8a46de" podStartSLOduration=1.462253458 podStartE2EDuration="4.896935496s" totalImagesPullingTime="3.434682038s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:41:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:41:33.896061606 +0000 UTC m=+869.638026718" watchObservedRunningTime="2026-09-10 15:41:33.896935496 +0000 UTC m=+869.638900599"
Sep 10 15:41:34.484386 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:34.484347    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7761-8a46de"]
Sep 10 15:41:34.488896 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:34.488870    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7761-8a46de"]
Sep 10 15:41:34.599187 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:34.599106    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7761-656ed8"]
Sep 10 15:41:34.599496 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:34.599484    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c05f6e0-4227-441a-a1ed-7d29c9be0ea5" containerName="test-curl-7761-8a46de"
Sep 10 15:41:34.599539 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:34.599498    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c05f6e0-4227-441a-a1ed-7d29c9be0ea5" containerName="test-curl-7761-8a46de"
Sep 10 15:41:34.599573 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:34.599562    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c05f6e0-4227-441a-a1ed-7d29c9be0ea5" containerName="test-curl-7761-8a46de"
Sep 10 15:41:34.602675 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:34.602660    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7761-656ed8"
Sep 10 15:41:34.610518 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:34.610095    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7761-656ed8"]
Sep 10 15:41:34.704693 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:34.704661    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kb68m\" (UniqueName: \"kubernetes.io/projected/902011f5-1c3a-4749-a0b5-3b6bad3c0905-kube-api-access-kb68m\") pod \"test-curl-7761-656ed8\" (UID: \"902011f5-1c3a-4749-a0b5-3b6bad3c0905\") " pod="opendatahub/test-curl-7761-656ed8"
Sep 10 15:41:34.805726 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:34.805696    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kb68m\" (UniqueName: \"kubernetes.io/projected/902011f5-1c3a-4749-a0b5-3b6bad3c0905-kube-api-access-kb68m\") pod \"test-curl-7761-656ed8\" (UID: \"902011f5-1c3a-4749-a0b5-3b6bad3c0905\") " pod="opendatahub/test-curl-7761-656ed8"
Sep 10 15:41:34.814305 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:34.814273    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kb68m\" (UniqueName: \"kubernetes.io/projected/902011f5-1c3a-4749-a0b5-3b6bad3c0905-kube-api-access-kb68m\") pod \"test-curl-7761-656ed8\" (UID: \"902011f5-1c3a-4749-a0b5-3b6bad3c0905\") " pod="opendatahub/test-curl-7761-656ed8"
Sep 10 15:41:34.884903 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:41:34.884823    2559 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7761-8a46de" container="test-curl-7761-8a46de"
Sep 10 15:41:34.913009 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:34.912983    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7761-656ed8"
Sep 10 15:41:35.185044 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:35.185019    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7761-656ed8"]
Sep 10 15:41:35.187074 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:41:35.187044    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod902011f5_1c3a_4749_a0b5_3b6bad3c0905.slice/crio-a61b5c893401a2cd4ba577bd20f6711ae83c689ff649ca4b7fe664bc2b52f224 WatchSource:0}: Error finding container a61b5c893401a2cd4ba577bd20f6711ae83c689ff649ca4b7fe664bc2b52f224: Status 404 returned error can't find the container with id a61b5c893401a2cd4ba577bd20f6711ae83c689ff649ca4b7fe664bc2b52f224
Sep 10 15:41:35.894035 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:35.893997    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7761-656ed8" event={"ID":"902011f5-1c3a-4749-a0b5-3b6bad3c0905","Type":"ContainerStarted","Data":"6616ca855833d1e82727a621f3e3d461cda45452d45d6120684146816768ca41"}
Sep 10 15:41:35.894485 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:35.894047    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7761-656ed8" event={"ID":"902011f5-1c3a-4749-a0b5-3b6bad3c0905","Type":"ContainerStarted","Data":"a61b5c893401a2cd4ba577bd20f6711ae83c689ff649ca4b7fe664bc2b52f224"}
Sep 10 15:41:35.894485 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:35.894097    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7761-8a46de" podUID="1c05f6e0-4227-441a-a1ed-7d29c9be0ea5" containerName="test-curl-7761-8a46de" containerID="cri-o://c1b8829a3fd0a46115a4f26d394ccbd32fa9c48be4a4faf6f7ea9de12310c07e" gracePeriod=2
Sep 10 15:41:35.909770 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:35.909725    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7761-656ed8" podStartSLOduration=1.909709072 podStartE2EDuration="1.909709072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:41:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:41:35.908369709 +0000 UTC m=+871.650334812" watchObservedRunningTime="2026-09-10 15:41:35.909709072 +0000 UTC m=+871.651674175"
Sep 10 15:41:36.504470 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:36.504430    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7761-656ed8"]
Sep 10 15:41:36.509148 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:36.509118    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7761-656ed8"]
Sep 10 15:41:36.905156 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:41:36.905126    2559 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7761-656ed8" container="test-curl-7761-656ed8"
Sep 10 15:41:37.911358 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:37.911318    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7761-656ed8" podUID="902011f5-1c3a-4749-a0b5-3b6bad3c0905" containerName="test-curl-7761-656ed8" containerID="cri-o://6616ca855833d1e82727a621f3e3d461cda45452d45d6120684146816768ca41" gracePeriod=2
Sep 10 15:41:38.075206 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:38.075173    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7761-8a46de"
Sep 10 15:41:38.237246 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:38.237208    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fc9ch\" (UniqueName: \"kubernetes.io/projected/1c05f6e0-4227-441a-a1ed-7d29c9be0ea5-kube-api-access-fc9ch\") pod \"1c05f6e0-4227-441a-a1ed-7d29c9be0ea5\" (UID: \"1c05f6e0-4227-441a-a1ed-7d29c9be0ea5\") "
Sep 10 15:41:38.239379 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:38.239356    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c05f6e0-4227-441a-a1ed-7d29c9be0ea5-kube-api-access-fc9ch" (OuterVolumeSpecName: "kube-api-access-fc9ch") pod "1c05f6e0-4227-441a-a1ed-7d29c9be0ea5" (UID: "1c05f6e0-4227-441a-a1ed-7d29c9be0ea5"). InnerVolumeSpecName "kube-api-access-fc9ch". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:41:38.338800 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:38.338765    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fc9ch\" (UniqueName: \"kubernetes.io/projected/1c05f6e0-4227-441a-a1ed-7d29c9be0ea5-kube-api-access-fc9ch\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:41:38.785668 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:38.785634    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c05f6e0-4227-441a-a1ed-7d29c9be0ea5" path="/var/lib/kubelet/pods/1c05f6e0-4227-441a-a1ed-7d29c9be0ea5/volumes"
Sep 10 15:41:38.917746 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:38.917708    2559 generic.go:356] "Generic (PLEG): container finished" podID="1c05f6e0-4227-441a-a1ed-7d29c9be0ea5" containerID="c1b8829a3fd0a46115a4f26d394ccbd32fa9c48be4a4faf6f7ea9de12310c07e" exitCode=137
Sep 10 15:41:38.918145 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:38.917752    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7761-8a46de"
Sep 10 15:41:38.918145 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:38.917807    2559 scope.go:117] "RemoveContainer" containerID="c1b8829a3fd0a46115a4f26d394ccbd32fa9c48be4a4faf6f7ea9de12310c07e"
Sep 10 15:41:38.926358 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:38.926340    2559 scope.go:117] "RemoveContainer" containerID="c1b8829a3fd0a46115a4f26d394ccbd32fa9c48be4a4faf6f7ea9de12310c07e"
Sep 10 15:41:38.926635 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:41:38.926606    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1b8829a3fd0a46115a4f26d394ccbd32fa9c48be4a4faf6f7ea9de12310c07e\": container with ID starting with c1b8829a3fd0a46115a4f26d394ccbd32fa9c48be4a4faf6f7ea9de12310c07e not found: ID does not exist" containerID="c1b8829a3fd0a46115a4f26d394ccbd32fa9c48be4a4faf6f7ea9de12310c07e"
Sep 10 15:41:38.926731 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:38.926641    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1b8829a3fd0a46115a4f26d394ccbd32fa9c48be4a4faf6f7ea9de12310c07e"} err="failed to get container status \"c1b8829a3fd0a46115a4f26d394ccbd32fa9c48be4a4faf6f7ea9de12310c07e\": rpc error: code = NotFound desc = could not find container \"c1b8829a3fd0a46115a4f26d394ccbd32fa9c48be4a4faf6f7ea9de12310c07e\": container with ID starting with c1b8829a3fd0a46115a4f26d394ccbd32fa9c48be4a4faf6f7ea9de12310c07e not found: ID does not exist"
Sep 10 15:41:40.092967 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:40.092937    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7761-656ed8"
Sep 10 15:41:40.256890 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:40.256796    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kb68m\" (UniqueName: \"kubernetes.io/projected/902011f5-1c3a-4749-a0b5-3b6bad3c0905-kube-api-access-kb68m\") pod \"902011f5-1c3a-4749-a0b5-3b6bad3c0905\" (UID: \"902011f5-1c3a-4749-a0b5-3b6bad3c0905\") "
Sep 10 15:41:40.258910 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:40.258888    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/902011f5-1c3a-4749-a0b5-3b6bad3c0905-kube-api-access-kb68m" (OuterVolumeSpecName: "kube-api-access-kb68m") pod "902011f5-1c3a-4749-a0b5-3b6bad3c0905" (UID: "902011f5-1c3a-4749-a0b5-3b6bad3c0905"). InnerVolumeSpecName "kube-api-access-kb68m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:41:40.358186 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:40.358147    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kb68m\" (UniqueName: \"kubernetes.io/projected/902011f5-1c3a-4749-a0b5-3b6bad3c0905-kube-api-access-kb68m\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:41:40.785144 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:40.785099    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="902011f5-1c3a-4749-a0b5-3b6bad3c0905" path="/var/lib/kubelet/pods/902011f5-1c3a-4749-a0b5-3b6bad3c0905/volumes"
Sep 10 15:41:40.931193 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:40.931155    2559 generic.go:356] "Generic (PLEG): container finished" podID="902011f5-1c3a-4749-a0b5-3b6bad3c0905" containerID="6616ca855833d1e82727a621f3e3d461cda45452d45d6120684146816768ca41" exitCode=137
Sep 10 15:41:40.931368 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:40.931197    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7761-656ed8"
Sep 10 15:41:40.931368 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:40.931216    2559 scope.go:117] "RemoveContainer" containerID="6616ca855833d1e82727a621f3e3d461cda45452d45d6120684146816768ca41"
Sep 10 15:41:40.939836 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:40.939813    2559 scope.go:117] "RemoveContainer" containerID="6616ca855833d1e82727a621f3e3d461cda45452d45d6120684146816768ca41"
Sep 10 15:41:40.940177 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:41:40.940154    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6616ca855833d1e82727a621f3e3d461cda45452d45d6120684146816768ca41\": container with ID starting with 6616ca855833d1e82727a621f3e3d461cda45452d45d6120684146816768ca41 not found: ID does not exist" containerID="6616ca855833d1e82727a621f3e3d461cda45452d45d6120684146816768ca41"
Sep 10 15:41:40.940265 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:40.940188    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6616ca855833d1e82727a621f3e3d461cda45452d45d6120684146816768ca41"} err="failed to get container status \"6616ca855833d1e82727a621f3e3d461cda45452d45d6120684146816768ca41\": rpc error: code = NotFound desc = could not find container \"6616ca855833d1e82727a621f3e3d461cda45452d45d6120684146816768ca41\": container with ID starting with 6616ca855833d1e82727a621f3e3d461cda45452d45d6120684146816768ca41 not found: ID does not exist"
Sep 10 15:41:43.902294 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:43.902197    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-bd5806-9d646cccd-pxjzq"]
Sep 10 15:41:43.902772 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:43.902643    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="902011f5-1c3a-4749-a0b5-3b6bad3c0905" containerName="test-curl-7761-656ed8"
Sep 10 15:41:43.902772 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:43.902658    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="902011f5-1c3a-4749-a0b5-3b6bad3c0905" containerName="test-curl-7761-656ed8"
Sep 10 15:41:43.902772 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:43.902733    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="902011f5-1c3a-4749-a0b5-3b6bad3c0905" containerName="test-curl-7761-656ed8"
Sep 10 15:41:43.907136 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:43.907116    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-bd5806-9d646cccd-pxjzq"
Sep 10 15:41:43.909908 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:43.909875    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 10 15:41:43.910144 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:43.909905    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 15:41:43.910144 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:43.909881    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-bd5806-serving-cert\""
Sep 10 15:41:43.911253 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:43.911234    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-5qzmv\""
Sep 10 15:41:43.911360 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:43.911299    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 10 15:41:43.918067 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:43.918036    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-bd5806-9d646cccd-pxjzq"]
Sep 10 15:41:44.091021 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:44.090984    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cdbedba-07c4-48fa-bbab-167fe45cbc20-openshift-service-ca\") pod \"maas-api-e2e-autores-bd5806-9d646cccd-pxjzq\" (UID: \"4cdbedba-07c4-48fa-bbab-167fe45cbc20\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-bd5806-9d646cccd-pxjzq"
Sep 10 15:41:44.091199 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:44.091035    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4cdbedba-07c4-48fa-bbab-167fe45cbc20-maas-api-tls\") pod \"maas-api-e2e-autores-bd5806-9d646cccd-pxjzq\" (UID: \"4cdbedba-07c4-48fa-bbab-167fe45cbc20\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-bd5806-9d646cccd-pxjzq"
Sep 10 15:41:44.091199 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:44.091085    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brm5x\" (UniqueName: \"kubernetes.io/projected/4cdbedba-07c4-48fa-bbab-167fe45cbc20-kube-api-access-brm5x\") pod \"maas-api-e2e-autores-bd5806-9d646cccd-pxjzq\" (UID: \"4cdbedba-07c4-48fa-bbab-167fe45cbc20\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-bd5806-9d646cccd-pxjzq"
Sep 10 15:41:44.091199 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:44.091172    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4cdbedba-07c4-48fa-bbab-167fe45cbc20-metrics-certs\") pod \"maas-api-e2e-autores-bd5806-9d646cccd-pxjzq\" (UID: \"4cdbedba-07c4-48fa-bbab-167fe45cbc20\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-bd5806-9d646cccd-pxjzq"
Sep 10 15:41:44.191768 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:44.191735    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cdbedba-07c4-48fa-bbab-167fe45cbc20-openshift-service-ca\") pod \"maas-api-e2e-autores-bd5806-9d646cccd-pxjzq\" (UID: \"4cdbedba-07c4-48fa-bbab-167fe45cbc20\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-bd5806-9d646cccd-pxjzq"
Sep 10 15:41:44.191938 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:44.191786    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4cdbedba-07c4-48fa-bbab-167fe45cbc20-maas-api-tls\") pod \"maas-api-e2e-autores-bd5806-9d646cccd-pxjzq\" (UID: \"4cdbedba-07c4-48fa-bbab-167fe45cbc20\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-bd5806-9d646cccd-pxjzq"
Sep 10 15:41:44.191938 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:44.191824    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brm5x\" (UniqueName: \"kubernetes.io/projected/4cdbedba-07c4-48fa-bbab-167fe45cbc20-kube-api-access-brm5x\") pod \"maas-api-e2e-autores-bd5806-9d646cccd-pxjzq\" (UID: \"4cdbedba-07c4-48fa-bbab-167fe45cbc20\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-bd5806-9d646cccd-pxjzq"
Sep 10 15:41:44.191938 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:44.191888    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4cdbedba-07c4-48fa-bbab-167fe45cbc20-metrics-certs\") pod \"maas-api-e2e-autores-bd5806-9d646cccd-pxjzq\" (UID: \"4cdbedba-07c4-48fa-bbab-167fe45cbc20\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-bd5806-9d646cccd-pxjzq"
Sep 10 15:41:44.192515 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:44.192494    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cdbedba-07c4-48fa-bbab-167fe45cbc20-openshift-service-ca\") pod \"maas-api-e2e-autores-bd5806-9d646cccd-pxjzq\" (UID: \"4cdbedba-07c4-48fa-bbab-167fe45cbc20\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-bd5806-9d646cccd-pxjzq"
Sep 10 15:41:44.194393 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:44.194357    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4cdbedba-07c4-48fa-bbab-167fe45cbc20-metrics-certs\") pod \"maas-api-e2e-autores-bd5806-9d646cccd-pxjzq\" (UID: \"4cdbedba-07c4-48fa-bbab-167fe45cbc20\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-bd5806-9d646cccd-pxjzq"
Sep 10 15:41:44.194508 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:44.194357    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4cdbedba-07c4-48fa-bbab-167fe45cbc20-maas-api-tls\") pod \"maas-api-e2e-autores-bd5806-9d646cccd-pxjzq\" (UID: \"4cdbedba-07c4-48fa-bbab-167fe45cbc20\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-bd5806-9d646cccd-pxjzq"
Sep 10 15:41:44.201463 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:44.201437    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-brm5x\" (UniqueName: \"kubernetes.io/projected/4cdbedba-07c4-48fa-bbab-167fe45cbc20-kube-api-access-brm5x\") pod \"maas-api-e2e-autores-bd5806-9d646cccd-pxjzq\" (UID: \"4cdbedba-07c4-48fa-bbab-167fe45cbc20\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-bd5806-9d646cccd-pxjzq"
Sep 10 15:41:44.219249 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:44.219218    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-bd5806-9d646cccd-pxjzq"
Sep 10 15:41:44.504116 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:44.504086    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-bd5806-9d646cccd-pxjzq"]
Sep 10 15:41:44.505691 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:41:44.505662    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cdbedba_07c4_48fa_bbab_167fe45cbc20.slice/crio-83bfe95cb5ac3284b79866532485a2193bbb78aeb78e773c88ca6e757e206a18 WatchSource:0}: Error finding container 83bfe95cb5ac3284b79866532485a2193bbb78aeb78e773c88ca6e757e206a18: Status 404 returned error can't find the container with id 83bfe95cb5ac3284b79866532485a2193bbb78aeb78e773c88ca6e757e206a18
Sep 10 15:41:44.956362 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:44.956327    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-bd5806-9d646cccd-pxjzq" event={"ID":"4cdbedba-07c4-48fa-bbab-167fe45cbc20","Type":"ContainerStarted","Data":"83bfe95cb5ac3284b79866532485a2193bbb78aeb78e773c88ca6e757e206a18"}
Sep 10 15:41:45.717296 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:45.717259    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bs7hd"]
Sep 10 15:41:46.235341 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:46.235308    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-90ca97-openshift-default-659f84d46b-9l4kj"]
Sep 10 15:41:46.240392 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:46.240367    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-90ca97-openshift-default-659f84d46b-9l4kj"
Sep 10 15:41:46.243102 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:46.243069    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 10 15:41:46.243102 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:46.243084    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-90ca97-openshift-default-dockercfg-2rzqz\""
Sep 10 15:41:46.254941 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:46.254916    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-90ca97-openshift-default-659f84d46b-9l4kj"]
Sep 10 15:41:46.413485 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:46.413449    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af-workload-socket\") pod \"e2e-inf-b-90ca97-openshift-default-659f84d46b-9l4kj\" (UID: \"f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af\") " pod="openshift-ingress/e2e-inf-b-90ca97-openshift-default-659f84d46b-9l4kj"
Sep 10 15:41:46.413672 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:46.413503    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75ks8\" (UniqueName: \"kubernetes.io/projected/f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af-kube-api-access-75ks8\") pod \"e2e-inf-b-90ca97-openshift-default-659f84d46b-9l4kj\" (UID: \"f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af\") " pod="openshift-ingress/e2e-inf-b-90ca97-openshift-default-659f84d46b-9l4kj"
Sep 10 15:41:46.413672 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:46.413528    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af-istio-token\") pod \"e2e-inf-b-90ca97-openshift-default-659f84d46b-9l4kj\" (UID: \"f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af\") " pod="openshift-ingress/e2e-inf-b-90ca97-openshift-default-659f84d46b-9l4kj"
Sep 10 15:41:46.413672 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:46.413553    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af-workload-certs\") pod \"e2e-inf-b-90ca97-openshift-default-659f84d46b-9l4kj\" (UID: \"f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af\") " pod="openshift-ingress/e2e-inf-b-90ca97-openshift-default-659f84d46b-9l4kj"
Sep 10 15:41:46.413672 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:46.413573    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af-istiod-ca-cert\") pod \"e2e-inf-b-90ca97-openshift-default-659f84d46b-9l4kj\" (UID: \"f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af\") " pod="openshift-ingress/e2e-inf-b-90ca97-openshift-default-659f84d46b-9l4kj"
Sep 10 15:41:46.413672 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:46.413658    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af-istio-data\") pod \"e2e-inf-b-90ca97-openshift-default-659f84d46b-9l4kj\" (UID: \"f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af\") " pod="openshift-ingress/e2e-inf-b-90ca97-openshift-default-659f84d46b-9l4kj"
Sep 10 15:41:46.413874 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:46.413707    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af-credential-socket\") pod \"e2e-inf-b-90ca97-openshift-default-659f84d46b-9l4kj\" (UID: \"f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af\") " pod="openshift-ingress/e2e-inf-b-90ca97-openshift-default-659f84d46b-9l4kj"
Sep 10 15:41:46.413874 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:46.413724    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af-istio-podinfo\") pod \"e2e-inf-b-90ca97-openshift-default-659f84d46b-9l4kj\" (UID: \"f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af\") " pod="openshift-ingress/e2e-inf-b-90ca97-openshift-default-659f84d46b-9l4kj"
Sep 10 15:41:46.413874 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:46.413740    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af-istio-envoy\") pod \"e2e-inf-b-90ca97-openshift-default-659f84d46b-9l4kj\" (UID: \"f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af\") " pod="openshift-ingress/e2e-inf-b-90ca97-openshift-default-659f84d46b-9l4kj"
Sep 10 15:41:46.514704 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:46.514606    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75ks8\" (UniqueName: \"kubernetes.io/projected/f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af-kube-api-access-75ks8\") pod \"e2e-inf-b-90ca97-openshift-default-659f84d46b-9l4kj\" (UID: \"f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af\") " pod="openshift-ingress/e2e-inf-b-90ca97-openshift-default-659f84d46b-9l4kj"
Sep 10 15:41:46.514704 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:46.514664    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af-istio-token\") pod \"e2e-inf-b-90ca97-openshift-default-659f84d46b-9l4kj\" (UID: \"f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af\") " pod="openshift-ingress/e2e-inf-b-90ca97-openshift-default-659f84d46b-9l4kj"
Sep 10 15:41:46.514927 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:46.514715    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af-workload-certs\") pod \"e2e-inf-b-90ca97-openshift-default-659f84d46b-9l4kj\" (UID: \"f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af\") " pod="openshift-ingress/e2e-inf-b-90ca97-openshift-default-659f84d46b-9l4kj"
Sep 10 15:41:46.514927 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:46.514744    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af-istiod-ca-cert\") pod \"e2e-inf-b-90ca97-openshift-default-659f84d46b-9l4kj\" (UID: \"f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af\") " pod="openshift-ingress/e2e-inf-b-90ca97-openshift-default-659f84d46b-9l4kj"
Sep 10 15:41:46.514927 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:46.514778    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af-istio-data\") pod \"e2e-inf-b-90ca97-openshift-default-659f84d46b-9l4kj\" (UID: \"f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af\") " pod="openshift-ingress/e2e-inf-b-90ca97-openshift-default-659f84d46b-9l4kj"
Sep 10 15:41:46.514927 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:46.514823    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af-credential-socket\") pod \"e2e-inf-b-90ca97-openshift-default-659f84d46b-9l4kj\" (UID: \"f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af\") " pod="openshift-ingress/e2e-inf-b-90ca97-openshift-default-659f84d46b-9l4kj"
Sep 10 15:41:46.514927 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:46.514847    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af-istio-podinfo\") pod \"e2e-inf-b-90ca97-openshift-default-659f84d46b-9l4kj\" (UID: \"f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af\") " pod="openshift-ingress/e2e-inf-b-90ca97-openshift-default-659f84d46b-9l4kj"
Sep 10 15:41:46.514927 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:46.514875    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af-istio-envoy\") pod \"e2e-inf-b-90ca97-openshift-default-659f84d46b-9l4kj\" (UID: \"f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af\") " pod="openshift-ingress/e2e-inf-b-90ca97-openshift-default-659f84d46b-9l4kj"
Sep 10 15:41:46.515222 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:46.514931    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af-workload-socket\") pod \"e2e-inf-b-90ca97-openshift-default-659f84d46b-9l4kj\" (UID: \"f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af\") " pod="openshift-ingress/e2e-inf-b-90ca97-openshift-default-659f84d46b-9l4kj"
Sep 10 15:41:46.515463 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:46.515362    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af-istio-data\") pod \"e2e-inf-b-90ca97-openshift-default-659f84d46b-9l4kj\" (UID: \"f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af\") " pod="openshift-ingress/e2e-inf-b-90ca97-openshift-default-659f84d46b-9l4kj"
Sep 10 15:41:46.515463 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:46.515391    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af-workload-socket\") pod \"e2e-inf-b-90ca97-openshift-default-659f84d46b-9l4kj\" (UID: \"f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af\") " pod="openshift-ingress/e2e-inf-b-90ca97-openshift-default-659f84d46b-9l4kj"
Sep 10 15:41:46.515682 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:46.515646    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af-workload-certs\") pod \"e2e-inf-b-90ca97-openshift-default-659f84d46b-9l4kj\" (UID: \"f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af\") " pod="openshift-ingress/e2e-inf-b-90ca97-openshift-default-659f84d46b-9l4kj"
Sep 10 15:41:46.515791 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:46.515744    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af-credential-socket\") pod \"e2e-inf-b-90ca97-openshift-default-659f84d46b-9l4kj\" (UID: \"f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af\") " pod="openshift-ingress/e2e-inf-b-90ca97-openshift-default-659f84d46b-9l4kj"
Sep 10 15:41:46.516350 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:46.516327    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af-istiod-ca-cert\") pod \"e2e-inf-b-90ca97-openshift-default-659f84d46b-9l4kj\" (UID: \"f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af\") " pod="openshift-ingress/e2e-inf-b-90ca97-openshift-default-659f84d46b-9l4kj"
Sep 10 15:41:46.517903 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:46.517884    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af-istio-envoy\") pod \"e2e-inf-b-90ca97-openshift-default-659f84d46b-9l4kj\" (UID: \"f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af\") " pod="openshift-ingress/e2e-inf-b-90ca97-openshift-default-659f84d46b-9l4kj"
Sep 10 15:41:46.518002 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:46.517979    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af-istio-podinfo\") pod \"e2e-inf-b-90ca97-openshift-default-659f84d46b-9l4kj\" (UID: \"f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af\") " pod="openshift-ingress/e2e-inf-b-90ca97-openshift-default-659f84d46b-9l4kj"
Sep 10 15:41:46.526404 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:46.526371    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af-istio-token\") pod \"e2e-inf-b-90ca97-openshift-default-659f84d46b-9l4kj\" (UID: \"f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af\") " pod="openshift-ingress/e2e-inf-b-90ca97-openshift-default-659f84d46b-9l4kj"
Sep 10 15:41:46.526688 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:46.526633    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-75ks8\" (UniqueName: \"kubernetes.io/projected/f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af-kube-api-access-75ks8\") pod \"e2e-inf-b-90ca97-openshift-default-659f84d46b-9l4kj\" (UID: \"f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af\") " pod="openshift-ingress/e2e-inf-b-90ca97-openshift-default-659f84d46b-9l4kj"
Sep 10 15:41:46.553127 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:46.553091    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-90ca97-openshift-default-659f84d46b-9l4kj"
Sep 10 15:41:46.848682 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:46.848592    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-90ca97-openshift-default-659f84d46b-9l4kj"]
Sep 10 15:41:46.854742 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:41:46.854703    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf318062b_f0b7_4baa_9c3c_3ec9d4f9f5af.slice/crio-149bf37e1f5ee477d66639d7cb3c2f5507b81a6d5549147fde960165e48b62c5 WatchSource:0}: Error finding container 149bf37e1f5ee477d66639d7cb3c2f5507b81a6d5549147fde960165e48b62c5: Status 404 returned error can't find the container with id 149bf37e1f5ee477d66639d7cb3c2f5507b81a6d5549147fde960165e48b62c5
Sep 10 15:41:46.974533 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:46.974486    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-90ca97-openshift-default-659f84d46b-9l4kj" event={"ID":"f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af","Type":"ContainerStarted","Data":"149bf37e1f5ee477d66639d7cb3c2f5507b81a6d5549147fde960165e48b62c5"}
Sep 10 15:41:48.989180 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:48.989130    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-bd5806-9d646cccd-pxjzq" event={"ID":"4cdbedba-07c4-48fa-bbab-167fe45cbc20","Type":"ContainerStarted","Data":"9e53cfa4e6281b417e131e6b0be63f315dba751640afc26b9e41b08fcdbf04cd"}
Sep 10 15:41:48.989673 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:48.989306    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-bd5806-9d646cccd-pxjzq"
Sep 10 15:41:49.010492 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:49.010434    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-bd5806-9d646cccd-pxjzq" podStartSLOduration=2.210151179 podStartE2EDuration="6.010416383s" totalImagesPullingTime="3.800265204s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:41:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:41:49.009569935 +0000 UTC m=+884.751535038" watchObservedRunningTime="2026-09-10 15:41:49.010416383 +0000 UTC m=+884.752381487"
Sep 10 15:41:50.285756 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:50.285721    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 15:41:50.286002 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:50.285793    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 15:41:50.286002 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:50.285827    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 15:41:51.001591 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:51.001553    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-90ca97-openshift-default-659f84d46b-9l4kj" event={"ID":"f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af","Type":"ContainerStarted","Data":"29bfb5137b99e6953e3e53c39f44311f7e0e2ab67ead091e0c3c09c5f8a351f7"}
Sep 10 15:41:51.022956 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:51.022901    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-90ca97-openshift-default-659f84d46b-9l4kj" podStartSLOduration=1.5952417639999998 podStartE2EDuration="5.022884234s" totalImagesPullingTime="3.42764247s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:41:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:41:51.021636407 +0000 UTC m=+886.763601511" watchObservedRunningTime="2026-09-10 15:41:51.022884234 +0000 UTC m=+886.764849338"
Sep 10 15:41:51.553405 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:51.553364    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-90ca97-openshift-default-659f84d46b-9l4kj"
Sep 10 15:41:51.558487 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:51.558457    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-90ca97-openshift-default-659f84d46b-9l4kj"
Sep 10 15:41:52.007214 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:52.007186    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-90ca97-openshift-default-659f84d46b-9l4kj"
Sep 10 15:41:52.008279 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:52.008259    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-90ca97-openshift-default-659f84d46b-9l4kj"
Sep 10 15:41:55.006664 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:55.006625    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-bd5806-9d646cccd-pxjzq"
Sep 10 15:41:56.918782 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:56.918672    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-90ca97-775464f5cc-spv5h"]
Sep 10 15:41:56.922637 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:56.922617    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-90ca97-775464f5cc-spv5h"
Sep 10 15:41:56.925070 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:56.925046    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-90ca97-serving-cert\""
Sep 10 15:41:56.934255 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:56.934228    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-90ca97-775464f5cc-spv5h"]
Sep 10 15:41:57.008685 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:57.008653    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a27d3806-fc72-4adf-bd20-fbe020d777f4-metrics-certs\") pod \"maas-api-e2e-inf-b-90ca97-775464f5cc-spv5h\" (UID: \"a27d3806-fc72-4adf-bd20-fbe020d777f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-90ca97-775464f5cc-spv5h"
Sep 10 15:41:57.008849 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:57.008730    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhhbp\" (UniqueName: \"kubernetes.io/projected/a27d3806-fc72-4adf-bd20-fbe020d777f4-kube-api-access-bhhbp\") pod \"maas-api-e2e-inf-b-90ca97-775464f5cc-spv5h\" (UID: \"a27d3806-fc72-4adf-bd20-fbe020d777f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-90ca97-775464f5cc-spv5h"
Sep 10 15:41:57.008849 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:57.008760    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a27d3806-fc72-4adf-bd20-fbe020d777f4-maas-api-tls\") pod \"maas-api-e2e-inf-b-90ca97-775464f5cc-spv5h\" (UID: \"a27d3806-fc72-4adf-bd20-fbe020d777f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-90ca97-775464f5cc-spv5h"
Sep 10 15:41:57.008849 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:57.008794    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a27d3806-fc72-4adf-bd20-fbe020d777f4-openshift-service-ca\") pod \"maas-api-e2e-inf-b-90ca97-775464f5cc-spv5h\" (UID: \"a27d3806-fc72-4adf-bd20-fbe020d777f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-90ca97-775464f5cc-spv5h"
Sep 10 15:41:57.109487 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:57.109454    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a27d3806-fc72-4adf-bd20-fbe020d777f4-openshift-service-ca\") pod \"maas-api-e2e-inf-b-90ca97-775464f5cc-spv5h\" (UID: \"a27d3806-fc72-4adf-bd20-fbe020d777f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-90ca97-775464f5cc-spv5h"
Sep 10 15:41:57.109734 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:57.109510    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a27d3806-fc72-4adf-bd20-fbe020d777f4-metrics-certs\") pod \"maas-api-e2e-inf-b-90ca97-775464f5cc-spv5h\" (UID: \"a27d3806-fc72-4adf-bd20-fbe020d777f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-90ca97-775464f5cc-spv5h"
Sep 10 15:41:57.109734 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:57.109659    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhhbp\" (UniqueName: \"kubernetes.io/projected/a27d3806-fc72-4adf-bd20-fbe020d777f4-kube-api-access-bhhbp\") pod \"maas-api-e2e-inf-b-90ca97-775464f5cc-spv5h\" (UID: \"a27d3806-fc72-4adf-bd20-fbe020d777f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-90ca97-775464f5cc-spv5h"
Sep 10 15:41:57.109734 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:57.109707    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a27d3806-fc72-4adf-bd20-fbe020d777f4-maas-api-tls\") pod \"maas-api-e2e-inf-b-90ca97-775464f5cc-spv5h\" (UID: \"a27d3806-fc72-4adf-bd20-fbe020d777f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-90ca97-775464f5cc-spv5h"
Sep 10 15:41:57.109901 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:41:57.109841    2559 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-90ca97-serving-cert: secret "maas-api-e2e-inf-b-90ca97-serving-cert" not found
Sep 10 15:41:57.109965 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:41:57.109904    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a27d3806-fc72-4adf-bd20-fbe020d777f4-maas-api-tls podName:a27d3806-fc72-4adf-bd20-fbe020d777f4 nodeName:}" failed. No retries permitted until 2026-09-10 15:41:57.609880754 +0000 UTC m=+893.351845861 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a27d3806-fc72-4adf-bd20-fbe020d777f4-maas-api-tls") pod "maas-api-e2e-inf-b-90ca97-775464f5cc-spv5h" (UID: "a27d3806-fc72-4adf-bd20-fbe020d777f4") : secret "maas-api-e2e-inf-b-90ca97-serving-cert" not found
Sep 10 15:41:57.110243 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:57.110218    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a27d3806-fc72-4adf-bd20-fbe020d777f4-openshift-service-ca\") pod \"maas-api-e2e-inf-b-90ca97-775464f5cc-spv5h\" (UID: \"a27d3806-fc72-4adf-bd20-fbe020d777f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-90ca97-775464f5cc-spv5h"
Sep 10 15:41:57.112278 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:57.112239    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a27d3806-fc72-4adf-bd20-fbe020d777f4-metrics-certs\") pod \"maas-api-e2e-inf-b-90ca97-775464f5cc-spv5h\" (UID: \"a27d3806-fc72-4adf-bd20-fbe020d777f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-90ca97-775464f5cc-spv5h"
Sep 10 15:41:57.121662 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:57.121636    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhhbp\" (UniqueName: \"kubernetes.io/projected/a27d3806-fc72-4adf-bd20-fbe020d777f4-kube-api-access-bhhbp\") pod \"maas-api-e2e-inf-b-90ca97-775464f5cc-spv5h\" (UID: \"a27d3806-fc72-4adf-bd20-fbe020d777f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-90ca97-775464f5cc-spv5h"
Sep 10 15:41:57.613685 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:57.613643    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a27d3806-fc72-4adf-bd20-fbe020d777f4-maas-api-tls\") pod \"maas-api-e2e-inf-b-90ca97-775464f5cc-spv5h\" (UID: \"a27d3806-fc72-4adf-bd20-fbe020d777f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-90ca97-775464f5cc-spv5h"
Sep 10 15:41:57.616011 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:57.615981    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a27d3806-fc72-4adf-bd20-fbe020d777f4-maas-api-tls\") pod \"maas-api-e2e-inf-b-90ca97-775464f5cc-spv5h\" (UID: \"a27d3806-fc72-4adf-bd20-fbe020d777f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-90ca97-775464f5cc-spv5h"
Sep 10 15:41:57.851198 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:57.851147    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-90ca97-775464f5cc-spv5h"
Sep 10 15:41:58.128997 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:58.128968    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-90ca97-775464f5cc-spv5h"]
Sep 10 15:41:58.130999 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:41:58.130968    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda27d3806_fc72_4adf_bd20_fbe020d777f4.slice/crio-912829acbb388d526a507a9d5643b9c86e3a72a9c5aee1dff92bf56c7f85c3fc WatchSource:0}: Error finding container 912829acbb388d526a507a9d5643b9c86e3a72a9c5aee1dff92bf56c7f85c3fc: Status 404 returned error can't find the container with id 912829acbb388d526a507a9d5643b9c86e3a72a9c5aee1dff92bf56c7f85c3fc
Sep 10 15:41:59.047461 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:59.047423    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-90ca97-775464f5cc-spv5h" event={"ID":"a27d3806-fc72-4adf-bd20-fbe020d777f4","Type":"ContainerStarted","Data":"15fc9b762ab1c8221e3e8b619e08d6c85c7fc9e988f3d4c0817f84bf1487921c"}
Sep 10 15:41:59.047691 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:59.047487    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-90ca97-775464f5cc-spv5h" event={"ID":"a27d3806-fc72-4adf-bd20-fbe020d777f4","Type":"ContainerStarted","Data":"912829acbb388d526a507a9d5643b9c86e3a72a9c5aee1dff92bf56c7f85c3fc"}
Sep 10 15:41:59.047691 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:59.047523    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-90ca97-775464f5cc-spv5h"
Sep 10 15:41:59.073794 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:41:59.073736    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-90ca97-775464f5cc-spv5h" podStartSLOduration=2.685519614 podStartE2EDuration="3.073717266s" totalImagesPullingTime="388.197652ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:41:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:41:59.071130366 +0000 UTC m=+894.813095469" watchObservedRunningTime="2026-09-10 15:41:59.073717266 +0000 UTC m=+894.815682368"
Sep 10 15:42:02.112839 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:02.112805    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bs7hd"]
Sep 10 15:42:05.066005 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:05.065907    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-90ca97-775464f5cc-spv5h"
Sep 10 15:42:13.126298 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:13.126257    2559 generic.go:356] "Generic (PLEG): container finished" podID="f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af" containerID="29bfb5137b99e6953e3e53c39f44311f7e0e2ab67ead091e0c3c09c5f8a351f7" exitCode=0
Sep 10 15:42:13.126736 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:13.126329    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-90ca97-openshift-default-659f84d46b-9l4kj" event={"ID":"f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af","Type":"ContainerDied","Data":"29bfb5137b99e6953e3e53c39f44311f7e0e2ab67ead091e0c3c09c5f8a351f7"}
Sep 10 15:42:13.126780 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:13.126737    2559 scope.go:117] "RemoveContainer" containerID="29bfb5137b99e6953e3e53c39f44311f7e0e2ab67ead091e0c3c09c5f8a351f7"
Sep 10 15:42:13.127806 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:13.127770    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 15:42:13.127928 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:13.127854    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 15:42:13.127998 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:13.127906    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 15:42:14.133639 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:14.133603    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-90ca97-openshift-default-659f84d46b-9l4kj" event={"ID":"f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af","Type":"ContainerStarted","Data":"30103093e03447cf221d184225d9a3d3187f2f1131c0cb61cddf9ed65c3389a3"}
Sep 10 15:42:14.553831 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:14.553794    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-90ca97-openshift-default-659f84d46b-9l4kj"
Sep 10 15:42:14.554222 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:14.554129    2559 patch_prober.go:28] interesting pod/e2e-inf-b-90ca97-openshift-default-659f84d46b-9l4kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Sep 10 15:42:14.554406 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:14.554382    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-90ca97-openshift-default-659f84d46b-9l4kj" podUID="f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Sep 10 15:42:15.554468 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:15.554425    2559 patch_prober.go:28] interesting pod/e2e-inf-b-90ca97-openshift-default-659f84d46b-9l4kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Sep 10 15:42:15.554965 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:15.554492    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-90ca97-openshift-default-659f84d46b-9l4kj" podUID="f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Sep 10 15:42:16.058538 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:16.058494    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f89c4f3a-6869759cc7-fvfkq"]
Sep 10 15:42:16.063054 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:16.063022    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f89c4f3a-6869759cc7-fvfkq"
Sep 10 15:42:16.065706 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:16.065682    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-f89c4f3a-serving-cert\""
Sep 10 15:42:16.072128 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:16.072102    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f89c4f3a-6869759cc7-fvfkq"]
Sep 10 15:42:16.192195 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:16.192153    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df42a21b-5ac3-4c04-9b57-64521a46090b-maas-api-tls\") pod \"maas-api-e2e-migrate-f89c4f3a-6869759cc7-fvfkq\" (UID: \"df42a21b-5ac3-4c04-9b57-64521a46090b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f89c4f3a-6869759cc7-fvfkq"
Sep 10 15:42:16.192386 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:16.192201    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df42a21b-5ac3-4c04-9b57-64521a46090b-openshift-service-ca\") pod \"maas-api-e2e-migrate-f89c4f3a-6869759cc7-fvfkq\" (UID: \"df42a21b-5ac3-4c04-9b57-64521a46090b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f89c4f3a-6869759cc7-fvfkq"
Sep 10 15:42:16.192386 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:16.192235    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df42a21b-5ac3-4c04-9b57-64521a46090b-metrics-certs\") pod \"maas-api-e2e-migrate-f89c4f3a-6869759cc7-fvfkq\" (UID: \"df42a21b-5ac3-4c04-9b57-64521a46090b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f89c4f3a-6869759cc7-fvfkq"
Sep 10 15:42:16.192386 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:16.192290    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjmnn\" (UniqueName: \"kubernetes.io/projected/df42a21b-5ac3-4c04-9b57-64521a46090b-kube-api-access-vjmnn\") pod \"maas-api-e2e-migrate-f89c4f3a-6869759cc7-fvfkq\" (UID: \"df42a21b-5ac3-4c04-9b57-64521a46090b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f89c4f3a-6869759cc7-fvfkq"
Sep 10 15:42:16.293133 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:16.293093    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df42a21b-5ac3-4c04-9b57-64521a46090b-maas-api-tls\") pod \"maas-api-e2e-migrate-f89c4f3a-6869759cc7-fvfkq\" (UID: \"df42a21b-5ac3-4c04-9b57-64521a46090b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f89c4f3a-6869759cc7-fvfkq"
Sep 10 15:42:16.293133 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:16.293138    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df42a21b-5ac3-4c04-9b57-64521a46090b-openshift-service-ca\") pod \"maas-api-e2e-migrate-f89c4f3a-6869759cc7-fvfkq\" (UID: \"df42a21b-5ac3-4c04-9b57-64521a46090b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f89c4f3a-6869759cc7-fvfkq"
Sep 10 15:42:16.293382 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:16.293313    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df42a21b-5ac3-4c04-9b57-64521a46090b-metrics-certs\") pod \"maas-api-e2e-migrate-f89c4f3a-6869759cc7-fvfkq\" (UID: \"df42a21b-5ac3-4c04-9b57-64521a46090b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f89c4f3a-6869759cc7-fvfkq"
Sep 10 15:42:16.293452 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:16.293388    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjmnn\" (UniqueName: \"kubernetes.io/projected/df42a21b-5ac3-4c04-9b57-64521a46090b-kube-api-access-vjmnn\") pod \"maas-api-e2e-migrate-f89c4f3a-6869759cc7-fvfkq\" (UID: \"df42a21b-5ac3-4c04-9b57-64521a46090b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f89c4f3a-6869759cc7-fvfkq"
Sep 10 15:42:16.293799 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:16.293774    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df42a21b-5ac3-4c04-9b57-64521a46090b-openshift-service-ca\") pod \"maas-api-e2e-migrate-f89c4f3a-6869759cc7-fvfkq\" (UID: \"df42a21b-5ac3-4c04-9b57-64521a46090b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f89c4f3a-6869759cc7-fvfkq"
Sep 10 15:42:16.295690 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:16.295660    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df42a21b-5ac3-4c04-9b57-64521a46090b-maas-api-tls\") pod \"maas-api-e2e-migrate-f89c4f3a-6869759cc7-fvfkq\" (UID: \"df42a21b-5ac3-4c04-9b57-64521a46090b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f89c4f3a-6869759cc7-fvfkq"
Sep 10 15:42:16.295805 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:16.295783    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df42a21b-5ac3-4c04-9b57-64521a46090b-metrics-certs\") pod \"maas-api-e2e-migrate-f89c4f3a-6869759cc7-fvfkq\" (UID: \"df42a21b-5ac3-4c04-9b57-64521a46090b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f89c4f3a-6869759cc7-fvfkq"
Sep 10 15:42:16.302944 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:16.302919    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjmnn\" (UniqueName: \"kubernetes.io/projected/df42a21b-5ac3-4c04-9b57-64521a46090b-kube-api-access-vjmnn\") pod \"maas-api-e2e-migrate-f89c4f3a-6869759cc7-fvfkq\" (UID: \"df42a21b-5ac3-4c04-9b57-64521a46090b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f89c4f3a-6869759cc7-fvfkq"
Sep 10 15:42:16.388696 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:16.388605    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f89c4f3a-6869759cc7-fvfkq"
Sep 10 15:42:16.562131 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:16.562093    2559 patch_prober.go:28] interesting pod/e2e-inf-b-90ca97-openshift-default-659f84d46b-9l4kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Sep 10 15:42:16.562753 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:16.562712    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-90ca97-openshift-default-659f84d46b-9l4kj" podUID="f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Sep 10 15:42:16.672474 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:16.672431    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f89c4f3a-6869759cc7-fvfkq"]
Sep 10 15:42:16.676294 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:42:16.676270    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf42a21b_5ac3_4c04_9b57_64521a46090b.slice/crio-4a0a20eb5b4d077bc5042e4b080d5bd09bf97e9903abe287354327ffb3cb7bf2 WatchSource:0}: Error finding container 4a0a20eb5b4d077bc5042e4b080d5bd09bf97e9903abe287354327ffb3cb7bf2: Status 404 returned error can't find the container with id 4a0a20eb5b4d077bc5042e4b080d5bd09bf97e9903abe287354327ffb3cb7bf2
Sep 10 15:42:17.178865 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:17.178689    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f89c4f3a-6869759cc7-fvfkq" event={"ID":"df42a21b-5ac3-4c04-9b57-64521a46090b","Type":"ContainerStarted","Data":"c79eb44bc245f7cc0e9378a6494f934fe5ddb0db4f0e78785924d4d9d6bec636"}
Sep 10 15:42:17.178865 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:17.178754    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f89c4f3a-6869759cc7-fvfkq" event={"ID":"df42a21b-5ac3-4c04-9b57-64521a46090b","Type":"ContainerStarted","Data":"4a0a20eb5b4d077bc5042e4b080d5bd09bf97e9903abe287354327ffb3cb7bf2"}
Sep 10 15:42:17.178865 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:17.178786    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f89c4f3a-6869759cc7-fvfkq"
Sep 10 15:42:17.197394 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:17.197337    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f89c4f3a-6869759cc7-fvfkq" podStartSLOduration=0.9458961 podStartE2EDuration="1.197318384s" totalImagesPullingTime="251.422284ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:42:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:42:17.196818649 +0000 UTC m=+912.938783752" watchObservedRunningTime="2026-09-10 15:42:17.197318384 +0000 UTC m=+912.939283490"
Sep 10 15:42:17.554067 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:17.553970    2559 patch_prober.go:28] interesting pod/e2e-inf-b-90ca97-openshift-default-659f84d46b-9l4kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Sep 10 15:42:17.554067 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:17.554041    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-90ca97-openshift-default-659f84d46b-9l4kj" podUID="f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Sep 10 15:42:18.554410 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:18.554348    2559 patch_prober.go:28] interesting pod/e2e-inf-b-90ca97-openshift-default-659f84d46b-9l4kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Sep 10 15:42:18.554950 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:18.554450    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-90ca97-openshift-default-659f84d46b-9l4kj" podUID="f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Sep 10 15:42:19.557985 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:19.557956    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-90ca97-openshift-default-659f84d46b-9l4kj"
Sep 10 15:42:19.558400 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:19.558335    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-90ca97-openshift-default-659f84d46b-9l4kj"
Sep 10 15:42:19.558986 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:19.558966    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-90ca97-openshift-default-659f84d46b-9l4kj"
Sep 10 15:42:22.710308 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:22.710270    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bs7hd"]
Sep 10 15:42:23.189048 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:23.189019    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f89c4f3a-6869759cc7-fvfkq"
Sep 10 15:42:33.896613 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:33.896564    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-a-fa9f2a54-kserve-54cb4c6799-xdv4x"]
Sep 10 15:42:33.899715 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:33.899698    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-fa9f2a54-kserve-54cb4c6799-xdv4x"
Sep 10 15:42:33.902195 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:33.902168    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-a-fa9f2a54-kserve-self-signed-certs\""
Sep 10 15:42:33.910959 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:33.910930    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-fa9f2a54-kserve-54cb4c6799-xdv4x"]
Sep 10 15:42:34.056736 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:34.056698    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a55c9635-edc4-4efe-b78c-e0c291910e61-model-cache\") pod \"e2e-conflict-a-fa9f2a54-kserve-54cb4c6799-xdv4x\" (UID: \"a55c9635-edc4-4efe-b78c-e0c291910e61\") " pod="llm/e2e-conflict-a-fa9f2a54-kserve-54cb4c6799-xdv4x"
Sep 10 15:42:34.056900 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:34.056758    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a55c9635-edc4-4efe-b78c-e0c291910e61-dshm\") pod \"e2e-conflict-a-fa9f2a54-kserve-54cb4c6799-xdv4x\" (UID: \"a55c9635-edc4-4efe-b78c-e0c291910e61\") " pod="llm/e2e-conflict-a-fa9f2a54-kserve-54cb4c6799-xdv4x"
Sep 10 15:42:34.056900 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:34.056795    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a55c9635-edc4-4efe-b78c-e0c291910e61-tls-certs\") pod \"e2e-conflict-a-fa9f2a54-kserve-54cb4c6799-xdv4x\" (UID: \"a55c9635-edc4-4efe-b78c-e0c291910e61\") " pod="llm/e2e-conflict-a-fa9f2a54-kserve-54cb4c6799-xdv4x"
Sep 10 15:42:34.056900 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:34.056824    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kcnf\" (UniqueName: \"kubernetes.io/projected/a55c9635-edc4-4efe-b78c-e0c291910e61-kube-api-access-9kcnf\") pod \"e2e-conflict-a-fa9f2a54-kserve-54cb4c6799-xdv4x\" (UID: \"a55c9635-edc4-4efe-b78c-e0c291910e61\") " pod="llm/e2e-conflict-a-fa9f2a54-kserve-54cb4c6799-xdv4x"
Sep 10 15:42:34.056900 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:34.056853    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a55c9635-edc4-4efe-b78c-e0c291910e61-tmp-dir\") pod \"e2e-conflict-a-fa9f2a54-kserve-54cb4c6799-xdv4x\" (UID: \"a55c9635-edc4-4efe-b78c-e0c291910e61\") " pod="llm/e2e-conflict-a-fa9f2a54-kserve-54cb4c6799-xdv4x"
Sep 10 15:42:34.056900 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:34.056880    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a55c9635-edc4-4efe-b78c-e0c291910e61-home\") pod \"e2e-conflict-a-fa9f2a54-kserve-54cb4c6799-xdv4x\" (UID: \"a55c9635-edc4-4efe-b78c-e0c291910e61\") " pod="llm/e2e-conflict-a-fa9f2a54-kserve-54cb4c6799-xdv4x"
Sep 10 15:42:34.157682 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:34.157603    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a55c9635-edc4-4efe-b78c-e0c291910e61-model-cache\") pod \"e2e-conflict-a-fa9f2a54-kserve-54cb4c6799-xdv4x\" (UID: \"a55c9635-edc4-4efe-b78c-e0c291910e61\") " pod="llm/e2e-conflict-a-fa9f2a54-kserve-54cb4c6799-xdv4x"
Sep 10 15:42:34.157682 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:34.157652    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a55c9635-edc4-4efe-b78c-e0c291910e61-dshm\") pod \"e2e-conflict-a-fa9f2a54-kserve-54cb4c6799-xdv4x\" (UID: \"a55c9635-edc4-4efe-b78c-e0c291910e61\") " pod="llm/e2e-conflict-a-fa9f2a54-kserve-54cb4c6799-xdv4x"
Sep 10 15:42:34.157899 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:34.157756    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a55c9635-edc4-4efe-b78c-e0c291910e61-tls-certs\") pod \"e2e-conflict-a-fa9f2a54-kserve-54cb4c6799-xdv4x\" (UID: \"a55c9635-edc4-4efe-b78c-e0c291910e61\") " pod="llm/e2e-conflict-a-fa9f2a54-kserve-54cb4c6799-xdv4x"
Sep 10 15:42:34.157899 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:34.157776    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kcnf\" (UniqueName: \"kubernetes.io/projected/a55c9635-edc4-4efe-b78c-e0c291910e61-kube-api-access-9kcnf\") pod \"e2e-conflict-a-fa9f2a54-kserve-54cb4c6799-xdv4x\" (UID: \"a55c9635-edc4-4efe-b78c-e0c291910e61\") " pod="llm/e2e-conflict-a-fa9f2a54-kserve-54cb4c6799-xdv4x"
Sep 10 15:42:34.157899 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:34.157805    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a55c9635-edc4-4efe-b78c-e0c291910e61-tmp-dir\") pod \"e2e-conflict-a-fa9f2a54-kserve-54cb4c6799-xdv4x\" (UID: \"a55c9635-edc4-4efe-b78c-e0c291910e61\") " pod="llm/e2e-conflict-a-fa9f2a54-kserve-54cb4c6799-xdv4x"
Sep 10 15:42:34.157899 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:34.157832    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a55c9635-edc4-4efe-b78c-e0c291910e61-home\") pod \"e2e-conflict-a-fa9f2a54-kserve-54cb4c6799-xdv4x\" (UID: \"a55c9635-edc4-4efe-b78c-e0c291910e61\") " pod="llm/e2e-conflict-a-fa9f2a54-kserve-54cb4c6799-xdv4x"
Sep 10 15:42:34.158260 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:34.158228    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a55c9635-edc4-4efe-b78c-e0c291910e61-tmp-dir\") pod \"e2e-conflict-a-fa9f2a54-kserve-54cb4c6799-xdv4x\" (UID: \"a55c9635-edc4-4efe-b78c-e0c291910e61\") " pod="llm/e2e-conflict-a-fa9f2a54-kserve-54cb4c6799-xdv4x"
Sep 10 15:42:34.158260 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:34.158244    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a55c9635-edc4-4efe-b78c-e0c291910e61-home\") pod \"e2e-conflict-a-fa9f2a54-kserve-54cb4c6799-xdv4x\" (UID: \"a55c9635-edc4-4efe-b78c-e0c291910e61\") " pod="llm/e2e-conflict-a-fa9f2a54-kserve-54cb4c6799-xdv4x"
Sep 10 15:42:34.158473 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:34.158287    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a55c9635-edc4-4efe-b78c-e0c291910e61-model-cache\") pod \"e2e-conflict-a-fa9f2a54-kserve-54cb4c6799-xdv4x\" (UID: \"a55c9635-edc4-4efe-b78c-e0c291910e61\") " pod="llm/e2e-conflict-a-fa9f2a54-kserve-54cb4c6799-xdv4x"
Sep 10 15:42:34.159937 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:34.159917    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a55c9635-edc4-4efe-b78c-e0c291910e61-dshm\") pod \"e2e-conflict-a-fa9f2a54-kserve-54cb4c6799-xdv4x\" (UID: \"a55c9635-edc4-4efe-b78c-e0c291910e61\") " pod="llm/e2e-conflict-a-fa9f2a54-kserve-54cb4c6799-xdv4x"
Sep 10 15:42:34.160646 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:34.160630    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a55c9635-edc4-4efe-b78c-e0c291910e61-tls-certs\") pod \"e2e-conflict-a-fa9f2a54-kserve-54cb4c6799-xdv4x\" (UID: \"a55c9635-edc4-4efe-b78c-e0c291910e61\") " pod="llm/e2e-conflict-a-fa9f2a54-kserve-54cb4c6799-xdv4x"
Sep 10 15:42:34.167901 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:34.167874    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kcnf\" (UniqueName: \"kubernetes.io/projected/a55c9635-edc4-4efe-b78c-e0c291910e61-kube-api-access-9kcnf\") pod \"e2e-conflict-a-fa9f2a54-kserve-54cb4c6799-xdv4x\" (UID: \"a55c9635-edc4-4efe-b78c-e0c291910e61\") " pod="llm/e2e-conflict-a-fa9f2a54-kserve-54cb4c6799-xdv4x"
Sep 10 15:42:34.209846 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:34.209811    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-fa9f2a54-kserve-54cb4c6799-xdv4x"
Sep 10 15:42:34.484779 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:34.484749    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-fa9f2a54-kserve-54cb4c6799-xdv4x"]
Sep 10 15:42:34.486681 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:42:34.486640    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda55c9635_edc4_4efe_b78c_e0c291910e61.slice/crio-c03d6e9a50707be089e0ef8661715f35e52c765e7b132c9b1e0fc397922a289e WatchSource:0}: Error finding container c03d6e9a50707be089e0ef8661715f35e52c765e7b132c9b1e0fc397922a289e: Status 404 returned error can't find the container with id c03d6e9a50707be089e0ef8661715f35e52c765e7b132c9b1e0fc397922a289e
Sep 10 15:42:35.284444 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:35.284402    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-fa9f2a54-kserve-54cb4c6799-xdv4x" event={"ID":"a55c9635-edc4-4efe-b78c-e0c291910e61","Type":"ContainerStarted","Data":"c6d17b7553443c672776935965f1db3fe9313f21356d33ae5cef0352c923a92a"}
Sep 10 15:42:35.284444 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:35.284444    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-fa9f2a54-kserve-54cb4c6799-xdv4x" event={"ID":"a55c9635-edc4-4efe-b78c-e0c291910e61","Type":"ContainerStarted","Data":"c03d6e9a50707be089e0ef8661715f35e52c765e7b132c9b1e0fc397922a289e"}
Sep 10 15:42:35.304346 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:35.304292    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-a-fa9f2a54-kserve-54cb4c6799-xdv4x" podStartSLOduration=2.304278434 podStartE2EDuration="2.304278434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:42:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:42:35.302175689 +0000 UTC m=+931.044140791" watchObservedRunningTime="2026-09-10 15:42:35.304278434 +0000 UTC m=+931.046243537"
Sep 10 15:42:36.199028 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:36.198992    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bs7hd"]
Sep 10 15:42:44.210370 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:44.210331    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-a-fa9f2a54-kserve-54cb4c6799-xdv4x"
Sep 10 15:42:44.210778 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:44.210640    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-a-fa9f2a54-kserve-54cb4c6799-xdv4x"
Sep 10 15:42:44.223902 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:44.223876    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-a-fa9f2a54-kserve-54cb4c6799-xdv4x"
Sep 10 15:42:44.349081 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:44.349049    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-a-fa9f2a54-kserve-54cb4c6799-xdv4x"
Sep 10 15:42:44.387523 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:44.387493    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f89c4f3a-6869759cc7-fvfkq"]
Sep 10 15:42:44.387756 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:44.387733    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f89c4f3a-6869759cc7-fvfkq" podUID="df42a21b-5ac3-4c04-9b57-64521a46090b" containerName="maas-api" containerID="cri-o://c79eb44bc245f7cc0e9378a6494f934fe5ddb0db4f0e78785924d4d9d6bec636" gracePeriod=30
Sep 10 15:42:44.649729 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:44.649704    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f89c4f3a-6869759cc7-fvfkq"
Sep 10 15:42:44.657858 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:44.657833    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vjmnn\" (UniqueName: \"kubernetes.io/projected/df42a21b-5ac3-4c04-9b57-64521a46090b-kube-api-access-vjmnn\") pod \"df42a21b-5ac3-4c04-9b57-64521a46090b\" (UID: \"df42a21b-5ac3-4c04-9b57-64521a46090b\") "
Sep 10 15:42:44.658012 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:44.657871    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df42a21b-5ac3-4c04-9b57-64521a46090b-openshift-service-ca\") pod \"df42a21b-5ac3-4c04-9b57-64521a46090b\" (UID: \"df42a21b-5ac3-4c04-9b57-64521a46090b\") "
Sep 10 15:42:44.658012 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:44.657894    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df42a21b-5ac3-4c04-9b57-64521a46090b-metrics-certs\") pod \"df42a21b-5ac3-4c04-9b57-64521a46090b\" (UID: \"df42a21b-5ac3-4c04-9b57-64521a46090b\") "
Sep 10 15:42:44.658012 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:44.657924    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df42a21b-5ac3-4c04-9b57-64521a46090b-maas-api-tls\") pod \"df42a21b-5ac3-4c04-9b57-64521a46090b\" (UID: \"df42a21b-5ac3-4c04-9b57-64521a46090b\") "
Sep 10 15:42:44.658265 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:44.658232    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df42a21b-5ac3-4c04-9b57-64521a46090b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "df42a21b-5ac3-4c04-9b57-64521a46090b" (UID: "df42a21b-5ac3-4c04-9b57-64521a46090b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:42:44.660326 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:44.660295    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df42a21b-5ac3-4c04-9b57-64521a46090b-kube-api-access-vjmnn" (OuterVolumeSpecName: "kube-api-access-vjmnn") pod "df42a21b-5ac3-4c04-9b57-64521a46090b" (UID: "df42a21b-5ac3-4c04-9b57-64521a46090b"). InnerVolumeSpecName "kube-api-access-vjmnn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:42:44.660482 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:44.660409    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df42a21b-5ac3-4c04-9b57-64521a46090b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "df42a21b-5ac3-4c04-9b57-64521a46090b" (UID: "df42a21b-5ac3-4c04-9b57-64521a46090b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:42:44.660482 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:44.660421    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df42a21b-5ac3-4c04-9b57-64521a46090b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "df42a21b-5ac3-4c04-9b57-64521a46090b" (UID: "df42a21b-5ac3-4c04-9b57-64521a46090b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:42:44.758657 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:44.758552    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjmnn\" (UniqueName: \"kubernetes.io/projected/df42a21b-5ac3-4c04-9b57-64521a46090b-kube-api-access-vjmnn\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:42:44.758657 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:44.758604    2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df42a21b-5ac3-4c04-9b57-64521a46090b-openshift-service-ca\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:42:44.758657 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:44.758622    2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df42a21b-5ac3-4c04-9b57-64521a46090b-metrics-certs\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:42:44.758657 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:44.758635    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df42a21b-5ac3-4c04-9b57-64521a46090b-maas-api-tls\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:42:45.344627 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:45.344563    2559 generic.go:356] "Generic (PLEG): container finished" podID="df42a21b-5ac3-4c04-9b57-64521a46090b" containerID="c79eb44bc245f7cc0e9378a6494f934fe5ddb0db4f0e78785924d4d9d6bec636" exitCode=0
Sep 10 15:42:45.344627 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:45.344614    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f89c4f3a-6869759cc7-fvfkq" event={"ID":"df42a21b-5ac3-4c04-9b57-64521a46090b","Type":"ContainerDied","Data":"c79eb44bc245f7cc0e9378a6494f934fe5ddb0db4f0e78785924d4d9d6bec636"}
Sep 10 15:42:45.345185 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:45.344634    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f89c4f3a-6869759cc7-fvfkq"
Sep 10 15:42:45.345185 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:45.344677    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f89c4f3a-6869759cc7-fvfkq" event={"ID":"df42a21b-5ac3-4c04-9b57-64521a46090b","Type":"ContainerDied","Data":"4a0a20eb5b4d077bc5042e4b080d5bd09bf97e9903abe287354327ffb3cb7bf2"}
Sep 10 15:42:45.345185 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:45.344707    2559 scope.go:117] "RemoveContainer" containerID="c79eb44bc245f7cc0e9378a6494f934fe5ddb0db4f0e78785924d4d9d6bec636"
Sep 10 15:42:45.354237 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:45.354218    2559 scope.go:117] "RemoveContainer" containerID="c79eb44bc245f7cc0e9378a6494f934fe5ddb0db4f0e78785924d4d9d6bec636"
Sep 10 15:42:45.354556 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:42:45.354537    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c79eb44bc245f7cc0e9378a6494f934fe5ddb0db4f0e78785924d4d9d6bec636\": container with ID starting with c79eb44bc245f7cc0e9378a6494f934fe5ddb0db4f0e78785924d4d9d6bec636 not found: ID does not exist" containerID="c79eb44bc245f7cc0e9378a6494f934fe5ddb0db4f0e78785924d4d9d6bec636"
Sep 10 15:42:45.354647 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:45.354564    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c79eb44bc245f7cc0e9378a6494f934fe5ddb0db4f0e78785924d4d9d6bec636"} err="failed to get container status \"c79eb44bc245f7cc0e9378a6494f934fe5ddb0db4f0e78785924d4d9d6bec636\": rpc error: code = NotFound desc = could not find container \"c79eb44bc245f7cc0e9378a6494f934fe5ddb0db4f0e78785924d4d9d6bec636\": container with ID starting with c79eb44bc245f7cc0e9378a6494f934fe5ddb0db4f0e78785924d4d9d6bec636 not found: ID does not exist"
Sep 10 15:42:45.374176 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:45.374150    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f89c4f3a-6869759cc7-fvfkq"]
Sep 10 15:42:45.384400 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:45.384379    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f89c4f3a-6869759cc7-fvfkq"]
Sep 10 15:42:45.601645 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:45.601546    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-bd8496a8-openshift-default-7656dcd86d-txrh4"]
Sep 10 15:42:45.602033 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:45.602014    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df42a21b-5ac3-4c04-9b57-64521a46090b" containerName="maas-api"
Sep 10 15:42:45.602124 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:45.602036    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="df42a21b-5ac3-4c04-9b57-64521a46090b" containerName="maas-api"
Sep 10 15:42:45.602176 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:45.602141    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="df42a21b-5ac3-4c04-9b57-64521a46090b" containerName="maas-api"
Sep 10 15:42:45.607095 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:45.607072    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-bd8496a8-openshift-default-7656dcd86d-txrh4"
Sep 10 15:42:45.609406 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:45.609387    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-bd8496a8-openshift-default-dockercfg-nswvc\""
Sep 10 15:42:45.625383 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:45.625351    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-bd8496a8-openshift-default-7656dcd86d-txrh4"]
Sep 10 15:42:45.665629 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:45.665590    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77094e7d-3bf4-4f13-a6b9-9f77ce209b9e-workload-socket\") pod \"e2e-ait-bd8496a8-openshift-default-7656dcd86d-txrh4\" (UID: \"77094e7d-3bf4-4f13-a6b9-9f77ce209b9e\") " pod="openshift-ingress/e2e-ait-bd8496a8-openshift-default-7656dcd86d-txrh4"
Sep 10 15:42:45.665783 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:45.665639    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77094e7d-3bf4-4f13-a6b9-9f77ce209b9e-istiod-ca-cert\") pod \"e2e-ait-bd8496a8-openshift-default-7656dcd86d-txrh4\" (UID: \"77094e7d-3bf4-4f13-a6b9-9f77ce209b9e\") " pod="openshift-ingress/e2e-ait-bd8496a8-openshift-default-7656dcd86d-txrh4"
Sep 10 15:42:45.665783 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:45.665671    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77094e7d-3bf4-4f13-a6b9-9f77ce209b9e-istio-data\") pod \"e2e-ait-bd8496a8-openshift-default-7656dcd86d-txrh4\" (UID: \"77094e7d-3bf4-4f13-a6b9-9f77ce209b9e\") " pod="openshift-ingress/e2e-ait-bd8496a8-openshift-default-7656dcd86d-txrh4"
Sep 10 15:42:45.665783 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:45.665706    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77094e7d-3bf4-4f13-a6b9-9f77ce209b9e-workload-certs\") pod \"e2e-ait-bd8496a8-openshift-default-7656dcd86d-txrh4\" (UID: \"77094e7d-3bf4-4f13-a6b9-9f77ce209b9e\") " pod="openshift-ingress/e2e-ait-bd8496a8-openshift-default-7656dcd86d-txrh4"
Sep 10 15:42:45.665783 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:45.665773    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77094e7d-3bf4-4f13-a6b9-9f77ce209b9e-credential-socket\") pod \"e2e-ait-bd8496a8-openshift-default-7656dcd86d-txrh4\" (UID: \"77094e7d-3bf4-4f13-a6b9-9f77ce209b9e\") " pod="openshift-ingress/e2e-ait-bd8496a8-openshift-default-7656dcd86d-txrh4"
Sep 10 15:42:45.665919 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:45.665807    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77094e7d-3bf4-4f13-a6b9-9f77ce209b9e-istio-token\") pod \"e2e-ait-bd8496a8-openshift-default-7656dcd86d-txrh4\" (UID: \"77094e7d-3bf4-4f13-a6b9-9f77ce209b9e\") " pod="openshift-ingress/e2e-ait-bd8496a8-openshift-default-7656dcd86d-txrh4"
Sep 10 15:42:45.665919 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:45.665833    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tgj5\" (UniqueName: \"kubernetes.io/projected/77094e7d-3bf4-4f13-a6b9-9f77ce209b9e-kube-api-access-7tgj5\") pod \"e2e-ait-bd8496a8-openshift-default-7656dcd86d-txrh4\" (UID: \"77094e7d-3bf4-4f13-a6b9-9f77ce209b9e\") " pod="openshift-ingress/e2e-ait-bd8496a8-openshift-default-7656dcd86d-txrh4"
Sep 10 15:42:45.665919 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:45.665868    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77094e7d-3bf4-4f13-a6b9-9f77ce209b9e-istio-envoy\") pod \"e2e-ait-bd8496a8-openshift-default-7656dcd86d-txrh4\" (UID: \"77094e7d-3bf4-4f13-a6b9-9f77ce209b9e\") " pod="openshift-ingress/e2e-ait-bd8496a8-openshift-default-7656dcd86d-txrh4"
Sep 10 15:42:45.665919 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:45.665889    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77094e7d-3bf4-4f13-a6b9-9f77ce209b9e-istio-podinfo\") pod \"e2e-ait-bd8496a8-openshift-default-7656dcd86d-txrh4\" (UID: \"77094e7d-3bf4-4f13-a6b9-9f77ce209b9e\") " pod="openshift-ingress/e2e-ait-bd8496a8-openshift-default-7656dcd86d-txrh4"
Sep 10 15:42:45.767092 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:45.767053    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77094e7d-3bf4-4f13-a6b9-9f77ce209b9e-istio-podinfo\") pod \"e2e-ait-bd8496a8-openshift-default-7656dcd86d-txrh4\" (UID: \"77094e7d-3bf4-4f13-a6b9-9f77ce209b9e\") " pod="openshift-ingress/e2e-ait-bd8496a8-openshift-default-7656dcd86d-txrh4"
Sep 10 15:42:45.767255 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:45.767111    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77094e7d-3bf4-4f13-a6b9-9f77ce209b9e-workload-socket\") pod \"e2e-ait-bd8496a8-openshift-default-7656dcd86d-txrh4\" (UID: \"77094e7d-3bf4-4f13-a6b9-9f77ce209b9e\") " pod="openshift-ingress/e2e-ait-bd8496a8-openshift-default-7656dcd86d-txrh4"
Sep 10 15:42:45.767255 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:45.767223    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77094e7d-3bf4-4f13-a6b9-9f77ce209b9e-istiod-ca-cert\") pod \"e2e-ait-bd8496a8-openshift-default-7656dcd86d-txrh4\" (UID: \"77094e7d-3bf4-4f13-a6b9-9f77ce209b9e\") " pod="openshift-ingress/e2e-ait-bd8496a8-openshift-default-7656dcd86d-txrh4"
Sep 10 15:42:45.767339 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:45.767269    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77094e7d-3bf4-4f13-a6b9-9f77ce209b9e-istio-data\") pod \"e2e-ait-bd8496a8-openshift-default-7656dcd86d-txrh4\" (UID: \"77094e7d-3bf4-4f13-a6b9-9f77ce209b9e\") " pod="openshift-ingress/e2e-ait-bd8496a8-openshift-default-7656dcd86d-txrh4"
Sep 10 15:42:45.767339 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:45.767302    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77094e7d-3bf4-4f13-a6b9-9f77ce209b9e-workload-certs\") pod \"e2e-ait-bd8496a8-openshift-default-7656dcd86d-txrh4\" (UID: \"77094e7d-3bf4-4f13-a6b9-9f77ce209b9e\") " pod="openshift-ingress/e2e-ait-bd8496a8-openshift-default-7656dcd86d-txrh4"
Sep 10 15:42:45.767339 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:45.767333    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77094e7d-3bf4-4f13-a6b9-9f77ce209b9e-credential-socket\") pod \"e2e-ait-bd8496a8-openshift-default-7656dcd86d-txrh4\" (UID: \"77094e7d-3bf4-4f13-a6b9-9f77ce209b9e\") " pod="openshift-ingress/e2e-ait-bd8496a8-openshift-default-7656dcd86d-txrh4"
Sep 10 15:42:45.767484 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:45.767357    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77094e7d-3bf4-4f13-a6b9-9f77ce209b9e-istio-token\") pod \"e2e-ait-bd8496a8-openshift-default-7656dcd86d-txrh4\" (UID: \"77094e7d-3bf4-4f13-a6b9-9f77ce209b9e\") " pod="openshift-ingress/e2e-ait-bd8496a8-openshift-default-7656dcd86d-txrh4"
Sep 10 15:42:45.767484 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:45.767387    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tgj5\" (UniqueName: \"kubernetes.io/projected/77094e7d-3bf4-4f13-a6b9-9f77ce209b9e-kube-api-access-7tgj5\") pod \"e2e-ait-bd8496a8-openshift-default-7656dcd86d-txrh4\" (UID: \"77094e7d-3bf4-4f13-a6b9-9f77ce209b9e\") " pod="openshift-ingress/e2e-ait-bd8496a8-openshift-default-7656dcd86d-txrh4"
Sep 10 15:42:45.767484 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:45.767437    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77094e7d-3bf4-4f13-a6b9-9f77ce209b9e-istio-envoy\") pod \"e2e-ait-bd8496a8-openshift-default-7656dcd86d-txrh4\" (UID: \"77094e7d-3bf4-4f13-a6b9-9f77ce209b9e\") " pod="openshift-ingress/e2e-ait-bd8496a8-openshift-default-7656dcd86d-txrh4"
Sep 10 15:42:45.767681 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:45.767497    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77094e7d-3bf4-4f13-a6b9-9f77ce209b9e-workload-socket\") pod \"e2e-ait-bd8496a8-openshift-default-7656dcd86d-txrh4\" (UID: \"77094e7d-3bf4-4f13-a6b9-9f77ce209b9e\") " pod="openshift-ingress/e2e-ait-bd8496a8-openshift-default-7656dcd86d-txrh4"
Sep 10 15:42:45.767738 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:45.767689    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77094e7d-3bf4-4f13-a6b9-9f77ce209b9e-istio-data\") pod \"e2e-ait-bd8496a8-openshift-default-7656dcd86d-txrh4\" (UID: \"77094e7d-3bf4-4f13-a6b9-9f77ce209b9e\") " pod="openshift-ingress/e2e-ait-bd8496a8-openshift-default-7656dcd86d-txrh4"
Sep 10 15:42:45.767813 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:45.767787    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77094e7d-3bf4-4f13-a6b9-9f77ce209b9e-workload-certs\") pod \"e2e-ait-bd8496a8-openshift-default-7656dcd86d-txrh4\" (UID: \"77094e7d-3bf4-4f13-a6b9-9f77ce209b9e\") " pod="openshift-ingress/e2e-ait-bd8496a8-openshift-default-7656dcd86d-txrh4"
Sep 10 15:42:45.767918 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:45.767901    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77094e7d-3bf4-4f13-a6b9-9f77ce209b9e-credential-socket\") pod \"e2e-ait-bd8496a8-openshift-default-7656dcd86d-txrh4\" (UID: \"77094e7d-3bf4-4f13-a6b9-9f77ce209b9e\") " pod="openshift-ingress/e2e-ait-bd8496a8-openshift-default-7656dcd86d-txrh4"
Sep 10 15:42:45.768165 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:45.768148    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77094e7d-3bf4-4f13-a6b9-9f77ce209b9e-istiod-ca-cert\") pod \"e2e-ait-bd8496a8-openshift-default-7656dcd86d-txrh4\" (UID: \"77094e7d-3bf4-4f13-a6b9-9f77ce209b9e\") " pod="openshift-ingress/e2e-ait-bd8496a8-openshift-default-7656dcd86d-txrh4"
Sep 10 15:42:45.769592 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:45.769557    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77094e7d-3bf4-4f13-a6b9-9f77ce209b9e-istio-podinfo\") pod \"e2e-ait-bd8496a8-openshift-default-7656dcd86d-txrh4\" (UID: \"77094e7d-3bf4-4f13-a6b9-9f77ce209b9e\") " pod="openshift-ingress/e2e-ait-bd8496a8-openshift-default-7656dcd86d-txrh4"
Sep 10 15:42:45.769700 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:45.769673    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77094e7d-3bf4-4f13-a6b9-9f77ce209b9e-istio-envoy\") pod \"e2e-ait-bd8496a8-openshift-default-7656dcd86d-txrh4\" (UID: \"77094e7d-3bf4-4f13-a6b9-9f77ce209b9e\") " pod="openshift-ingress/e2e-ait-bd8496a8-openshift-default-7656dcd86d-txrh4"
Sep 10 15:42:45.780251 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:45.780227    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77094e7d-3bf4-4f13-a6b9-9f77ce209b9e-istio-token\") pod \"e2e-ait-bd8496a8-openshift-default-7656dcd86d-txrh4\" (UID: \"77094e7d-3bf4-4f13-a6b9-9f77ce209b9e\") " pod="openshift-ingress/e2e-ait-bd8496a8-openshift-default-7656dcd86d-txrh4"
Sep 10 15:42:45.781757 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:45.781740    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tgj5\" (UniqueName: \"kubernetes.io/projected/77094e7d-3bf4-4f13-a6b9-9f77ce209b9e-kube-api-access-7tgj5\") pod \"e2e-ait-bd8496a8-openshift-default-7656dcd86d-txrh4\" (UID: \"77094e7d-3bf4-4f13-a6b9-9f77ce209b9e\") " pod="openshift-ingress/e2e-ait-bd8496a8-openshift-default-7656dcd86d-txrh4"
Sep 10 15:42:45.917617 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:45.917562    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-bd8496a8-openshift-default-7656dcd86d-txrh4"
Sep 10 15:42:46.203052 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:46.203020    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-bd8496a8-openshift-default-7656dcd86d-txrh4"]
Sep 10 15:42:46.205010 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:42:46.204985    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77094e7d_3bf4_4f13_a6b9_9f77ce209b9e.slice/crio-5da13499866bf42218298badf00a16720c27547700b7ab078f5dcf713b791030 WatchSource:0}: Error finding container 5da13499866bf42218298badf00a16720c27547700b7ab078f5dcf713b791030: Status 404 returned error can't find the container with id 5da13499866bf42218298badf00a16720c27547700b7ab078f5dcf713b791030
Sep 10 15:42:46.207262 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:46.207230    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 15:42:46.207346 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:46.207296    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 15:42:46.207346 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:46.207325    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 15:42:46.355016 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:46.354978    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-bd8496a8-openshift-default-7656dcd86d-txrh4" event={"ID":"77094e7d-3bf4-4f13-a6b9-9f77ce209b9e","Type":"ContainerStarted","Data":"c6a8bc511f629d772ddae3cf30af363a150607c5bb73809bd63a1cbc01ffcaa6"}
Sep 10 15:42:46.355486 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:46.355038    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-bd8496a8-openshift-default-7656dcd86d-txrh4" event={"ID":"77094e7d-3bf4-4f13-a6b9-9f77ce209b9e","Type":"ContainerStarted","Data":"5da13499866bf42218298badf00a16720c27547700b7ab078f5dcf713b791030"}
Sep 10 15:42:46.381294 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:46.381241    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-bd8496a8-openshift-default-7656dcd86d-txrh4" podStartSLOduration=1.3812246830000001 podStartE2EDuration="1.381224683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:42:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:42:46.378703804 +0000 UTC m=+942.120668921" watchObservedRunningTime="2026-09-10 15:42:46.381224683 +0000 UTC m=+942.123189785"
Sep 10 15:42:46.784752 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:46.784717    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df42a21b-5ac3-4c04-9b57-64521a46090b" path="/var/lib/kubelet/pods/df42a21b-5ac3-4c04-9b57-64521a46090b/volumes"
Sep 10 15:42:46.918523 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:46.918489    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-bd8496a8-openshift-default-7656dcd86d-txrh4"
Sep 10 15:42:47.923279 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:47.923246    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-bd8496a8-openshift-default-7656dcd86d-txrh4"
Sep 10 15:42:48.365512 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:48.365432    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-bd8496a8-openshift-default-7656dcd86d-txrh4"
Sep 10 15:42:48.366321 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:48.366301    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-bd8496a8-openshift-default-7656dcd86d-txrh4"
Sep 10 15:42:50.119617 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:42:50.119571    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bs7hd"]
Sep 10 15:43:07.813272 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:07.813232    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bs7hd"]
Sep 10 15:43:08.484464 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:08.484432    2559 generic.go:356] "Generic (PLEG): container finished" podID="77094e7d-3bf4-4f13-a6b9-9f77ce209b9e" containerID="c6a8bc511f629d772ddae3cf30af363a150607c5bb73809bd63a1cbc01ffcaa6" exitCode=0
Sep 10 15:43:08.484682 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:08.484500    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-bd8496a8-openshift-default-7656dcd86d-txrh4" event={"ID":"77094e7d-3bf4-4f13-a6b9-9f77ce209b9e","Type":"ContainerDied","Data":"c6a8bc511f629d772ddae3cf30af363a150607c5bb73809bd63a1cbc01ffcaa6"}
Sep 10 15:43:08.484922 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:08.484906    2559 scope.go:117] "RemoveContainer" containerID="c6a8bc511f629d772ddae3cf30af363a150607c5bb73809bd63a1cbc01ffcaa6"
Sep 10 15:43:08.485933 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:08.485905    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 15:43:08.485992 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:08.485971    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 15:43:08.486027 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:08.485999    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 15:43:09.491414 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:09.491381    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-bd8496a8-openshift-default-7656dcd86d-txrh4" event={"ID":"77094e7d-3bf4-4f13-a6b9-9f77ce209b9e","Type":"ContainerStarted","Data":"39703b1d6fb6db856fdae73252fe7538ce19b07a58819aeac7dbbab7161abe09"}
Sep 10 15:43:09.918256 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:09.918233    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-bd8496a8-openshift-default-7656dcd86d-txrh4"
Sep 10 15:43:09.918514 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:09.918489    2559 patch_prober.go:28] interesting pod/e2e-ait-bd8496a8-openshift-default-7656dcd86d-txrh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Sep 10 15:43:09.918568 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:09.918542    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-bd8496a8-openshift-default-7656dcd86d-txrh4" podUID="77094e7d-3bf4-4f13-a6b9-9f77ce209b9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Sep 10 15:43:10.918350 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:10.918316    2559 patch_prober.go:28] interesting pod/e2e-ait-bd8496a8-openshift-default-7656dcd86d-txrh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Sep 10 15:43:10.918765 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:10.918366    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-bd8496a8-openshift-default-7656dcd86d-txrh4" podUID="77094e7d-3bf4-4f13-a6b9-9f77ce209b9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Sep 10 15:43:11.918655 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:11.918615    2559 patch_prober.go:28] interesting pod/e2e-ait-bd8496a8-openshift-default-7656dcd86d-txrh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Sep 10 15:43:11.919065 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:11.918676    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-bd8496a8-openshift-default-7656dcd86d-txrh4" podUID="77094e7d-3bf4-4f13-a6b9-9f77ce209b9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Sep 10 15:43:12.918586 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:12.918551    2559 patch_prober.go:28] interesting pod/e2e-ait-bd8496a8-openshift-default-7656dcd86d-txrh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Sep 10 15:43:12.918765 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:12.918627    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-bd8496a8-openshift-default-7656dcd86d-txrh4" podUID="77094e7d-3bf4-4f13-a6b9-9f77ce209b9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Sep 10 15:43:13.918816 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:13.918767    2559 patch_prober.go:28] interesting pod/e2e-ait-bd8496a8-openshift-default-7656dcd86d-txrh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Sep 10 15:43:13.919214 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:13.918837    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-bd8496a8-openshift-default-7656dcd86d-txrh4" podUID="77094e7d-3bf4-4f13-a6b9-9f77ce209b9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Sep 10 15:43:15.024247 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:15.024212    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-bd8496a8-openshift-default-7656dcd86d-txrh4"
Sep 10 15:43:15.025336 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:15.024783    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-bd8496a8-openshift-default-7656dcd86d-txrh4"
Sep 10 15:43:15.026114 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:15.026093    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-bd8496a8-openshift-default-7656dcd86d-txrh4"
Sep 10 15:43:16.402039 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:16.402004    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-139155/test-model-139155-kserve-69bf99cdbf-4ng2n"]
Sep 10 15:43:16.405856 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:16.405829    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-139155/test-model-139155-kserve-69bf99cdbf-4ng2n"
Sep 10 15:43:16.411698 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:16.411436    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-139155\"/\"default-dockercfg-v8l6d\""
Sep 10 15:43:16.411698 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:16.411473    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-139155\"/\"openshift-service-ca.crt\""
Sep 10 15:43:16.411698 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:16.411473    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-139155\"/\"test-model-139155-kserve-self-signed-certs\""
Sep 10 15:43:16.411698 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:16.411670    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-139155\"/\"kube-root-ca.crt\""
Sep 10 15:43:16.422008 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:16.421986    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-139155/test-model-139155-kserve-69bf99cdbf-4ng2n"]
Sep 10 15:43:16.566473 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:16.566432    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94e7b3b9-5dab-483c-8dc2-a7226098ee76-home\") pod \"test-model-139155-kserve-69bf99cdbf-4ng2n\" (UID: \"94e7b3b9-5dab-483c-8dc2-a7226098ee76\") " pod="ai-tenant-e2e-inf-a-139155/test-model-139155-kserve-69bf99cdbf-4ng2n"
Sep 10 15:43:16.566708 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:16.566500    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nltcr\" (UniqueName: \"kubernetes.io/projected/94e7b3b9-5dab-483c-8dc2-a7226098ee76-kube-api-access-nltcr\") pod \"test-model-139155-kserve-69bf99cdbf-4ng2n\" (UID: \"94e7b3b9-5dab-483c-8dc2-a7226098ee76\") " pod="ai-tenant-e2e-inf-a-139155/test-model-139155-kserve-69bf99cdbf-4ng2n"
Sep 10 15:43:16.566708 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:16.566632    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94e7b3b9-5dab-483c-8dc2-a7226098ee76-dshm\") pod \"test-model-139155-kserve-69bf99cdbf-4ng2n\" (UID: \"94e7b3b9-5dab-483c-8dc2-a7226098ee76\") " pod="ai-tenant-e2e-inf-a-139155/test-model-139155-kserve-69bf99cdbf-4ng2n"
Sep 10 15:43:16.566708 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:16.566670    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94e7b3b9-5dab-483c-8dc2-a7226098ee76-tmp-dir\") pod \"test-model-139155-kserve-69bf99cdbf-4ng2n\" (UID: \"94e7b3b9-5dab-483c-8dc2-a7226098ee76\") " pod="ai-tenant-e2e-inf-a-139155/test-model-139155-kserve-69bf99cdbf-4ng2n"
Sep 10 15:43:16.566708 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:16.566703    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94e7b3b9-5dab-483c-8dc2-a7226098ee76-model-cache\") pod \"test-model-139155-kserve-69bf99cdbf-4ng2n\" (UID: \"94e7b3b9-5dab-483c-8dc2-a7226098ee76\") " pod="ai-tenant-e2e-inf-a-139155/test-model-139155-kserve-69bf99cdbf-4ng2n"
Sep 10 15:43:16.566890 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:16.566738    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94e7b3b9-5dab-483c-8dc2-a7226098ee76-tls-certs\") pod \"test-model-139155-kserve-69bf99cdbf-4ng2n\" (UID: \"94e7b3b9-5dab-483c-8dc2-a7226098ee76\") " pod="ai-tenant-e2e-inf-a-139155/test-model-139155-kserve-69bf99cdbf-4ng2n"
Sep 10 15:43:16.667979 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:16.667938    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94e7b3b9-5dab-483c-8dc2-a7226098ee76-dshm\") pod \"test-model-139155-kserve-69bf99cdbf-4ng2n\" (UID: \"94e7b3b9-5dab-483c-8dc2-a7226098ee76\") " pod="ai-tenant-e2e-inf-a-139155/test-model-139155-kserve-69bf99cdbf-4ng2n"
Sep 10 15:43:16.667979 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:16.667977    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94e7b3b9-5dab-483c-8dc2-a7226098ee76-tmp-dir\") pod \"test-model-139155-kserve-69bf99cdbf-4ng2n\" (UID: \"94e7b3b9-5dab-483c-8dc2-a7226098ee76\") " pod="ai-tenant-e2e-inf-a-139155/test-model-139155-kserve-69bf99cdbf-4ng2n"
Sep 10 15:43:16.668247 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:16.668000    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94e7b3b9-5dab-483c-8dc2-a7226098ee76-model-cache\") pod \"test-model-139155-kserve-69bf99cdbf-4ng2n\" (UID: \"94e7b3b9-5dab-483c-8dc2-a7226098ee76\") " pod="ai-tenant-e2e-inf-a-139155/test-model-139155-kserve-69bf99cdbf-4ng2n"
Sep 10 15:43:16.668247 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:16.668021    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94e7b3b9-5dab-483c-8dc2-a7226098ee76-tls-certs\") pod \"test-model-139155-kserve-69bf99cdbf-4ng2n\" (UID: \"94e7b3b9-5dab-483c-8dc2-a7226098ee76\") " pod="ai-tenant-e2e-inf-a-139155/test-model-139155-kserve-69bf99cdbf-4ng2n"
Sep 10 15:43:16.668247 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:16.668133    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94e7b3b9-5dab-483c-8dc2-a7226098ee76-home\") pod \"test-model-139155-kserve-69bf99cdbf-4ng2n\" (UID: \"94e7b3b9-5dab-483c-8dc2-a7226098ee76\") " pod="ai-tenant-e2e-inf-a-139155/test-model-139155-kserve-69bf99cdbf-4ng2n"
Sep 10 15:43:16.668247 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:16.668190    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nltcr\" (UniqueName: \"kubernetes.io/projected/94e7b3b9-5dab-483c-8dc2-a7226098ee76-kube-api-access-nltcr\") pod \"test-model-139155-kserve-69bf99cdbf-4ng2n\" (UID: \"94e7b3b9-5dab-483c-8dc2-a7226098ee76\") " pod="ai-tenant-e2e-inf-a-139155/test-model-139155-kserve-69bf99cdbf-4ng2n"
Sep 10 15:43:16.668505 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:16.668482    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94e7b3b9-5dab-483c-8dc2-a7226098ee76-model-cache\") pod \"test-model-139155-kserve-69bf99cdbf-4ng2n\" (UID: \"94e7b3b9-5dab-483c-8dc2-a7226098ee76\") " pod="ai-tenant-e2e-inf-a-139155/test-model-139155-kserve-69bf99cdbf-4ng2n"
Sep 10 15:43:16.668563 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:16.668503    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94e7b3b9-5dab-483c-8dc2-a7226098ee76-home\") pod \"test-model-139155-kserve-69bf99cdbf-4ng2n\" (UID: \"94e7b3b9-5dab-483c-8dc2-a7226098ee76\") " pod="ai-tenant-e2e-inf-a-139155/test-model-139155-kserve-69bf99cdbf-4ng2n"
Sep 10 15:43:16.668748 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:16.668727    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94e7b3b9-5dab-483c-8dc2-a7226098ee76-tmp-dir\") pod \"test-model-139155-kserve-69bf99cdbf-4ng2n\" (UID: \"94e7b3b9-5dab-483c-8dc2-a7226098ee76\") " pod="ai-tenant-e2e-inf-a-139155/test-model-139155-kserve-69bf99cdbf-4ng2n"
Sep 10 15:43:16.670324 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:16.670302    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94e7b3b9-5dab-483c-8dc2-a7226098ee76-dshm\") pod \"test-model-139155-kserve-69bf99cdbf-4ng2n\" (UID: \"94e7b3b9-5dab-483c-8dc2-a7226098ee76\") " pod="ai-tenant-e2e-inf-a-139155/test-model-139155-kserve-69bf99cdbf-4ng2n"
Sep 10 15:43:16.670599 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:16.670569    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94e7b3b9-5dab-483c-8dc2-a7226098ee76-tls-certs\") pod \"test-model-139155-kserve-69bf99cdbf-4ng2n\" (UID: \"94e7b3b9-5dab-483c-8dc2-a7226098ee76\") " pod="ai-tenant-e2e-inf-a-139155/test-model-139155-kserve-69bf99cdbf-4ng2n"
Sep 10 15:43:16.682984 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:16.682962    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nltcr\" (UniqueName: \"kubernetes.io/projected/94e7b3b9-5dab-483c-8dc2-a7226098ee76-kube-api-access-nltcr\") pod \"test-model-139155-kserve-69bf99cdbf-4ng2n\" (UID: \"94e7b3b9-5dab-483c-8dc2-a7226098ee76\") " pod="ai-tenant-e2e-inf-a-139155/test-model-139155-kserve-69bf99cdbf-4ng2n"
Sep 10 15:43:16.715879 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:16.715843    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-139155/test-model-139155-kserve-69bf99cdbf-4ng2n"
Sep 10 15:43:16.998516 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:16.998366    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-139155/test-model-139155-kserve-69bf99cdbf-4ng2n"]
Sep 10 15:43:17.001294 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:43:17.001269    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod94e7b3b9_5dab_483c_8dc2_a7226098ee76.slice/crio-f55f0c5df2a346777063bca7fd7cc7eb7599c49eb17d90ab1d76ce4cf5f22155 WatchSource:0}: Error finding container f55f0c5df2a346777063bca7fd7cc7eb7599c49eb17d90ab1d76ce4cf5f22155: Status 404 returned error can't find the container with id f55f0c5df2a346777063bca7fd7cc7eb7599c49eb17d90ab1d76ce4cf5f22155
Sep 10 15:43:17.537879 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:17.537838    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-139155/test-model-139155-kserve-69bf99cdbf-4ng2n" event={"ID":"94e7b3b9-5dab-483c-8dc2-a7226098ee76","Type":"ContainerStarted","Data":"dd1611e03b15a570f3dcfd8e2bde7f6493ad7d6c6a12eb7009349a84c2f28466"}
Sep 10 15:43:17.537879 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:17.537882    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-139155/test-model-139155-kserve-69bf99cdbf-4ng2n" event={"ID":"94e7b3b9-5dab-483c-8dc2-a7226098ee76","Type":"ContainerStarted","Data":"f55f0c5df2a346777063bca7fd7cc7eb7599c49eb17d90ab1d76ce4cf5f22155"}
Sep 10 15:43:17.574280 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:17.574227    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-139155/test-model-139155-kserve-69bf99cdbf-4ng2n" podStartSLOduration=1.574210988 podStartE2EDuration="1.574210988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:43:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:43:17.573629972 +0000 UTC m=+973.315595075" watchObservedRunningTime="2026-09-10 15:43:17.574210988 +0000 UTC m=+973.316176091"
Sep 10 15:43:26.716471 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:26.716384    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-139155/test-model-139155-kserve-69bf99cdbf-4ng2n"
Sep 10 15:43:26.716471 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:26.716450    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-139155/test-model-139155-kserve-69bf99cdbf-4ng2n"
Sep 10 15:43:26.729125 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:26.729096    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-139155/test-model-139155-kserve-69bf99cdbf-4ng2n"
Sep 10 15:43:26.894984 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:26.894951    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bs7hd"]
Sep 10 15:43:27.605964 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:27.605934    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-139155/test-model-139155-kserve-69bf99cdbf-4ng2n"
Sep 10 15:43:31.933388 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:31.933358    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-bd8496a8-openshift-default-7656dcd86d-txrh4"]
Sep 10 15:43:31.933854 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:31.933566    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-bd8496a8-openshift-default-7656dcd86d-txrh4" podUID="77094e7d-3bf4-4f13-a6b9-9f77ce209b9e" containerName="istio-proxy" containerID="cri-o://39703b1d6fb6db856fdae73252fe7538ce19b07a58819aeac7dbbab7161abe09" gracePeriod=30
Sep 10 15:43:37.193195 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:37.193170    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-bd8496a8-openshift-default-7656dcd86d-txrh4"
Sep 10 15:43:37.246884 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:37.246742    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77094e7d-3bf4-4f13-a6b9-9f77ce209b9e-istiod-ca-cert\") pod \"77094e7d-3bf4-4f13-a6b9-9f77ce209b9e\" (UID: \"77094e7d-3bf4-4f13-a6b9-9f77ce209b9e\") "
Sep 10 15:43:37.246884 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:37.246872    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77094e7d-3bf4-4f13-a6b9-9f77ce209b9e-credential-socket\") pod \"77094e7d-3bf4-4f13-a6b9-9f77ce209b9e\" (UID: \"77094e7d-3bf4-4f13-a6b9-9f77ce209b9e\") "
Sep 10 15:43:37.247107 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:37.246901    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77094e7d-3bf4-4f13-a6b9-9f77ce209b9e-istio-envoy\") pod \"77094e7d-3bf4-4f13-a6b9-9f77ce209b9e\" (UID: \"77094e7d-3bf4-4f13-a6b9-9f77ce209b9e\") "
Sep 10 15:43:37.247107 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:37.246940    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77094e7d-3bf4-4f13-a6b9-9f77ce209b9e-istio-data\") pod \"77094e7d-3bf4-4f13-a6b9-9f77ce209b9e\" (UID: \"77094e7d-3bf4-4f13-a6b9-9f77ce209b9e\") "
Sep 10 15:43:37.247107 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:37.246965    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77094e7d-3bf4-4f13-a6b9-9f77ce209b9e-workload-socket\") pod \"77094e7d-3bf4-4f13-a6b9-9f77ce209b9e\" (UID: \"77094e7d-3bf4-4f13-a6b9-9f77ce209b9e\") "
Sep 10 15:43:37.247107 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:37.246987    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77094e7d-3bf4-4f13-a6b9-9f77ce209b9e-workload-certs\") pod \"77094e7d-3bf4-4f13-a6b9-9f77ce209b9e\" (UID: \"77094e7d-3bf4-4f13-a6b9-9f77ce209b9e\") "
Sep 10 15:43:37.247107 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:37.247010    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7tgj5\" (UniqueName: \"kubernetes.io/projected/77094e7d-3bf4-4f13-a6b9-9f77ce209b9e-kube-api-access-7tgj5\") pod \"77094e7d-3bf4-4f13-a6b9-9f77ce209b9e\" (UID: \"77094e7d-3bf4-4f13-a6b9-9f77ce209b9e\") "
Sep 10 15:43:37.247107 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:37.247085    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77094e7d-3bf4-4f13-a6b9-9f77ce209b9e-istio-podinfo\") pod \"77094e7d-3bf4-4f13-a6b9-9f77ce209b9e\" (UID: \"77094e7d-3bf4-4f13-a6b9-9f77ce209b9e\") "
Sep 10 15:43:37.247383 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:37.247116    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77094e7d-3bf4-4f13-a6b9-9f77ce209b9e-istio-token\") pod \"77094e7d-3bf4-4f13-a6b9-9f77ce209b9e\" (UID: \"77094e7d-3bf4-4f13-a6b9-9f77ce209b9e\") "
Sep 10 15:43:37.247383 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:37.247163    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77094e7d-3bf4-4f13-a6b9-9f77ce209b9e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "77094e7d-3bf4-4f13-a6b9-9f77ce209b9e" (UID: "77094e7d-3bf4-4f13-a6b9-9f77ce209b9e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:43:37.247383 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:37.247121    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77094e7d-3bf4-4f13-a6b9-9f77ce209b9e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "77094e7d-3bf4-4f13-a6b9-9f77ce209b9e" (UID: "77094e7d-3bf4-4f13-a6b9-9f77ce209b9e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:43:37.247610 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:37.247420    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77094e7d-3bf4-4f13-a6b9-9f77ce209b9e-istio-data" (OuterVolumeSpecName: "istio-data") pod "77094e7d-3bf4-4f13-a6b9-9f77ce209b9e" (UID: "77094e7d-3bf4-4f13-a6b9-9f77ce209b9e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:43:37.247610 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:37.247481    2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77094e7d-3bf4-4f13-a6b9-9f77ce209b9e-istiod-ca-cert\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:43:37.247610 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:37.247501    2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77094e7d-3bf4-4f13-a6b9-9f77ce209b9e-credential-socket\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:43:37.247610 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:37.247514    2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77094e7d-3bf4-4f13-a6b9-9f77ce209b9e-istio-data\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:43:37.247894 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:37.247772    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77094e7d-3bf4-4f13-a6b9-9f77ce209b9e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "77094e7d-3bf4-4f13-a6b9-9f77ce209b9e" (UID: "77094e7d-3bf4-4f13-a6b9-9f77ce209b9e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:43:37.248061 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:37.248019    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77094e7d-3bf4-4f13-a6b9-9f77ce209b9e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "77094e7d-3bf4-4f13-a6b9-9f77ce209b9e" (UID: "77094e7d-3bf4-4f13-a6b9-9f77ce209b9e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:43:37.249941 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:37.249911    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77094e7d-3bf4-4f13-a6b9-9f77ce209b9e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "77094e7d-3bf4-4f13-a6b9-9f77ce209b9e" (UID: "77094e7d-3bf4-4f13-a6b9-9f77ce209b9e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:43:37.250487 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:37.250411    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/77094e7d-3bf4-4f13-a6b9-9f77ce209b9e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "77094e7d-3bf4-4f13-a6b9-9f77ce209b9e" (UID: "77094e7d-3bf4-4f13-a6b9-9f77ce209b9e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 15:43:37.250487 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:37.250417    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77094e7d-3bf4-4f13-a6b9-9f77ce209b9e-istio-token" (OuterVolumeSpecName: "istio-token") pod "77094e7d-3bf4-4f13-a6b9-9f77ce209b9e" (UID: "77094e7d-3bf4-4f13-a6b9-9f77ce209b9e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:43:37.250487 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:37.250407    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77094e7d-3bf4-4f13-a6b9-9f77ce209b9e-kube-api-access-7tgj5" (OuterVolumeSpecName: "kube-api-access-7tgj5") pod "77094e7d-3bf4-4f13-a6b9-9f77ce209b9e" (UID: "77094e7d-3bf4-4f13-a6b9-9f77ce209b9e"). InnerVolumeSpecName "kube-api-access-7tgj5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:43:37.349003 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:37.348963    2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77094e7d-3bf4-4f13-a6b9-9f77ce209b9e-istio-envoy\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:43:37.349003 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:37.348994    2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77094e7d-3bf4-4f13-a6b9-9f77ce209b9e-workload-socket\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:43:37.349003 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:37.349003    2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77094e7d-3bf4-4f13-a6b9-9f77ce209b9e-workload-certs\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:43:37.349003 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:37.349013    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7tgj5\" (UniqueName: \"kubernetes.io/projected/77094e7d-3bf4-4f13-a6b9-9f77ce209b9e-kube-api-access-7tgj5\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:43:37.349360 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:37.349022    2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77094e7d-3bf4-4f13-a6b9-9f77ce209b9e-istio-podinfo\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:43:37.349360 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:37.349030    2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77094e7d-3bf4-4f13-a6b9-9f77ce209b9e-istio-token\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:43:37.657767 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:37.657675    2559 generic.go:356] "Generic (PLEG): container finished" podID="77094e7d-3bf4-4f13-a6b9-9f77ce209b9e" containerID="39703b1d6fb6db856fdae73252fe7538ce19b07a58819aeac7dbbab7161abe09" exitCode=0
Sep 10 15:43:37.657767 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:37.657744    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-bd8496a8-openshift-default-7656dcd86d-txrh4"
Sep 10 15:43:37.657961 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:37.657769    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-bd8496a8-openshift-default-7656dcd86d-txrh4" event={"ID":"77094e7d-3bf4-4f13-a6b9-9f77ce209b9e","Type":"ContainerDied","Data":"39703b1d6fb6db856fdae73252fe7538ce19b07a58819aeac7dbbab7161abe09"}
Sep 10 15:43:37.657961 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:37.657863    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-bd8496a8-openshift-default-7656dcd86d-txrh4" event={"ID":"77094e7d-3bf4-4f13-a6b9-9f77ce209b9e","Type":"ContainerDied","Data":"5da13499866bf42218298badf00a16720c27547700b7ab078f5dcf713b791030"}
Sep 10 15:43:37.657961 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:37.657896    2559 scope.go:117] "RemoveContainer" containerID="39703b1d6fb6db856fdae73252fe7538ce19b07a58819aeac7dbbab7161abe09"
Sep 10 15:43:37.667494 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:37.667472    2559 scope.go:117] "RemoveContainer" containerID="c6a8bc511f629d772ddae3cf30af363a150607c5bb73809bd63a1cbc01ffcaa6"
Sep 10 15:43:37.676407 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:37.676393    2559 scope.go:117] "RemoveContainer" containerID="39703b1d6fb6db856fdae73252fe7538ce19b07a58819aeac7dbbab7161abe09"
Sep 10 15:43:37.676694 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:43:37.676676    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39703b1d6fb6db856fdae73252fe7538ce19b07a58819aeac7dbbab7161abe09\": container with ID starting with 39703b1d6fb6db856fdae73252fe7538ce19b07a58819aeac7dbbab7161abe09 not found: ID does not exist" containerID="39703b1d6fb6db856fdae73252fe7538ce19b07a58819aeac7dbbab7161abe09"
Sep 10 15:43:37.676740 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:37.676703    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39703b1d6fb6db856fdae73252fe7538ce19b07a58819aeac7dbbab7161abe09"} err="failed to get container status \"39703b1d6fb6db856fdae73252fe7538ce19b07a58819aeac7dbbab7161abe09\": rpc error: code = NotFound desc = could not find container \"39703b1d6fb6db856fdae73252fe7538ce19b07a58819aeac7dbbab7161abe09\": container with ID starting with 39703b1d6fb6db856fdae73252fe7538ce19b07a58819aeac7dbbab7161abe09 not found: ID does not exist"
Sep 10 15:43:37.676740 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:37.676720    2559 scope.go:117] "RemoveContainer" containerID="c6a8bc511f629d772ddae3cf30af363a150607c5bb73809bd63a1cbc01ffcaa6"
Sep 10 15:43:37.676972 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:43:37.676956    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6a8bc511f629d772ddae3cf30af363a150607c5bb73809bd63a1cbc01ffcaa6\": container with ID starting with c6a8bc511f629d772ddae3cf30af363a150607c5bb73809bd63a1cbc01ffcaa6 not found: ID does not exist" containerID="c6a8bc511f629d772ddae3cf30af363a150607c5bb73809bd63a1cbc01ffcaa6"
Sep 10 15:43:37.677013 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:37.676980    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6a8bc511f629d772ddae3cf30af363a150607c5bb73809bd63a1cbc01ffcaa6"} err="failed to get container status \"c6a8bc511f629d772ddae3cf30af363a150607c5bb73809bd63a1cbc01ffcaa6\": rpc error: code = NotFound desc = could not find container \"c6a8bc511f629d772ddae3cf30af363a150607c5bb73809bd63a1cbc01ffcaa6\": container with ID starting with c6a8bc511f629d772ddae3cf30af363a150607c5bb73809bd63a1cbc01ffcaa6 not found: ID does not exist"
Sep 10 15:43:37.690333 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:37.690305    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-bd8496a8-openshift-default-7656dcd86d-txrh4"]
Sep 10 15:43:37.694932 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:37.694910    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-bd8496a8-openshift-default-7656dcd86d-txrh4"]
Sep 10 15:43:38.784937 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:38.784907    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77094e7d-3bf4-4f13-a6b9-9f77ce209b9e" path="/var/lib/kubelet/pods/77094e7d-3bf4-4f13-a6b9-9f77ce209b9e/volumes"
Sep 10 15:43:48.509691 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:48.509654    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bs7hd"]
Sep 10 15:43:54.162232 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:54.162199    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-bd5806-9d646cccd-pxjzq"]
Sep 10 15:43:54.162708 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:54.162393    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-bd5806-9d646cccd-pxjzq" podUID="4cdbedba-07c4-48fa-bbab-167fe45cbc20" containerName="maas-api" containerID="cri-o://9e53cfa4e6281b417e131e6b0be63f315dba751640afc26b9e41b08fcdbf04cd" gracePeriod=30
Sep 10 15:43:54.413789 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:54.413730    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-bd5806-9d646cccd-pxjzq"
Sep 10 15:43:54.505109 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:54.505071    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4cdbedba-07c4-48fa-bbab-167fe45cbc20-metrics-certs\") pod \"4cdbedba-07c4-48fa-bbab-167fe45cbc20\" (UID: \"4cdbedba-07c4-48fa-bbab-167fe45cbc20\") "
Sep 10 15:43:54.505286 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:54.505149    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cdbedba-07c4-48fa-bbab-167fe45cbc20-openshift-service-ca\") pod \"4cdbedba-07c4-48fa-bbab-167fe45cbc20\" (UID: \"4cdbedba-07c4-48fa-bbab-167fe45cbc20\") "
Sep 10 15:43:54.505286 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:54.505189    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-brm5x\" (UniqueName: \"kubernetes.io/projected/4cdbedba-07c4-48fa-bbab-167fe45cbc20-kube-api-access-brm5x\") pod \"4cdbedba-07c4-48fa-bbab-167fe45cbc20\" (UID: \"4cdbedba-07c4-48fa-bbab-167fe45cbc20\") "
Sep 10 15:43:54.505286 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:54.505207    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4cdbedba-07c4-48fa-bbab-167fe45cbc20-maas-api-tls\") pod \"4cdbedba-07c4-48fa-bbab-167fe45cbc20\" (UID: \"4cdbedba-07c4-48fa-bbab-167fe45cbc20\") "
Sep 10 15:43:54.505536 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:54.505510    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cdbedba-07c4-48fa-bbab-167fe45cbc20-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4cdbedba-07c4-48fa-bbab-167fe45cbc20" (UID: "4cdbedba-07c4-48fa-bbab-167fe45cbc20"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:43:54.507408 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:54.507374    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cdbedba-07c4-48fa-bbab-167fe45cbc20-kube-api-access-brm5x" (OuterVolumeSpecName: "kube-api-access-brm5x") pod "4cdbedba-07c4-48fa-bbab-167fe45cbc20" (UID: "4cdbedba-07c4-48fa-bbab-167fe45cbc20"). InnerVolumeSpecName "kube-api-access-brm5x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:43:54.507408 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:54.507392    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cdbedba-07c4-48fa-bbab-167fe45cbc20-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4cdbedba-07c4-48fa-bbab-167fe45cbc20" (UID: "4cdbedba-07c4-48fa-bbab-167fe45cbc20"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:43:54.507564 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:54.507456    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cdbedba-07c4-48fa-bbab-167fe45cbc20-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "4cdbedba-07c4-48fa-bbab-167fe45cbc20" (UID: "4cdbedba-07c4-48fa-bbab-167fe45cbc20"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:43:54.606549 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:54.606509    2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cdbedba-07c4-48fa-bbab-167fe45cbc20-openshift-service-ca\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:43:54.606549 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:54.606542    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brm5x\" (UniqueName: \"kubernetes.io/projected/4cdbedba-07c4-48fa-bbab-167fe45cbc20-kube-api-access-brm5x\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:43:54.606549 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:54.606553    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4cdbedba-07c4-48fa-bbab-167fe45cbc20-maas-api-tls\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:43:54.606803 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:54.606561    2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4cdbedba-07c4-48fa-bbab-167fe45cbc20-metrics-certs\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:43:54.756998 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:54.756911    2559 generic.go:356] "Generic (PLEG): container finished" podID="4cdbedba-07c4-48fa-bbab-167fe45cbc20" containerID="9e53cfa4e6281b417e131e6b0be63f315dba751640afc26b9e41b08fcdbf04cd" exitCode=0
Sep 10 15:43:54.756998 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:54.756960    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-bd5806-9d646cccd-pxjzq"
Sep 10 15:43:54.757236 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:54.757001    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-bd5806-9d646cccd-pxjzq" event={"ID":"4cdbedba-07c4-48fa-bbab-167fe45cbc20","Type":"ContainerDied","Data":"9e53cfa4e6281b417e131e6b0be63f315dba751640afc26b9e41b08fcdbf04cd"}
Sep 10 15:43:54.757236 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:54.757064    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-bd5806-9d646cccd-pxjzq" event={"ID":"4cdbedba-07c4-48fa-bbab-167fe45cbc20","Type":"ContainerDied","Data":"83bfe95cb5ac3284b79866532485a2193bbb78aeb78e773c88ca6e757e206a18"}
Sep 10 15:43:54.757236 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:54.757093    2559 scope.go:117] "RemoveContainer" containerID="9e53cfa4e6281b417e131e6b0be63f315dba751640afc26b9e41b08fcdbf04cd"
Sep 10 15:43:54.766901 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:54.766884    2559 scope.go:117] "RemoveContainer" containerID="9e53cfa4e6281b417e131e6b0be63f315dba751640afc26b9e41b08fcdbf04cd"
Sep 10 15:43:54.767415 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:43:54.767295    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e53cfa4e6281b417e131e6b0be63f315dba751640afc26b9e41b08fcdbf04cd\": container with ID starting with 9e53cfa4e6281b417e131e6b0be63f315dba751640afc26b9e41b08fcdbf04cd not found: ID does not exist" containerID="9e53cfa4e6281b417e131e6b0be63f315dba751640afc26b9e41b08fcdbf04cd"
Sep 10 15:43:54.767415 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:54.767352    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e53cfa4e6281b417e131e6b0be63f315dba751640afc26b9e41b08fcdbf04cd"} err="failed to get container status \"9e53cfa4e6281b417e131e6b0be63f315dba751640afc26b9e41b08fcdbf04cd\": rpc error: code = NotFound desc = could not find container \"9e53cfa4e6281b417e131e6b0be63f315dba751640afc26b9e41b08fcdbf04cd\": container with ID starting with 9e53cfa4e6281b417e131e6b0be63f315dba751640afc26b9e41b08fcdbf04cd not found: ID does not exist"
Sep 10 15:43:54.786188 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:54.786153    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-bd5806-9d646cccd-pxjzq"]
Sep 10 15:43:54.786188 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:54.786191    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-bd5806-9d646cccd-pxjzq"]
Sep 10 15:43:56.784945 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:43:56.784912    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4cdbedba-07c4-48fa-bbab-167fe45cbc20" path="/var/lib/kubelet/pods/4cdbedba-07c4-48fa-bbab-167fe45cbc20/volumes"
Sep 10 15:44:06.784763 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:44:06.784734    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-fa9f2a54-kserve-54cb4c6799-xdv4x"]
Sep 10 15:44:08.463961 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:44:08.463923    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-fc495e-openshift-default-bcdb5fcb-bgtbx"]
Sep 10 15:44:08.464387 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:44:08.464327    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4cdbedba-07c4-48fa-bbab-167fe45cbc20" containerName="maas-api"
Sep 10 15:44:08.464387 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:44:08.464341    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="4cdbedba-07c4-48fa-bbab-167fe45cbc20" containerName="maas-api"
Sep 10 15:44:08.464387 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:44:08.464352    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77094e7d-3bf4-4f13-a6b9-9f77ce209b9e" containerName="istio-proxy"
Sep 10 15:44:08.464387 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:44:08.464358    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="77094e7d-3bf4-4f13-a6b9-9f77ce209b9e" containerName="istio-proxy"
Sep 10 15:44:08.464606 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:44:08.464440    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="77094e7d-3bf4-4f13-a6b9-9f77ce209b9e" containerName="istio-proxy"
Sep 10 15:44:08.464606 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:44:08.464455    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="77094e7d-3bf4-4f13-a6b9-9f77ce209b9e" containerName="istio-proxy"
Sep 10 15:44:08.464606 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:44:08.464464    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="4cdbedba-07c4-48fa-bbab-167fe45cbc20" containerName="maas-api"
Sep 10 15:44:08.468965 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:44:08.468949    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-fc495e-openshift-default-bcdb5fcb-bgtbx"
Sep 10 15:44:08.472162 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:44:08.472140    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-fc495e-openshift-default-dockercfg-gcxr7\""
Sep 10 15:44:08.518192 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:44:08.518160    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-fc495e-openshift-default-bcdb5fcb-bgtbx"]
Sep 10 15:44:08.634953 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:44:08.634918    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpp8v\" (UniqueName: \"kubernetes.io/projected/c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7-kube-api-access-rpp8v\") pod \"e2e-explicit-fc495e-openshift-default-bcdb5fcb-bgtbx\" (UID: \"c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7\") " pod="openshift-ingress/e2e-explicit-fc495e-openshift-default-bcdb5fcb-bgtbx"
Sep 10 15:44:08.635128 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:44:08.634964    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7-istio-token\") pod \"e2e-explicit-fc495e-openshift-default-bcdb5fcb-bgtbx\" (UID: \"c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7\") " pod="openshift-ingress/e2e-explicit-fc495e-openshift-default-bcdb5fcb-bgtbx"
Sep 10 15:44:08.635128 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:44:08.634987    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7-istio-data\") pod \"e2e-explicit-fc495e-openshift-default-bcdb5fcb-bgtbx\" (UID: \"c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7\") " pod="openshift-ingress/e2e-explicit-fc495e-openshift-default-bcdb5fcb-bgtbx"
Sep 10 15:44:08.635128 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:44:08.635071    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7-workload-certs\") pod \"e2e-explicit-fc495e-openshift-default-bcdb5fcb-bgtbx\" (UID: \"c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7\") " pod="openshift-ingress/e2e-explicit-fc495e-openshift-default-bcdb5fcb-bgtbx"
Sep 10 15:44:08.635128 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:44:08.635113    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7-istio-envoy\") pod \"e2e-explicit-fc495e-openshift-default-bcdb5fcb-bgtbx\" (UID: \"c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7\") " pod="openshift-ingress/e2e-explicit-fc495e-openshift-default-bcdb5fcb-bgtbx"
Sep 10 15:44:08.635275 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:44:08.635172    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7-istiod-ca-cert\") pod \"e2e-explicit-fc495e-openshift-default-bcdb5fcb-bgtbx\" (UID: \"c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7\") " pod="openshift-ingress/e2e-explicit-fc495e-openshift-default-bcdb5fcb-bgtbx"
Sep 10 15:44:08.635275 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:44:08.635229    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7-credential-socket\") pod \"e2e-explicit-fc495e-openshift-default-bcdb5fcb-bgtbx\" (UID: \"c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7\") " pod="openshift-ingress/e2e-explicit-fc495e-openshift-default-bcdb5fcb-bgtbx"
Sep 10 15:44:08.635275 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:44:08.635252    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7-istio-podinfo\") pod \"e2e-explicit-fc495e-openshift-default-bcdb5fcb-bgtbx\" (UID: \"c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7\") " pod="openshift-ingress/e2e-explicit-fc495e-openshift-default-bcdb5fcb-bgtbx"
Sep 10 15:44:08.635275 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:44:08.635271    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7-workload-socket\") pod \"e2e-explicit-fc495e-openshift-default-bcdb5fcb-bgtbx\" (UID: \"c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7\") " pod="openshift-ingress/e2e-explicit-fc495e-openshift-default-bcdb5fcb-bgtbx"
Sep 10 15:44:08.736852 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:44:08.736769    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7-credential-socket\") pod \"e2e-explicit-fc495e-openshift-default-bcdb5fcb-bgtbx\" (UID: \"c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7\") " pod="openshift-ingress/e2e-explicit-fc495e-openshift-default-bcdb5fcb-bgtbx"
Sep 10 15:44:08.736852 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:44:08.736812    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7-istio-podinfo\") pod \"e2e-explicit-fc495e-openshift-default-bcdb5fcb-bgtbx\" (UID: \"c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7\") " pod="openshift-ingress/e2e-explicit-fc495e-openshift-default-bcdb5fcb-bgtbx"
Sep 10 15:44:08.737038 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:44:08.736928    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7-workload-socket\") pod \"e2e-explicit-fc495e-openshift-default-bcdb5fcb-bgtbx\" (UID: \"c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7\") " pod="openshift-ingress/e2e-explicit-fc495e-openshift-default-bcdb5fcb-bgtbx"
Sep 10 15:44:08.737038 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:44:08.736983    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpp8v\" (UniqueName: \"kubernetes.io/projected/c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7-kube-api-access-rpp8v\") pod \"e2e-explicit-fc495e-openshift-default-bcdb5fcb-bgtbx\" (UID: \"c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7\") " pod="openshift-ingress/e2e-explicit-fc495e-openshift-default-bcdb5fcb-bgtbx"
Sep 10 15:44:08.737038 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:44:08.737026    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7-istio-token\") pod \"e2e-explicit-fc495e-openshift-default-bcdb5fcb-bgtbx\" (UID: \"c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7\") " pod="openshift-ingress/e2e-explicit-fc495e-openshift-default-bcdb5fcb-bgtbx"
Sep 10 15:44:08.737159 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:44:08.737049    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7-istio-data\") pod \"e2e-explicit-fc495e-openshift-default-bcdb5fcb-bgtbx\" (UID: \"c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7\") " pod="openshift-ingress/e2e-explicit-fc495e-openshift-default-bcdb5fcb-bgtbx"
Sep 10 15:44:08.737159 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:44:08.737108    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7-workload-certs\") pod \"e2e-explicit-fc495e-openshift-default-bcdb5fcb-bgtbx\" (UID: \"c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7\") " pod="openshift-ingress/e2e-explicit-fc495e-openshift-default-bcdb5fcb-bgtbx"
Sep 10 15:44:08.737159 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:44:08.737148    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7-istio-envoy\") pod \"e2e-explicit-fc495e-openshift-default-bcdb5fcb-bgtbx\" (UID: \"c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7\") " pod="openshift-ingress/e2e-explicit-fc495e-openshift-default-bcdb5fcb-bgtbx"
Sep 10 15:44:08.737318 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:44:08.737205    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7-istiod-ca-cert\") pod \"e2e-explicit-fc495e-openshift-default-bcdb5fcb-bgtbx\" (UID: \"c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7\") " pod="openshift-ingress/e2e-explicit-fc495e-openshift-default-bcdb5fcb-bgtbx"
Sep 10 15:44:08.737318 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:44:08.737238    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7-credential-socket\") pod \"e2e-explicit-fc495e-openshift-default-bcdb5fcb-bgtbx\" (UID: \"c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7\") " pod="openshift-ingress/e2e-explicit-fc495e-openshift-default-bcdb5fcb-bgtbx"
Sep 10 15:44:08.737318 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:44:08.737243    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7-workload-socket\") pod \"e2e-explicit-fc495e-openshift-default-bcdb5fcb-bgtbx\" (UID: \"c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7\") " pod="openshift-ingress/e2e-explicit-fc495e-openshift-default-bcdb5fcb-bgtbx"
Sep 10 15:44:08.737588 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:44:08.737555    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7-workload-certs\") pod \"e2e-explicit-fc495e-openshift-default-bcdb5fcb-bgtbx\" (UID: \"c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7\") " pod="openshift-ingress/e2e-explicit-fc495e-openshift-default-bcdb5fcb-bgtbx"
Sep 10 15:44:08.737822 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:44:08.737802    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7-istio-data\") pod \"e2e-explicit-fc495e-openshift-default-bcdb5fcb-bgtbx\" (UID: \"c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7\") " pod="openshift-ingress/e2e-explicit-fc495e-openshift-default-bcdb5fcb-bgtbx"
Sep 10 15:44:08.737906 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:44:08.737858    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7-istiod-ca-cert\") pod \"e2e-explicit-fc495e-openshift-default-bcdb5fcb-bgtbx\" (UID: \"c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7\") " pod="openshift-ingress/e2e-explicit-fc495e-openshift-default-bcdb5fcb-bgtbx"
Sep 10 15:44:08.739277 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:44:08.739255    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7-istio-envoy\") pod \"e2e-explicit-fc495e-openshift-default-bcdb5fcb-bgtbx\" (UID: \"c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7\") " pod="openshift-ingress/e2e-explicit-fc495e-openshift-default-bcdb5fcb-bgtbx"
Sep 10 15:44:08.739372 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:44:08.739342    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7-istio-podinfo\") pod \"e2e-explicit-fc495e-openshift-default-bcdb5fcb-bgtbx\" (UID: \"c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7\") " pod="openshift-ingress/e2e-explicit-fc495e-openshift-default-bcdb5fcb-bgtbx"
Sep 10 15:44:08.771286 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:44:08.771259    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7-istio-token\") pod \"e2e-explicit-fc495e-openshift-default-bcdb5fcb-bgtbx\" (UID: \"c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7\") " pod="openshift-ingress/e2e-explicit-fc495e-openshift-default-bcdb5fcb-bgtbx"
Sep 10 15:44:08.772770 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:44:08.772752    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpp8v\" (UniqueName: \"kubernetes.io/projected/c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7-kube-api-access-rpp8v\") pod \"e2e-explicit-fc495e-openshift-default-bcdb5fcb-bgtbx\" (UID: \"c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7\") " pod="openshift-ingress/e2e-explicit-fc495e-openshift-default-bcdb5fcb-bgtbx"
Sep 10 15:44:08.778697 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:44:08.778674    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-fc495e-openshift-default-bcdb5fcb-bgtbx"
Sep 10 15:44:09.067172 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:44:09.067146    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-fc495e-openshift-default-bcdb5fcb-bgtbx"]
Sep 10 15:44:09.069193 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:44:09.069165    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3aea63e_68a5_4e23_8e0e_e3bbbb23a1c7.slice/crio-d78f6057f6333b7fa1f239415533ecf491ff4d70e9f86da7b367dcb75f32a199 WatchSource:0}: Error finding container d78f6057f6333b7fa1f239415533ecf491ff4d70e9f86da7b367dcb75f32a199: Status 404 returned error can't find the container with id d78f6057f6333b7fa1f239415533ecf491ff4d70e9f86da7b367dcb75f32a199
Sep 10 15:44:09.071486 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:44:09.071455    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 15:44:09.071597 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:44:09.071520    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 15:44:09.071597 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:44:09.071550    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 15:44:09.511275 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:44:09.511240    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bs7hd"]
Sep 10 15:44:09.845854 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:44:09.845755    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-fc495e-openshift-default-bcdb5fcb-bgtbx" event={"ID":"c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7","Type":"ContainerStarted","Data":"8e05245294c3a4890181a60845683c17125cd434811b8c2a7eb9a7cba79a0bb8"}
Sep 10 15:44:09.845854 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:44:09.845812    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-fc495e-openshift-default-bcdb5fcb-bgtbx" event={"ID":"c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7","Type":"ContainerStarted","Data":"d78f6057f6333b7fa1f239415533ecf491ff4d70e9f86da7b367dcb75f32a199"}
Sep 10 15:44:09.872362 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:44:09.872312    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-fc495e-openshift-default-bcdb5fcb-bgtbx" podStartSLOduration=1.8722962490000001 podStartE2EDuration="1.872296249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:44:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:44:09.870211185 +0000 UTC m=+1025.612176300" watchObservedRunningTime="2026-09-10 15:44:09.872296249 +0000 UTC m=+1025.614261352"
Sep 10 15:44:10.785242 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:44:10.785212    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-fc495e-openshift-default-bcdb5fcb-bgtbx"
Sep 10 15:44:10.785636 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:44:10.785302    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-fc495e-openshift-default-bcdb5fcb-bgtbx"
Sep 10 15:44:10.851966 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:44:10.851933    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-fc495e-openshift-default-bcdb5fcb-bgtbx"
Sep 10 15:44:10.852774 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:44:10.852753    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-fc495e-openshift-default-bcdb5fcb-bgtbx"
Sep 10 15:44:21.804126 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:44:21.804087    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-a-fa9f2a54-kserve-54cb4c6799-xdv4x" podUID="a55c9635-edc4-4efe-b78c-e0c291910e61" containerName="main" containerID="cri-o://c6d17b7553443c672776935965f1db3fe9313f21356d33ae5cef0352c923a92a" gracePeriod=45
Sep 10 15:44:22.095679 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:44:22.095657    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-fa9f2a54-kserve-54cb4c6799-xdv4x"
Sep 10 15:44:22.171883 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:44:22.171849    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a55c9635-edc4-4efe-b78c-e0c291910e61-tmp-dir\") pod \"a55c9635-edc4-4efe-b78c-e0c291910e61\" (UID: \"a55c9635-edc4-4efe-b78c-e0c291910e61\") "
Sep 10 15:44:22.171883 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:44:22.171886    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a55c9635-edc4-4efe-b78c-e0c291910e61-model-cache\") pod \"a55c9635-edc4-4efe-b78c-e0c291910e61\" (UID: \"a55c9635-edc4-4efe-b78c-e0c291910e61\") "
Sep 10 15:44:22.172126 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:44:22.171906    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a55c9635-edc4-4efe-b78c-e0c291910e61-tls-certs\") pod \"a55c9635-edc4-4efe-b78c-e0c291910e61\" (UID: \"a55c9635-edc4-4efe-b78c-e0c291910e61\") "
Sep 10 15:44:22.172126 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:44:22.171929    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a55c9635-edc4-4efe-b78c-e0c291910e61-home\") pod \"a55c9635-edc4-4efe-b78c-e0c291910e61\" (UID: \"a55c9635-edc4-4efe-b78c-e0c291910e61\") "
Sep 10 15:44:22.172126 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:44:22.171990    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a55c9635-edc4-4efe-b78c-e0c291910e61-dshm\") pod \"a55c9635-edc4-4efe-b78c-e0c291910e61\" (UID: \"a55c9635-edc4-4efe-b78c-e0c291910e61\") "
Sep 10 15:44:22.172126 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:44:22.172031    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9kcnf\" (UniqueName: \"kubernetes.io/projected/a55c9635-edc4-4efe-b78c-e0c291910e61-kube-api-access-9kcnf\") pod \"a55c9635-edc4-4efe-b78c-e0c291910e61\" (UID: \"a55c9635-edc4-4efe-b78c-e0c291910e61\") "
Sep 10 15:44:22.172126 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:44:22.172105    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a55c9635-edc4-4efe-b78c-e0c291910e61-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a55c9635-edc4-4efe-b78c-e0c291910e61" (UID: "a55c9635-edc4-4efe-b78c-e0c291910e61"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:44:22.172126 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:44:22.172121    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a55c9635-edc4-4efe-b78c-e0c291910e61-model-cache" (OuterVolumeSpecName: "model-cache") pod "a55c9635-edc4-4efe-b78c-e0c291910e61" (UID: "a55c9635-edc4-4efe-b78c-e0c291910e61"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:44:22.172426 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:44:22.172180    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a55c9635-edc4-4efe-b78c-e0c291910e61-home" (OuterVolumeSpecName: "home") pod "a55c9635-edc4-4efe-b78c-e0c291910e61" (UID: "a55c9635-edc4-4efe-b78c-e0c291910e61"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:44:22.172426 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:44:22.172387    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a55c9635-edc4-4efe-b78c-e0c291910e61-tmp-dir\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:44:22.172426 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:44:22.172414    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a55c9635-edc4-4efe-b78c-e0c291910e61-model-cache\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:44:22.172547 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:44:22.172429    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a55c9635-edc4-4efe-b78c-e0c291910e61-home\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:44:22.174054 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:44:22.174023    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a55c9635-edc4-4efe-b78c-e0c291910e61-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a55c9635-edc4-4efe-b78c-e0c291910e61" (UID: "a55c9635-edc4-4efe-b78c-e0c291910e61"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:44:22.174161 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:44:22.174112    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a55c9635-edc4-4efe-b78c-e0c291910e61-dshm" (OuterVolumeSpecName: "dshm") pod "a55c9635-edc4-4efe-b78c-e0c291910e61" (UID: "a55c9635-edc4-4efe-b78c-e0c291910e61"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:44:22.174203 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:44:22.174190    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a55c9635-edc4-4efe-b78c-e0c291910e61-kube-api-access-9kcnf" (OuterVolumeSpecName: "kube-api-access-9kcnf") pod "a55c9635-edc4-4efe-b78c-e0c291910e61" (UID: "a55c9635-edc4-4efe-b78c-e0c291910e61"). InnerVolumeSpecName "kube-api-access-9kcnf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:44:22.273523 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:44:22.273489    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a55c9635-edc4-4efe-b78c-e0c291910e61-dshm\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:44:22.273523 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:44:22.273521    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kcnf\" (UniqueName: \"kubernetes.io/projected/a55c9635-edc4-4efe-b78c-e0c291910e61-kube-api-access-9kcnf\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:44:22.273747 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:44:22.273532    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a55c9635-edc4-4efe-b78c-e0c291910e61-tls-certs\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:44:22.918423 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:44:22.918390    2559 generic.go:356] "Generic (PLEG): container finished" podID="a55c9635-edc4-4efe-b78c-e0c291910e61" containerID="c6d17b7553443c672776935965f1db3fe9313f21356d33ae5cef0352c923a92a" exitCode=0
Sep 10 15:44:22.918840 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:44:22.918433    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-fa9f2a54-kserve-54cb4c6799-xdv4x"
Sep 10 15:44:22.918840 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:44:22.918464    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-fa9f2a54-kserve-54cb4c6799-xdv4x" event={"ID":"a55c9635-edc4-4efe-b78c-e0c291910e61","Type":"ContainerDied","Data":"c6d17b7553443c672776935965f1db3fe9313f21356d33ae5cef0352c923a92a"}
Sep 10 15:44:22.918840 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:44:22.918501    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-fa9f2a54-kserve-54cb4c6799-xdv4x" event={"ID":"a55c9635-edc4-4efe-b78c-e0c291910e61","Type":"ContainerDied","Data":"c03d6e9a50707be089e0ef8661715f35e52c765e7b132c9b1e0fc397922a289e"}
Sep 10 15:44:22.918840 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:44:22.918518    2559 scope.go:117] "RemoveContainer" containerID="c6d17b7553443c672776935965f1db3fe9313f21356d33ae5cef0352c923a92a"
Sep 10 15:44:22.929841 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:44:22.929816    2559 scope.go:117] "RemoveContainer" containerID="c6d17b7553443c672776935965f1db3fe9313f21356d33ae5cef0352c923a92a"
Sep 10 15:44:22.930117 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:44:22.930097    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6d17b7553443c672776935965f1db3fe9313f21356d33ae5cef0352c923a92a\": container with ID starting with c6d17b7553443c672776935965f1db3fe9313f21356d33ae5cef0352c923a92a not found: ID does not exist" containerID="c6d17b7553443c672776935965f1db3fe9313f21356d33ae5cef0352c923a92a"
Sep 10 15:44:22.930185 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:44:22.930138    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6d17b7553443c672776935965f1db3fe9313f21356d33ae5cef0352c923a92a"} err="failed to get container status \"c6d17b7553443c672776935965f1db3fe9313f21356d33ae5cef0352c923a92a\": rpc error: code = NotFound desc = could not find container \"c6d17b7553443c672776935965f1db3fe9313f21356d33ae5cef0352c923a92a\": container with ID starting with c6d17b7553443c672776935965f1db3fe9313f21356d33ae5cef0352c923a92a not found: ID does not exist"
Sep 10 15:44:22.950675 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:44:22.950643    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-fa9f2a54-kserve-54cb4c6799-xdv4x"]
Sep 10 15:44:22.953182 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:44:22.953154    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-a-fa9f2a54-kserve-54cb4c6799-xdv4x"]
Sep 10 15:44:24.786707 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:44:24.786671    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a55c9635-edc4-4efe-b78c-e0c291910e61" path="/var/lib/kubelet/pods/a55c9635-edc4-4efe-b78c-e0c291910e61/volumes"
Sep 10 15:44:28.610983 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:44:28.610939    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bs7hd"]
Sep 10 15:44:29.959947 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:44:29.959912    2559 generic.go:356] "Generic (PLEG): container finished" podID="c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7" containerID="8e05245294c3a4890181a60845683c17125cd434811b8c2a7eb9a7cba79a0bb8" exitCode=0
Sep 10 15:44:29.960391 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:44:29.959987    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-fc495e-openshift-default-bcdb5fcb-bgtbx" event={"ID":"c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7","Type":"ContainerDied","Data":"8e05245294c3a4890181a60845683c17125cd434811b8c2a7eb9a7cba79a0bb8"}
Sep 10 15:44:29.960391 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:44:29.960387    2559 scope.go:117] "RemoveContainer" containerID="8e05245294c3a4890181a60845683c17125cd434811b8c2a7eb9a7cba79a0bb8"
Sep 10 15:44:29.961440 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:44:29.961415    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 15:44:29.961493 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:44:29.961483    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 15:44:29.961529 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:44:29.961513    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 15:44:30.972520 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:44:30.972473    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-fc495e-openshift-default-bcdb5fcb-bgtbx" event={"ID":"c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7","Type":"ContainerStarted","Data":"e6d3af49ad385b52f371b15106fbb9aaffc96202a4a1abe92835107164d5ee88"}
Sep 10 15:44:31.779154 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:44:31.779109    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-fc495e-openshift-default-bcdb5fcb-bgtbx"
Sep 10 15:44:31.779379 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:44:31.779356    2559 patch_prober.go:28] interesting pod/e2e-explicit-fc495e-openshift-default-bcdb5fcb-bgtbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Sep 10 15:44:31.779460 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:44:31.779411    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-fc495e-openshift-default-bcdb5fcb-bgtbx" podUID="c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Sep 10 15:44:32.779174 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:44:32.779138    2559 patch_prober.go:28] interesting pod/e2e-explicit-fc495e-openshift-default-bcdb5fcb-bgtbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Sep 10 15:44:32.779565 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:44:32.779196    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-fc495e-openshift-default-bcdb5fcb-bgtbx" podUID="c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Sep 10 15:44:33.779036 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:44:33.779001    2559 patch_prober.go:28] interesting pod/e2e-explicit-fc495e-openshift-default-bcdb5fcb-bgtbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Sep 10 15:44:33.779206 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:44:33.779057    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-fc495e-openshift-default-bcdb5fcb-bgtbx" podUID="c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Sep 10 15:44:34.779283 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:44:34.779247    2559 patch_prober.go:28] interesting pod/e2e-explicit-fc495e-openshift-default-bcdb5fcb-bgtbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Sep 10 15:44:34.779677 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:44:34.779310    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-fc495e-openshift-default-bcdb5fcb-bgtbx" podUID="c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Sep 10 15:44:35.779357 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:44:35.779318    2559 patch_prober.go:28] interesting pod/e2e-explicit-fc495e-openshift-default-bcdb5fcb-bgtbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Sep 10 15:44:35.779871 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:44:35.779376    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-fc495e-openshift-default-bcdb5fcb-bgtbx" podUID="c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Sep 10 15:44:36.786329 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:44:36.786301    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-fc495e-openshift-default-bcdb5fcb-bgtbx"
Sep 10 15:44:36.786735 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:44:36.786566    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-fc495e-openshift-default-bcdb5fcb-bgtbx"
Sep 10 15:44:36.787286 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:44:36.787264    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-fc495e-openshift-default-bcdb5fcb-bgtbx"
Sep 10 15:44:48.503447 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:44:48.503407    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bs7hd"]
Sep 10 15:45:00.150489 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:00.150444    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-90ca97-29817585-6mdbm"]
Sep 10 15:45:00.150887 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:00.150858    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77094e7d-3bf4-4f13-a6b9-9f77ce209b9e" containerName="istio-proxy"
Sep 10 15:45:00.150887 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:00.150870    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="77094e7d-3bf4-4f13-a6b9-9f77ce209b9e" containerName="istio-proxy"
Sep 10 15:45:00.150959 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:00.150895    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a55c9635-edc4-4efe-b78c-e0c291910e61" containerName="main"
Sep 10 15:45:00.150959 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:00.150900    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a55c9635-edc4-4efe-b78c-e0c291910e61" containerName="main"
Sep 10 15:45:00.150959 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:00.150955    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="a55c9635-edc4-4efe-b78c-e0c291910e61" containerName="main"
Sep 10 15:45:00.154158 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:00.154142    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817585-wqsgn"]
Sep 10 15:45:00.154275 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:00.154255    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-90ca97-29817585-6mdbm"
Sep 10 15:45:00.157381 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:00.157070    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vfhtz\""
Sep 10 15:45:00.157557 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:00.157459    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817585-wqsgn"
Sep 10 15:45:00.161567 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:00.161542    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817585-wqsgn"]
Sep 10 15:45:00.165416 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:00.165216    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-90ca97-29817585-6mdbm"]
Sep 10 15:45:00.170044 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:00.170021    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-90ca97-29817585-6mdbm"
Sep 10 15:45:00.177148 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:00.177129    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817585-wqsgn"
Sep 10 15:45:00.449535 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:00.449505    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-90ca97-29817585-6mdbm"]
Sep 10 15:45:00.451721 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:45:00.451690    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9f1fcbb_c17a_44eb_9b2c_986fc2869601.slice/crio-dfe8107ea537d46f602096713156cd701246664367ad4072e907741dbc622a81 WatchSource:0}: Error finding container dfe8107ea537d46f602096713156cd701246664367ad4072e907741dbc622a81: Status 404 returned error can't find the container with id dfe8107ea537d46f602096713156cd701246664367ad4072e907741dbc622a81
Sep 10 15:45:00.453723 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:00.453705    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 15:45:00.473632 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:00.473609    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817585-wqsgn"]
Sep 10 15:45:00.475397 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:45:00.475366    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1f6c3c4_a9b8_4783_9a5a_218e1d82d727.slice/crio-035faa1531643defafb487bad04149a807a10ba22487a395d01e097d7683fb54 WatchSource:0}: Error finding container 035faa1531643defafb487bad04149a807a10ba22487a395d01e097d7683fb54: Status 404 returned error can't find the container with id 035faa1531643defafb487bad04149a807a10ba22487a395d01e097d7683fb54
Sep 10 15:45:01.138046 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:01.138009    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817585-wqsgn" event={"ID":"f1f6c3c4-a9b8-4783-9a5a-218e1d82d727","Type":"ContainerStarted","Data":"035faa1531643defafb487bad04149a807a10ba22487a395d01e097d7683fb54"}
Sep 10 15:45:01.139142 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:01.139118    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-90ca97-29817585-6mdbm" event={"ID":"e9f1fcbb-c17a-44eb-9b2c-986fc2869601","Type":"ContainerStarted","Data":"dfe8107ea537d46f602096713156cd701246664367ad4072e907741dbc622a81"}
Sep 10 15:45:04.163446 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:04.163408    2559 generic.go:356] "Generic (PLEG): container finished" podID="f1f6c3c4-a9b8-4783-9a5a-218e1d82d727" containerID="baed08b5c7143c53f97a291def069635c0e9a9f955bd425dd11ae5ca48fe6d6d" exitCode=0
Sep 10 15:45:04.163912 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:04.163502    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817585-wqsgn" event={"ID":"f1f6c3c4-a9b8-4783-9a5a-218e1d82d727","Type":"ContainerDied","Data":"baed08b5c7143c53f97a291def069635c0e9a9f955bd425dd11ae5ca48fe6d6d"}
Sep 10 15:45:04.165169 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:04.165134    2559 generic.go:356] "Generic (PLEG): container finished" podID="e9f1fcbb-c17a-44eb-9b2c-986fc2869601" containerID="e9a16da97b706ec2381a6082ed604e88674367dc2dff31ff6d8ee72c5edc69ea" exitCode=0
Sep 10 15:45:04.165303 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:04.165223    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-90ca97-29817585-6mdbm" event={"ID":"e9f1fcbb-c17a-44eb-9b2c-986fc2869601","Type":"ContainerDied","Data":"e9a16da97b706ec2381a6082ed604e88674367dc2dff31ff6d8ee72c5edc69ea"}
Sep 10 15:45:05.310889 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:05.310859    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-90ca97-29817585-6mdbm"
Sep 10 15:45:05.344255 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:05.344227    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817585-wqsgn"
Sep 10 15:45:06.006104 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:06.006065    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bs7hd"]
Sep 10 15:45:06.179071 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:06.179042    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817585-wqsgn"
Sep 10 15:45:06.179071 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:06.179061    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817585-wqsgn" event={"ID":"f1f6c3c4-a9b8-4783-9a5a-218e1d82d727","Type":"ContainerDied","Data":"035faa1531643defafb487bad04149a807a10ba22487a395d01e097d7683fb54"}
Sep 10 15:45:06.179388 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:06.179121    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="035faa1531643defafb487bad04149a807a10ba22487a395d01e097d7683fb54"
Sep 10 15:45:06.181402 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:06.181374    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-90ca97-29817585-6mdbm" event={"ID":"e9f1fcbb-c17a-44eb-9b2c-986fc2869601","Type":"ContainerDied","Data":"dfe8107ea537d46f602096713156cd701246664367ad4072e907741dbc622a81"}
Sep 10 15:45:06.181547 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:06.181422    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dfe8107ea537d46f602096713156cd701246664367ad4072e907741dbc622a81"
Sep 10 15:45:06.181547 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:06.181384    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-90ca97-29817585-6mdbm"
Sep 10 15:45:06.204233 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:06.204205    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-fc495e/explicit-model-fc495e-kserve-79dff7dd85-85vkt"]
Sep 10 15:45:06.204601 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:06.204572    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1f6c3c4-a9b8-4783-9a5a-218e1d82d727" containerName="cleanup"
Sep 10 15:45:06.204644 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:06.204603    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1f6c3c4-a9b8-4783-9a5a-218e1d82d727" containerName="cleanup"
Sep 10 15:45:06.204644 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:06.204623    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9f1fcbb-c17a-44eb-9b2c-986fc2869601" containerName="cleanup"
Sep 10 15:45:06.204644 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:06.204628    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9f1fcbb-c17a-44eb-9b2c-986fc2869601" containerName="cleanup"
Sep 10 15:45:06.204742 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:06.204686    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1f6c3c4-a9b8-4783-9a5a-218e1d82d727" containerName="cleanup"
Sep 10 15:45:06.204742 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:06.204698    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9f1fcbb-c17a-44eb-9b2c-986fc2869601" containerName="cleanup"
Sep 10 15:45:06.207827 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:06.207810    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-fc495e/explicit-model-fc495e-kserve-79dff7dd85-85vkt"
Sep 10 15:45:06.210876 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:06.210851    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-fc495e\"/\"default-dockercfg-b5d27\""
Sep 10 15:45:06.210989 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:06.210882    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-fc495e\"/\"openshift-service-ca.crt\""
Sep 10 15:45:06.210989 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:06.210911    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-fc495e\"/\"kube-root-ca.crt\""
Sep 10 15:45:06.210989 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:06.210911    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-fc495e\"/\"explicit-model-fc495e-kserve-self-signed-certs\""
Sep 10 15:45:06.216827 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:06.216802    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-fc495e/explicit-model-fc495e-kserve-79dff7dd85-85vkt"]
Sep 10 15:45:06.288268 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:06.288173    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1ee57bf-1a5b-4b28-a528-54cfa7fd380a-model-cache\") pod \"explicit-model-fc495e-kserve-79dff7dd85-85vkt\" (UID: \"f1ee57bf-1a5b-4b28-a528-54cfa7fd380a\") " pod="ai-tenant-e2e-explicit-fc495e/explicit-model-fc495e-kserve-79dff7dd85-85vkt"
Sep 10 15:45:06.288268 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:06.288208    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1ee57bf-1a5b-4b28-a528-54cfa7fd380a-tls-certs\") pod \"explicit-model-fc495e-kserve-79dff7dd85-85vkt\" (UID: \"f1ee57bf-1a5b-4b28-a528-54cfa7fd380a\") " pod="ai-tenant-e2e-explicit-fc495e/explicit-model-fc495e-kserve-79dff7dd85-85vkt"
Sep 10 15:45:06.288268 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:06.288239    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1ee57bf-1a5b-4b28-a528-54cfa7fd380a-home\") pod \"explicit-model-fc495e-kserve-79dff7dd85-85vkt\" (UID: \"f1ee57bf-1a5b-4b28-a528-54cfa7fd380a\") " pod="ai-tenant-e2e-explicit-fc495e/explicit-model-fc495e-kserve-79dff7dd85-85vkt"
Sep 10 15:45:06.288559 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:06.288294    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1ee57bf-1a5b-4b28-a528-54cfa7fd380a-dshm\") pod \"explicit-model-fc495e-kserve-79dff7dd85-85vkt\" (UID: \"f1ee57bf-1a5b-4b28-a528-54cfa7fd380a\") " pod="ai-tenant-e2e-explicit-fc495e/explicit-model-fc495e-kserve-79dff7dd85-85vkt"
Sep 10 15:45:06.288559 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:06.288392    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1ee57bf-1a5b-4b28-a528-54cfa7fd380a-tmp-dir\") pod \"explicit-model-fc495e-kserve-79dff7dd85-85vkt\" (UID: \"f1ee57bf-1a5b-4b28-a528-54cfa7fd380a\") " pod="ai-tenant-e2e-explicit-fc495e/explicit-model-fc495e-kserve-79dff7dd85-85vkt"
Sep 10 15:45:06.288559 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:06.288448    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gctzr\" (UniqueName: \"kubernetes.io/projected/f1ee57bf-1a5b-4b28-a528-54cfa7fd380a-kube-api-access-gctzr\") pod \"explicit-model-fc495e-kserve-79dff7dd85-85vkt\" (UID: \"f1ee57bf-1a5b-4b28-a528-54cfa7fd380a\") " pod="ai-tenant-e2e-explicit-fc495e/explicit-model-fc495e-kserve-79dff7dd85-85vkt"
Sep 10 15:45:06.389670 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:06.389630    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1ee57bf-1a5b-4b28-a528-54cfa7fd380a-tmp-dir\") pod \"explicit-model-fc495e-kserve-79dff7dd85-85vkt\" (UID: \"f1ee57bf-1a5b-4b28-a528-54cfa7fd380a\") " pod="ai-tenant-e2e-explicit-fc495e/explicit-model-fc495e-kserve-79dff7dd85-85vkt"
Sep 10 15:45:06.390145 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:06.389684    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gctzr\" (UniqueName: \"kubernetes.io/projected/f1ee57bf-1a5b-4b28-a528-54cfa7fd380a-kube-api-access-gctzr\") pod \"explicit-model-fc495e-kserve-79dff7dd85-85vkt\" (UID: \"f1ee57bf-1a5b-4b28-a528-54cfa7fd380a\") " pod="ai-tenant-e2e-explicit-fc495e/explicit-model-fc495e-kserve-79dff7dd85-85vkt"
Sep 10 15:45:06.390145 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:06.389716    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1ee57bf-1a5b-4b28-a528-54cfa7fd380a-model-cache\") pod \"explicit-model-fc495e-kserve-79dff7dd85-85vkt\" (UID: \"f1ee57bf-1a5b-4b28-a528-54cfa7fd380a\") " pod="ai-tenant-e2e-explicit-fc495e/explicit-model-fc495e-kserve-79dff7dd85-85vkt"
Sep 10 15:45:06.390145 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:06.389738    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1ee57bf-1a5b-4b28-a528-54cfa7fd380a-tls-certs\") pod \"explicit-model-fc495e-kserve-79dff7dd85-85vkt\" (UID: \"f1ee57bf-1a5b-4b28-a528-54cfa7fd380a\") " pod="ai-tenant-e2e-explicit-fc495e/explicit-model-fc495e-kserve-79dff7dd85-85vkt"
Sep 10 15:45:06.390145 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:06.389766    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1ee57bf-1a5b-4b28-a528-54cfa7fd380a-home\") pod \"explicit-model-fc495e-kserve-79dff7dd85-85vkt\" (UID: \"f1ee57bf-1a5b-4b28-a528-54cfa7fd380a\") " pod="ai-tenant-e2e-explicit-fc495e/explicit-model-fc495e-kserve-79dff7dd85-85vkt"
Sep 10 15:45:06.390145 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:06.389783    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1ee57bf-1a5b-4b28-a528-54cfa7fd380a-dshm\") pod \"explicit-model-fc495e-kserve-79dff7dd85-85vkt\" (UID: \"f1ee57bf-1a5b-4b28-a528-54cfa7fd380a\") " pod="ai-tenant-e2e-explicit-fc495e/explicit-model-fc495e-kserve-79dff7dd85-85vkt"
Sep 10 15:45:06.390145 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:06.390112    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1ee57bf-1a5b-4b28-a528-54cfa7fd380a-tmp-dir\") pod \"explicit-model-fc495e-kserve-79dff7dd85-85vkt\" (UID: \"f1ee57bf-1a5b-4b28-a528-54cfa7fd380a\") " pod="ai-tenant-e2e-explicit-fc495e/explicit-model-fc495e-kserve-79dff7dd85-85vkt"
Sep 10 15:45:06.390145 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:06.390118    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1ee57bf-1a5b-4b28-a528-54cfa7fd380a-model-cache\") pod \"explicit-model-fc495e-kserve-79dff7dd85-85vkt\" (UID: \"f1ee57bf-1a5b-4b28-a528-54cfa7fd380a\") " pod="ai-tenant-e2e-explicit-fc495e/explicit-model-fc495e-kserve-79dff7dd85-85vkt"
Sep 10 15:45:06.390437 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:06.390206    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1ee57bf-1a5b-4b28-a528-54cfa7fd380a-home\") pod \"explicit-model-fc495e-kserve-79dff7dd85-85vkt\" (UID: \"f1ee57bf-1a5b-4b28-a528-54cfa7fd380a\") " pod="ai-tenant-e2e-explicit-fc495e/explicit-model-fc495e-kserve-79dff7dd85-85vkt"
Sep 10 15:45:06.392105 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:06.392084    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1ee57bf-1a5b-4b28-a528-54cfa7fd380a-dshm\") pod \"explicit-model-fc495e-kserve-79dff7dd85-85vkt\" (UID: \"f1ee57bf-1a5b-4b28-a528-54cfa7fd380a\") " pod="ai-tenant-e2e-explicit-fc495e/explicit-model-fc495e-kserve-79dff7dd85-85vkt"
Sep 10 15:45:06.392244 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:06.392227    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1ee57bf-1a5b-4b28-a528-54cfa7fd380a-tls-certs\") pod \"explicit-model-fc495e-kserve-79dff7dd85-85vkt\" (UID: \"f1ee57bf-1a5b-4b28-a528-54cfa7fd380a\") " pod="ai-tenant-e2e-explicit-fc495e/explicit-model-fc495e-kserve-79dff7dd85-85vkt"
Sep 10 15:45:06.401823 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:06.401797    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gctzr\" (UniqueName: \"kubernetes.io/projected/f1ee57bf-1a5b-4b28-a528-54cfa7fd380a-kube-api-access-gctzr\") pod \"explicit-model-fc495e-kserve-79dff7dd85-85vkt\" (UID: \"f1ee57bf-1a5b-4b28-a528-54cfa7fd380a\") " pod="ai-tenant-e2e-explicit-fc495e/explicit-model-fc495e-kserve-79dff7dd85-85vkt"
Sep 10 15:45:06.518747 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:06.518702    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-fc495e/explicit-model-fc495e-kserve-79dff7dd85-85vkt"
Sep 10 15:45:06.794663 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:06.794639    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-fc495e/explicit-model-fc495e-kserve-79dff7dd85-85vkt"]
Sep 10 15:45:06.796195 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:45:06.796169    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf1ee57bf_1a5b_4b28_a528_54cfa7fd380a.slice/crio-2ea840dc99d9194ea67ad43e47b292cb375feae83dc54758378945be779c45dd WatchSource:0}: Error finding container 2ea840dc99d9194ea67ad43e47b292cb375feae83dc54758378945be779c45dd: Status 404 returned error can't find the container with id 2ea840dc99d9194ea67ad43e47b292cb375feae83dc54758378945be779c45dd
Sep 10 15:45:07.188989 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:07.188955    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-fc495e/explicit-model-fc495e-kserve-79dff7dd85-85vkt" event={"ID":"f1ee57bf-1a5b-4b28-a528-54cfa7fd380a","Type":"ContainerStarted","Data":"c42b037e7c345532539f0d1e8a7a2c4f2d905a77f39004383aceee4aabe4ee46"}
Sep 10 15:45:07.189177 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:07.188996    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-fc495e/explicit-model-fc495e-kserve-79dff7dd85-85vkt" event={"ID":"f1ee57bf-1a5b-4b28-a528-54cfa7fd380a","Type":"ContainerStarted","Data":"2ea840dc99d9194ea67ad43e47b292cb375feae83dc54758378945be779c45dd"}
Sep 10 15:45:07.207594 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:07.207520    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-fc495e/explicit-model-fc495e-kserve-79dff7dd85-85vkt" podStartSLOduration=1.207501951 podStartE2EDuration="1.207501951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:45:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:45:07.206839415 +0000 UTC m=+1082.948804518" watchObservedRunningTime="2026-09-10 15:45:07.207501951 +0000 UTC m=+1082.949467055"
Sep 10 15:45:08.881894 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:08.881860    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-fc495e/explicit-model-fc495e-kserve-79dff7dd85-85vkt"]
Sep 10 15:45:16.519268 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:16.519220    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-fc495e/explicit-model-fc495e-kserve-79dff7dd85-85vkt"
Sep 10 15:45:21.707899 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:21.707857    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bs7hd"]
Sep 10 15:45:24.219184 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:24.219142    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-fc495e/explicit-model-fc495e-kserve-79dff7dd85-85vkt" podUID="f1ee57bf-1a5b-4b28-a528-54cfa7fd380a" containerName="main" containerID="cri-o://c42b037e7c345532539f0d1e8a7a2c4f2d905a77f39004383aceee4aabe4ee46" gracePeriod=45
Sep 10 15:45:24.470303 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:24.470249    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-fc495e/explicit-model-fc495e-kserve-79dff7dd85-85vkt"
Sep 10 15:45:24.575389 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:24.575357    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1ee57bf-1a5b-4b28-a528-54cfa7fd380a-dshm\") pod \"f1ee57bf-1a5b-4b28-a528-54cfa7fd380a\" (UID: \"f1ee57bf-1a5b-4b28-a528-54cfa7fd380a\") "
Sep 10 15:45:24.575549 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:24.575413    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1ee57bf-1a5b-4b28-a528-54cfa7fd380a-home\") pod \"f1ee57bf-1a5b-4b28-a528-54cfa7fd380a\" (UID: \"f1ee57bf-1a5b-4b28-a528-54cfa7fd380a\") "
Sep 10 15:45:24.575549 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:24.575432    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1ee57bf-1a5b-4b28-a528-54cfa7fd380a-tls-certs\") pod \"f1ee57bf-1a5b-4b28-a528-54cfa7fd380a\" (UID: \"f1ee57bf-1a5b-4b28-a528-54cfa7fd380a\") "
Sep 10 15:45:24.575549 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:24.575460    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1ee57bf-1a5b-4b28-a528-54cfa7fd380a-model-cache\") pod \"f1ee57bf-1a5b-4b28-a528-54cfa7fd380a\" (UID: \"f1ee57bf-1a5b-4b28-a528-54cfa7fd380a\") "
Sep 10 15:45:24.575549 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:24.575477    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gctzr\" (UniqueName: \"kubernetes.io/projected/f1ee57bf-1a5b-4b28-a528-54cfa7fd380a-kube-api-access-gctzr\") pod \"f1ee57bf-1a5b-4b28-a528-54cfa7fd380a\" (UID: \"f1ee57bf-1a5b-4b28-a528-54cfa7fd380a\") "
Sep 10 15:45:24.575549 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:24.575512    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1ee57bf-1a5b-4b28-a528-54cfa7fd380a-tmp-dir\") pod \"f1ee57bf-1a5b-4b28-a528-54cfa7fd380a\" (UID: \"f1ee57bf-1a5b-4b28-a528-54cfa7fd380a\") "
Sep 10 15:45:24.575856 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:24.575682    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1ee57bf-1a5b-4b28-a528-54cfa7fd380a-home" (OuterVolumeSpecName: "home") pod "f1ee57bf-1a5b-4b28-a528-54cfa7fd380a" (UID: "f1ee57bf-1a5b-4b28-a528-54cfa7fd380a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:45:24.575856 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:24.575740    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1ee57bf-1a5b-4b28-a528-54cfa7fd380a-model-cache" (OuterVolumeSpecName: "model-cache") pod "f1ee57bf-1a5b-4b28-a528-54cfa7fd380a" (UID: "f1ee57bf-1a5b-4b28-a528-54cfa7fd380a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:45:24.575973 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:24.575904    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1ee57bf-1a5b-4b28-a528-54cfa7fd380a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f1ee57bf-1a5b-4b28-a528-54cfa7fd380a" (UID: "f1ee57bf-1a5b-4b28-a528-54cfa7fd380a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:45:24.575973 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:24.575916    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1ee57bf-1a5b-4b28-a528-54cfa7fd380a-home\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:45:24.575973 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:24.575961    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1ee57bf-1a5b-4b28-a528-54cfa7fd380a-model-cache\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:45:24.577723 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:24.577705    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1ee57bf-1a5b-4b28-a528-54cfa7fd380a-dshm" (OuterVolumeSpecName: "dshm") pod "f1ee57bf-1a5b-4b28-a528-54cfa7fd380a" (UID: "f1ee57bf-1a5b-4b28-a528-54cfa7fd380a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:45:24.577723 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:24.577710    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1ee57bf-1a5b-4b28-a528-54cfa7fd380a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f1ee57bf-1a5b-4b28-a528-54cfa7fd380a" (UID: "f1ee57bf-1a5b-4b28-a528-54cfa7fd380a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:45:24.577830 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:24.577798    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1ee57bf-1a5b-4b28-a528-54cfa7fd380a-kube-api-access-gctzr" (OuterVolumeSpecName: "kube-api-access-gctzr") pod "f1ee57bf-1a5b-4b28-a528-54cfa7fd380a" (UID: "f1ee57bf-1a5b-4b28-a528-54cfa7fd380a"). InnerVolumeSpecName "kube-api-access-gctzr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:45:24.676679 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:24.676646    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1ee57bf-1a5b-4b28-a528-54cfa7fd380a-dshm\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:45:24.676679 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:24.676669    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1ee57bf-1a5b-4b28-a528-54cfa7fd380a-tls-certs\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:45:24.676679 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:24.676681    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gctzr\" (UniqueName: \"kubernetes.io/projected/f1ee57bf-1a5b-4b28-a528-54cfa7fd380a-kube-api-access-gctzr\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:45:24.676876 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:24.676690    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1ee57bf-1a5b-4b28-a528-54cfa7fd380a-tmp-dir\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:45:25.287922 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:25.287884    2559 generic.go:356] "Generic (PLEG): container finished" podID="f1ee57bf-1a5b-4b28-a528-54cfa7fd380a" containerID="c42b037e7c345532539f0d1e8a7a2c4f2d905a77f39004383aceee4aabe4ee46" exitCode=0
Sep 10 15:45:25.288346 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:25.287932    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-fc495e/explicit-model-fc495e-kserve-79dff7dd85-85vkt"
Sep 10 15:45:25.288346 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:25.287946    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-fc495e/explicit-model-fc495e-kserve-79dff7dd85-85vkt" event={"ID":"f1ee57bf-1a5b-4b28-a528-54cfa7fd380a","Type":"ContainerDied","Data":"c42b037e7c345532539f0d1e8a7a2c4f2d905a77f39004383aceee4aabe4ee46"}
Sep 10 15:45:25.288346 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:25.287986    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-fc495e/explicit-model-fc495e-kserve-79dff7dd85-85vkt" event={"ID":"f1ee57bf-1a5b-4b28-a528-54cfa7fd380a","Type":"ContainerDied","Data":"2ea840dc99d9194ea67ad43e47b292cb375feae83dc54758378945be779c45dd"}
Sep 10 15:45:25.288346 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:25.288009    2559 scope.go:117] "RemoveContainer" containerID="c42b037e7c345532539f0d1e8a7a2c4f2d905a77f39004383aceee4aabe4ee46"
Sep 10 15:45:25.296896 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:25.296871    2559 scope.go:117] "RemoveContainer" containerID="c42b037e7c345532539f0d1e8a7a2c4f2d905a77f39004383aceee4aabe4ee46"
Sep 10 15:45:25.297198 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:45:25.297172    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c42b037e7c345532539f0d1e8a7a2c4f2d905a77f39004383aceee4aabe4ee46\": container with ID starting with c42b037e7c345532539f0d1e8a7a2c4f2d905a77f39004383aceee4aabe4ee46 not found: ID does not exist" containerID="c42b037e7c345532539f0d1e8a7a2c4f2d905a77f39004383aceee4aabe4ee46"
Sep 10 15:45:25.297270 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:25.297206    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c42b037e7c345532539f0d1e8a7a2c4f2d905a77f39004383aceee4aabe4ee46"} err="failed to get container status \"c42b037e7c345532539f0d1e8a7a2c4f2d905a77f39004383aceee4aabe4ee46\": rpc error: code = NotFound desc = could not find container \"c42b037e7c345532539f0d1e8a7a2c4f2d905a77f39004383aceee4aabe4ee46\": container with ID starting with c42b037e7c345532539f0d1e8a7a2c4f2d905a77f39004383aceee4aabe4ee46 not found: ID does not exist"
Sep 10 15:45:25.313396 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:25.313373    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-fc495e/explicit-model-fc495e-kserve-79dff7dd85-85vkt"]
Sep 10 15:45:25.321015 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:25.320989    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-fc495e/explicit-model-fc495e-kserve-79dff7dd85-85vkt"]
Sep 10 15:45:26.785494 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:26.785453    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f1ee57bf-1a5b-4b28-a528-54cfa7fd380a" path="/var/lib/kubelet/pods/f1ee57bf-1a5b-4b28-a528-54cfa7fd380a/volumes"
Sep 10 15:45:27.167595 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:27.165203    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-fc495e-openshift-default-bcdb5fcb-bgtbx"]
Sep 10 15:45:27.167595 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:27.165533    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-fc495e-openshift-default-bcdb5fcb-bgtbx" podUID="c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7" containerName="istio-proxy" containerID="cri-o://e6d3af49ad385b52f371b15106fbb9aaffc96202a4a1abe92835107164d5ee88" gracePeriod=30
Sep 10 15:45:29.882210 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:29.882173    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-139155/test-model-139155-kserve-69bf99cdbf-4ng2n"]
Sep 10 15:45:32.330292 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:32.330252    2559 generic.go:356] "Generic (PLEG): container finished" podID="c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7" containerID="e6d3af49ad385b52f371b15106fbb9aaffc96202a4a1abe92835107164d5ee88" exitCode=0
Sep 10 15:45:32.330292 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:32.330314    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-fc495e-openshift-default-bcdb5fcb-bgtbx" event={"ID":"c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7","Type":"ContainerDied","Data":"e6d3af49ad385b52f371b15106fbb9aaffc96202a4a1abe92835107164d5ee88"}
Sep 10 15:45:32.330823 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:32.330366    2559 scope.go:117] "RemoveContainer" containerID="8e05245294c3a4890181a60845683c17125cd434811b8c2a7eb9a7cba79a0bb8"
Sep 10 15:45:32.426939 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:32.426910    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-fc495e-openshift-default-bcdb5fcb-bgtbx"
Sep 10 15:45:32.552246 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:32.552126    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7-istio-podinfo\") pod \"c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7\" (UID: \"c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7\") "
Sep 10 15:45:32.552246 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:32.552187    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rpp8v\" (UniqueName: \"kubernetes.io/projected/c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7-kube-api-access-rpp8v\") pod \"c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7\" (UID: \"c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7\") "
Sep 10 15:45:32.552246 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:32.552213    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7-credential-socket\") pod \"c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7\" (UID: \"c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7\") "
Sep 10 15:45:32.552246 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:32.552245    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7-istio-token\") pod \"c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7\" (UID: \"c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7\") "
Sep 10 15:45:32.552597 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:32.552276    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7-istio-data\") pod \"c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7\" (UID: \"c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7\") "
Sep 10 15:45:32.552597 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:32.552304    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7-workload-socket\") pod \"c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7\" (UID: \"c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7\") "
Sep 10 15:45:32.552597 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:32.552347    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7-istio-envoy\") pod \"c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7\" (UID: \"c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7\") "
Sep 10 15:45:32.552597 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:32.552399    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7-istiod-ca-cert\") pod \"c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7\" (UID: \"c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7\") "
Sep 10 15:45:32.552597 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:32.552449    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7-workload-certs\") pod \"c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7\" (UID: \"c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7\") "
Sep 10 15:45:32.552850 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:32.552700    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7" (UID: "c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:45:32.552850 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:32.552784    2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7-credential-socket\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:45:32.553186 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:32.553130    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7" (UID: "c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:45:32.553283 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:32.553181    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7" (UID: "c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:45:32.553411 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:32.553373    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7-istio-data" (OuterVolumeSpecName: "istio-data") pod "c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7" (UID: "c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:45:32.553477 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:32.553451    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7" (UID: "c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:45:32.554890 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:32.554851    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7" (UID: "c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 15:45:32.555012 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:32.554985    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7-istio-token" (OuterVolumeSpecName: "istio-token") pod "c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7" (UID: "c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:45:32.555092 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:32.555052    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7" (UID: "c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:45:32.555364 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:32.555347    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7-kube-api-access-rpp8v" (OuterVolumeSpecName: "kube-api-access-rpp8v") pod "c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7" (UID: "c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7"). InnerVolumeSpecName "kube-api-access-rpp8v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:45:32.653785 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:32.653748    2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7-istio-podinfo\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:45:32.653785 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:32.653777    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpp8v\" (UniqueName: \"kubernetes.io/projected/c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7-kube-api-access-rpp8v\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:45:32.653785 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:32.653788    2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7-istio-token\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:45:32.653785 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:32.653797    2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7-istio-data\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:45:32.654163 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:32.653806    2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7-workload-socket\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:45:32.654163 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:32.653814    2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7-istio-envoy\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:45:32.654163 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:32.653826    2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7-istiod-ca-cert\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:45:32.654163 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:32.653837    2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7-workload-certs\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:45:33.337649 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:33.337611    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-fc495e-openshift-default-bcdb5fcb-bgtbx"
Sep 10 15:45:33.337649 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:33.337622    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-fc495e-openshift-default-bcdb5fcb-bgtbx" event={"ID":"c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7","Type":"ContainerDied","Data":"d78f6057f6333b7fa1f239415533ecf491ff4d70e9f86da7b367dcb75f32a199"}
Sep 10 15:45:33.338166 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:33.337730    2559 scope.go:117] "RemoveContainer" containerID="e6d3af49ad385b52f371b15106fbb9aaffc96202a4a1abe92835107164d5ee88"
Sep 10 15:45:33.359433 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:33.359408    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-fc495e-openshift-default-bcdb5fcb-bgtbx"]
Sep 10 15:45:33.364246 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:33.364224    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-fc495e-openshift-default-bcdb5fcb-bgtbx"]
Sep 10 15:45:34.786821 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:34.786788    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7" path="/var/lib/kubelet/pods/c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7/volumes"
Sep 10 15:45:36.802397 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:36.802363    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bs7hd"]
Sep 10 15:45:44.902321 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:44.902255    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-139155/test-model-139155-kserve-69bf99cdbf-4ng2n" podUID="94e7b3b9-5dab-483c-8dc2-a7226098ee76" containerName="main" containerID="cri-o://dd1611e03b15a570f3dcfd8e2bde7f6493ad7d6c6a12eb7009349a84c2f28466" gracePeriod=45
Sep 10 15:45:45.167557 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:45.167533    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-139155/test-model-139155-kserve-69bf99cdbf-4ng2n"
Sep 10 15:45:45.268721 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:45.268691    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nltcr\" (UniqueName: \"kubernetes.io/projected/94e7b3b9-5dab-483c-8dc2-a7226098ee76-kube-api-access-nltcr\") pod \"94e7b3b9-5dab-483c-8dc2-a7226098ee76\" (UID: \"94e7b3b9-5dab-483c-8dc2-a7226098ee76\") "
Sep 10 15:45:45.268721 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:45.268725    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94e7b3b9-5dab-483c-8dc2-a7226098ee76-tls-certs\") pod \"94e7b3b9-5dab-483c-8dc2-a7226098ee76\" (UID: \"94e7b3b9-5dab-483c-8dc2-a7226098ee76\") "
Sep 10 15:45:45.268970 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:45.268750    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94e7b3b9-5dab-483c-8dc2-a7226098ee76-home\") pod \"94e7b3b9-5dab-483c-8dc2-a7226098ee76\" (UID: \"94e7b3b9-5dab-483c-8dc2-a7226098ee76\") "
Sep 10 15:45:45.268970 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:45.268772    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94e7b3b9-5dab-483c-8dc2-a7226098ee76-dshm\") pod \"94e7b3b9-5dab-483c-8dc2-a7226098ee76\" (UID: \"94e7b3b9-5dab-483c-8dc2-a7226098ee76\") "
Sep 10 15:45:45.268970 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:45.268825    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94e7b3b9-5dab-483c-8dc2-a7226098ee76-model-cache\") pod \"94e7b3b9-5dab-483c-8dc2-a7226098ee76\" (UID: \"94e7b3b9-5dab-483c-8dc2-a7226098ee76\") "
Sep 10 15:45:45.268970 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:45.268877    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94e7b3b9-5dab-483c-8dc2-a7226098ee76-tmp-dir\") pod \"94e7b3b9-5dab-483c-8dc2-a7226098ee76\" (UID: \"94e7b3b9-5dab-483c-8dc2-a7226098ee76\") "
Sep 10 15:45:45.269185 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:45.269048    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94e7b3b9-5dab-483c-8dc2-a7226098ee76-home" (OuterVolumeSpecName: "home") pod "94e7b3b9-5dab-483c-8dc2-a7226098ee76" (UID: "94e7b3b9-5dab-483c-8dc2-a7226098ee76"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:45:45.269185 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:45.269144    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94e7b3b9-5dab-483c-8dc2-a7226098ee76-model-cache" (OuterVolumeSpecName: "model-cache") pod "94e7b3b9-5dab-483c-8dc2-a7226098ee76" (UID: "94e7b3b9-5dab-483c-8dc2-a7226098ee76"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:45:45.269185 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:45.269153    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94e7b3b9-5dab-483c-8dc2-a7226098ee76-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "94e7b3b9-5dab-483c-8dc2-a7226098ee76" (UID: "94e7b3b9-5dab-483c-8dc2-a7226098ee76"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:45:45.269305 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:45.269215    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94e7b3b9-5dab-483c-8dc2-a7226098ee76-home\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:45:45.271189 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:45.271163    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94e7b3b9-5dab-483c-8dc2-a7226098ee76-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "94e7b3b9-5dab-483c-8dc2-a7226098ee76" (UID: "94e7b3b9-5dab-483c-8dc2-a7226098ee76"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:45:45.271189 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:45.271170    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94e7b3b9-5dab-483c-8dc2-a7226098ee76-dshm" (OuterVolumeSpecName: "dshm") pod "94e7b3b9-5dab-483c-8dc2-a7226098ee76" (UID: "94e7b3b9-5dab-483c-8dc2-a7226098ee76"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:45:45.271189 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:45.271180    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94e7b3b9-5dab-483c-8dc2-a7226098ee76-kube-api-access-nltcr" (OuterVolumeSpecName: "kube-api-access-nltcr") pod "94e7b3b9-5dab-483c-8dc2-a7226098ee76" (UID: "94e7b3b9-5dab-483c-8dc2-a7226098ee76"). InnerVolumeSpecName "kube-api-access-nltcr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:45:45.369808 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:45.369773    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nltcr\" (UniqueName: \"kubernetes.io/projected/94e7b3b9-5dab-483c-8dc2-a7226098ee76-kube-api-access-nltcr\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:45:45.369808 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:45.369805    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94e7b3b9-5dab-483c-8dc2-a7226098ee76-tls-certs\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:45:45.369808 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:45.369815    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94e7b3b9-5dab-483c-8dc2-a7226098ee76-dshm\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:45:45.370044 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:45.369824    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94e7b3b9-5dab-483c-8dc2-a7226098ee76-model-cache\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:45:45.370044 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:45.369831    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94e7b3b9-5dab-483c-8dc2-a7226098ee76-tmp-dir\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:45:45.405242 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:45.405152    2559 generic.go:356] "Generic (PLEG): container finished" podID="94e7b3b9-5dab-483c-8dc2-a7226098ee76" containerID="dd1611e03b15a570f3dcfd8e2bde7f6493ad7d6c6a12eb7009349a84c2f28466" exitCode=0
Sep 10 15:45:45.405242 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:45.405201    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-139155/test-model-139155-kserve-69bf99cdbf-4ng2n"
Sep 10 15:45:45.405242 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:45.405209    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-139155/test-model-139155-kserve-69bf99cdbf-4ng2n" event={"ID":"94e7b3b9-5dab-483c-8dc2-a7226098ee76","Type":"ContainerDied","Data":"dd1611e03b15a570f3dcfd8e2bde7f6493ad7d6c6a12eb7009349a84c2f28466"}
Sep 10 15:45:45.405242 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:45.405242    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-139155/test-model-139155-kserve-69bf99cdbf-4ng2n" event={"ID":"94e7b3b9-5dab-483c-8dc2-a7226098ee76","Type":"ContainerDied","Data":"f55f0c5df2a346777063bca7fd7cc7eb7599c49eb17d90ab1d76ce4cf5f22155"}
Sep 10 15:45:45.405568 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:45.405259    2559 scope.go:117] "RemoveContainer" containerID="dd1611e03b15a570f3dcfd8e2bde7f6493ad7d6c6a12eb7009349a84c2f28466"
Sep 10 15:45:45.414545 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:45.414522    2559 scope.go:117] "RemoveContainer" containerID="dd1611e03b15a570f3dcfd8e2bde7f6493ad7d6c6a12eb7009349a84c2f28466"
Sep 10 15:45:45.414835 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:45:45.414816    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd1611e03b15a570f3dcfd8e2bde7f6493ad7d6c6a12eb7009349a84c2f28466\": container with ID starting with dd1611e03b15a570f3dcfd8e2bde7f6493ad7d6c6a12eb7009349a84c2f28466 not found: ID does not exist" containerID="dd1611e03b15a570f3dcfd8e2bde7f6493ad7d6c6a12eb7009349a84c2f28466"
Sep 10 15:45:45.414912 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:45.414842    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd1611e03b15a570f3dcfd8e2bde7f6493ad7d6c6a12eb7009349a84c2f28466"} err="failed to get container status \"dd1611e03b15a570f3dcfd8e2bde7f6493ad7d6c6a12eb7009349a84c2f28466\": rpc error: code = NotFound desc = could not find container \"dd1611e03b15a570f3dcfd8e2bde7f6493ad7d6c6a12eb7009349a84c2f28466\": container with ID starting with dd1611e03b15a570f3dcfd8e2bde7f6493ad7d6c6a12eb7009349a84c2f28466 not found: ID does not exist"
Sep 10 15:45:45.428242 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:45.428214    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-139155/test-model-139155-kserve-69bf99cdbf-4ng2n"]
Sep 10 15:45:45.435275 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:45.435252    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-139155/test-model-139155-kserve-69bf99cdbf-4ng2n"]
Sep 10 15:45:46.785528 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:46.785494    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94e7b3b9-5dab-483c-8dc2-a7226098ee76" path="/var/lib/kubelet/pods/94e7b3b9-5dab-483c-8dc2-a7226098ee76/volumes"
Sep 10 15:45:47.934546 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:47.934506    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-f4a9663c-openshift-default-bf4d48b4d-m9lwx"]
Sep 10 15:45:47.935064 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:47.935044    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94e7b3b9-5dab-483c-8dc2-a7226098ee76" containerName="main"
Sep 10 15:45:47.935159 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:47.935068    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="94e7b3b9-5dab-483c-8dc2-a7226098ee76" containerName="main"
Sep 10 15:45:47.935159 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:47.935080    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1ee57bf-1a5b-4b28-a528-54cfa7fd380a" containerName="main"
Sep 10 15:45:47.935159 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:47.935088    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1ee57bf-1a5b-4b28-a528-54cfa7fd380a" containerName="main"
Sep 10 15:45:47.935159 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:47.935127    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7" containerName="istio-proxy"
Sep 10 15:45:47.935159 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:47.935136    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7" containerName="istio-proxy"
Sep 10 15:45:47.935159 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:47.935148    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7" containerName="istio-proxy"
Sep 10 15:45:47.935159 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:47.935157    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7" containerName="istio-proxy"
Sep 10 15:45:47.935495 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:47.935256    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7" containerName="istio-proxy"
Sep 10 15:45:47.935495 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:47.935270    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1ee57bf-1a5b-4b28-a528-54cfa7fd380a" containerName="main"
Sep 10 15:45:47.935495 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:47.935282    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="94e7b3b9-5dab-483c-8dc2-a7226098ee76" containerName="main"
Sep 10 15:45:47.943756 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:47.943723    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f4a9663c-openshift-default-bf4d48b4d-m9lwx"
Sep 10 15:45:47.946891 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:47.946865    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-f4a9663c-openshift-default-dockercfg-kfzq5\""
Sep 10 15:45:47.955193 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:47.955170    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f4a9663c-openshift-default-bf4d48b4d-m9lwx"]
Sep 10 15:45:47.993754 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:47.993722    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd88c3dc-642d-4c97-bcfc-80bed1479c0d-workload-certs\") pod \"e2e-mt-f4a9663c-openshift-default-bf4d48b4d-m9lwx\" (UID: \"cd88c3dc-642d-4c97-bcfc-80bed1479c0d\") " pod="openshift-ingress/e2e-mt-f4a9663c-openshift-default-bf4d48b4d-m9lwx"
Sep 10 15:45:47.993754 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:47.993761    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd88c3dc-642d-4c97-bcfc-80bed1479c0d-istio-envoy\") pod \"e2e-mt-f4a9663c-openshift-default-bf4d48b4d-m9lwx\" (UID: \"cd88c3dc-642d-4c97-bcfc-80bed1479c0d\") " pod="openshift-ingress/e2e-mt-f4a9663c-openshift-default-bf4d48b4d-m9lwx"
Sep 10 15:45:47.994019 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:47.993778    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd88c3dc-642d-4c97-bcfc-80bed1479c0d-istiod-ca-cert\") pod \"e2e-mt-f4a9663c-openshift-default-bf4d48b4d-m9lwx\" (UID: \"cd88c3dc-642d-4c97-bcfc-80bed1479c0d\") " pod="openshift-ingress/e2e-mt-f4a9663c-openshift-default-bf4d48b4d-m9lwx"
Sep 10 15:45:47.994019 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:47.993801    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd88c3dc-642d-4c97-bcfc-80bed1479c0d-credential-socket\") pod \"e2e-mt-f4a9663c-openshift-default-bf4d48b4d-m9lwx\" (UID: \"cd88c3dc-642d-4c97-bcfc-80bed1479c0d\") " pod="openshift-ingress/e2e-mt-f4a9663c-openshift-default-bf4d48b4d-m9lwx"
Sep 10 15:45:47.994019 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:47.993845    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd88c3dc-642d-4c97-bcfc-80bed1479c0d-workload-socket\") pod \"e2e-mt-f4a9663c-openshift-default-bf4d48b4d-m9lwx\" (UID: \"cd88c3dc-642d-4c97-bcfc-80bed1479c0d\") " pod="openshift-ingress/e2e-mt-f4a9663c-openshift-default-bf4d48b4d-m9lwx"
Sep 10 15:45:47.994019 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:47.993866    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd88c3dc-642d-4c97-bcfc-80bed1479c0d-istio-data\") pod \"e2e-mt-f4a9663c-openshift-default-bf4d48b4d-m9lwx\" (UID: \"cd88c3dc-642d-4c97-bcfc-80bed1479c0d\") " pod="openshift-ingress/e2e-mt-f4a9663c-openshift-default-bf4d48b4d-m9lwx"
Sep 10 15:45:47.994019 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:47.993922    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd88c3dc-642d-4c97-bcfc-80bed1479c0d-istio-podinfo\") pod \"e2e-mt-f4a9663c-openshift-default-bf4d48b4d-m9lwx\" (UID: \"cd88c3dc-642d-4c97-bcfc-80bed1479c0d\") " pod="openshift-ingress/e2e-mt-f4a9663c-openshift-default-bf4d48b4d-m9lwx"
Sep 10 15:45:47.994019 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:47.993946    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2s65\" (UniqueName: \"kubernetes.io/projected/cd88c3dc-642d-4c97-bcfc-80bed1479c0d-kube-api-access-r2s65\") pod \"e2e-mt-f4a9663c-openshift-default-bf4d48b4d-m9lwx\" (UID: \"cd88c3dc-642d-4c97-bcfc-80bed1479c0d\") " pod="openshift-ingress/e2e-mt-f4a9663c-openshift-default-bf4d48b4d-m9lwx"
Sep 10 15:45:47.994019 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:47.993990    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd88c3dc-642d-4c97-bcfc-80bed1479c0d-istio-token\") pod \"e2e-mt-f4a9663c-openshift-default-bf4d48b4d-m9lwx\" (UID: \"cd88c3dc-642d-4c97-bcfc-80bed1479c0d\") " pod="openshift-ingress/e2e-mt-f4a9663c-openshift-default-bf4d48b4d-m9lwx"
Sep 10 15:45:48.094418 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:48.094373    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd88c3dc-642d-4c97-bcfc-80bed1479c0d-workload-certs\") pod \"e2e-mt-f4a9663c-openshift-default-bf4d48b4d-m9lwx\" (UID: \"cd88c3dc-642d-4c97-bcfc-80bed1479c0d\") " pod="openshift-ingress/e2e-mt-f4a9663c-openshift-default-bf4d48b4d-m9lwx"
Sep 10 15:45:48.094678 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:48.094423    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd88c3dc-642d-4c97-bcfc-80bed1479c0d-istio-envoy\") pod \"e2e-mt-f4a9663c-openshift-default-bf4d48b4d-m9lwx\" (UID: \"cd88c3dc-642d-4c97-bcfc-80bed1479c0d\") " pod="openshift-ingress/e2e-mt-f4a9663c-openshift-default-bf4d48b4d-m9lwx"
Sep 10 15:45:48.094678 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:48.094452    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd88c3dc-642d-4c97-bcfc-80bed1479c0d-istiod-ca-cert\") pod \"e2e-mt-f4a9663c-openshift-default-bf4d48b4d-m9lwx\" (UID: \"cd88c3dc-642d-4c97-bcfc-80bed1479c0d\") " pod="openshift-ingress/e2e-mt-f4a9663c-openshift-default-bf4d48b4d-m9lwx"
Sep 10 15:45:48.094678 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:48.094492    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd88c3dc-642d-4c97-bcfc-80bed1479c0d-credential-socket\") pod \"e2e-mt-f4a9663c-openshift-default-bf4d48b4d-m9lwx\" (UID: \"cd88c3dc-642d-4c97-bcfc-80bed1479c0d\") " pod="openshift-ingress/e2e-mt-f4a9663c-openshift-default-bf4d48b4d-m9lwx"
Sep 10 15:45:48.094678 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:48.094528    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd88c3dc-642d-4c97-bcfc-80bed1479c0d-workload-socket\") pod \"e2e-mt-f4a9663c-openshift-default-bf4d48b4d-m9lwx\" (UID: \"cd88c3dc-642d-4c97-bcfc-80bed1479c0d\") " pod="openshift-ingress/e2e-mt-f4a9663c-openshift-default-bf4d48b4d-m9lwx"
Sep 10 15:45:48.094678 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:48.094561    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd88c3dc-642d-4c97-bcfc-80bed1479c0d-istio-data\") pod \"e2e-mt-f4a9663c-openshift-default-bf4d48b4d-m9lwx\" (UID: \"cd88c3dc-642d-4c97-bcfc-80bed1479c0d\") " pod="openshift-ingress/e2e-mt-f4a9663c-openshift-default-bf4d48b4d-m9lwx"
Sep 10 15:45:48.094678 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:48.094631    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd88c3dc-642d-4c97-bcfc-80bed1479c0d-istio-podinfo\") pod \"e2e-mt-f4a9663c-openshift-default-bf4d48b4d-m9lwx\" (UID: \"cd88c3dc-642d-4c97-bcfc-80bed1479c0d\") " pod="openshift-ingress/e2e-mt-f4a9663c-openshift-default-bf4d48b4d-m9lwx"
Sep 10 15:45:48.094678 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:48.094661    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2s65\" (UniqueName: \"kubernetes.io/projected/cd88c3dc-642d-4c97-bcfc-80bed1479c0d-kube-api-access-r2s65\") pod \"e2e-mt-f4a9663c-openshift-default-bf4d48b4d-m9lwx\" (UID: \"cd88c3dc-642d-4c97-bcfc-80bed1479c0d\") " pod="openshift-ingress/e2e-mt-f4a9663c-openshift-default-bf4d48b4d-m9lwx"
Sep 10 15:45:48.095030 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:48.094732    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd88c3dc-642d-4c97-bcfc-80bed1479c0d-istio-token\") pod \"e2e-mt-f4a9663c-openshift-default-bf4d48b4d-m9lwx\" (UID: \"cd88c3dc-642d-4c97-bcfc-80bed1479c0d\") " pod="openshift-ingress/e2e-mt-f4a9663c-openshift-default-bf4d48b4d-m9lwx"
Sep 10 15:45:48.095030 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:48.094914    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd88c3dc-642d-4c97-bcfc-80bed1479c0d-workload-certs\") pod \"e2e-mt-f4a9663c-openshift-default-bf4d48b4d-m9lwx\" (UID: \"cd88c3dc-642d-4c97-bcfc-80bed1479c0d\") " pod="openshift-ingress/e2e-mt-f4a9663c-openshift-default-bf4d48b4d-m9lwx"
Sep 10 15:45:48.095030 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:48.095000    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd88c3dc-642d-4c97-bcfc-80bed1479c0d-istio-data\") pod \"e2e-mt-f4a9663c-openshift-default-bf4d48b4d-m9lwx\" (UID: \"cd88c3dc-642d-4c97-bcfc-80bed1479c0d\") " pod="openshift-ingress/e2e-mt-f4a9663c-openshift-default-bf4d48b4d-m9lwx"
Sep 10 15:45:48.095162 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:48.095046    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd88c3dc-642d-4c97-bcfc-80bed1479c0d-workload-socket\") pod \"e2e-mt-f4a9663c-openshift-default-bf4d48b4d-m9lwx\" (UID: \"cd88c3dc-642d-4c97-bcfc-80bed1479c0d\") " pod="openshift-ingress/e2e-mt-f4a9663c-openshift-default-bf4d48b4d-m9lwx"
Sep 10 15:45:48.095300 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:48.095273    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd88c3dc-642d-4c97-bcfc-80bed1479c0d-istiod-ca-cert\") pod \"e2e-mt-f4a9663c-openshift-default-bf4d48b4d-m9lwx\" (UID: \"cd88c3dc-642d-4c97-bcfc-80bed1479c0d\") " pod="openshift-ingress/e2e-mt-f4a9663c-openshift-default-bf4d48b4d-m9lwx"
Sep 10 15:45:48.095404 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:48.095388    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd88c3dc-642d-4c97-bcfc-80bed1479c0d-credential-socket\") pod \"e2e-mt-f4a9663c-openshift-default-bf4d48b4d-m9lwx\" (UID: \"cd88c3dc-642d-4c97-bcfc-80bed1479c0d\") " pod="openshift-ingress/e2e-mt-f4a9663c-openshift-default-bf4d48b4d-m9lwx"
Sep 10 15:45:48.096883 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:48.096860    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd88c3dc-642d-4c97-bcfc-80bed1479c0d-istio-envoy\") pod \"e2e-mt-f4a9663c-openshift-default-bf4d48b4d-m9lwx\" (UID: \"cd88c3dc-642d-4c97-bcfc-80bed1479c0d\") " pod="openshift-ingress/e2e-mt-f4a9663c-openshift-default-bf4d48b4d-m9lwx"
Sep 10 15:45:48.097007 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:48.096989    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd88c3dc-642d-4c97-bcfc-80bed1479c0d-istio-podinfo\") pod \"e2e-mt-f4a9663c-openshift-default-bf4d48b4d-m9lwx\" (UID: \"cd88c3dc-642d-4c97-bcfc-80bed1479c0d\") " pod="openshift-ingress/e2e-mt-f4a9663c-openshift-default-bf4d48b4d-m9lwx"
Sep 10 15:45:48.105277 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:48.105246    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd88c3dc-642d-4c97-bcfc-80bed1479c0d-istio-token\") pod \"e2e-mt-f4a9663c-openshift-default-bf4d48b4d-m9lwx\" (UID: \"cd88c3dc-642d-4c97-bcfc-80bed1479c0d\") " pod="openshift-ingress/e2e-mt-f4a9663c-openshift-default-bf4d48b4d-m9lwx"
Sep 10 15:45:48.105389 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:48.105344    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2s65\" (UniqueName: \"kubernetes.io/projected/cd88c3dc-642d-4c97-bcfc-80bed1479c0d-kube-api-access-r2s65\") pod \"e2e-mt-f4a9663c-openshift-default-bf4d48b4d-m9lwx\" (UID: \"cd88c3dc-642d-4c97-bcfc-80bed1479c0d\") " pod="openshift-ingress/e2e-mt-f4a9663c-openshift-default-bf4d48b4d-m9lwx"
Sep 10 15:45:48.255735 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:48.255642    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f4a9663c-openshift-default-bf4d48b4d-m9lwx"
Sep 10 15:45:48.555841 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:48.555806    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f4a9663c-openshift-default-bf4d48b4d-m9lwx"]
Sep 10 15:45:48.556202 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:45:48.556178    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd88c3dc_642d_4c97_bcfc_80bed1479c0d.slice/crio-ee4107f806f5dc30a323f021872cb161c19a01e8a85743bfd443e78a83a1619b WatchSource:0}: Error finding container ee4107f806f5dc30a323f021872cb161c19a01e8a85743bfd443e78a83a1619b: Status 404 returned error can't find the container with id ee4107f806f5dc30a323f021872cb161c19a01e8a85743bfd443e78a83a1619b
Sep 10 15:45:48.558515 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:48.558485    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 15:45:48.558600 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:48.558550    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 15:45:48.558600 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:48.558595    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 15:45:49.428955 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:49.428912    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f4a9663c-openshift-default-bf4d48b4d-m9lwx" event={"ID":"cd88c3dc-642d-4c97-bcfc-80bed1479c0d","Type":"ContainerStarted","Data":"244b67a06f0ed54a2a6812f22f4f6f0c6c42a10162e80cf5774248da0bca9e4d"}
Sep 10 15:45:49.429355 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:49.428970    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f4a9663c-openshift-default-bf4d48b4d-m9lwx" event={"ID":"cd88c3dc-642d-4c97-bcfc-80bed1479c0d","Type":"ContainerStarted","Data":"ee4107f806f5dc30a323f021872cb161c19a01e8a85743bfd443e78a83a1619b"}
Sep 10 15:45:49.452835 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:49.452780    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-f4a9663c-openshift-default-bf4d48b4d-m9lwx" podStartSLOduration=2.452763249 podStartE2EDuration="2.452763249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:45:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:45:49.450642999 +0000 UTC m=+1125.192608102" watchObservedRunningTime="2026-09-10 15:45:49.452763249 +0000 UTC m=+1125.194728351"
Sep 10 15:45:50.256531 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:50.256487    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-f4a9663c-openshift-default-bf4d48b4d-m9lwx"
Sep 10 15:45:50.261564 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:50.261538    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-f4a9663c-openshift-default-bf4d48b4d-m9lwx"
Sep 10 15:45:50.435059 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:50.435021    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-f4a9663c-openshift-default-bf4d48b4d-m9lwx"
Sep 10 15:45:50.435849 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:50.435830    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-f4a9663c-openshift-default-bf4d48b4d-m9lwx"
Sep 10 15:45:50.896992 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:50.896956    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bs7hd"]
Sep 10 15:45:57.289860 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:57.289825    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-2a38a4/noten-model-2a38a4-kserve-84d6957c9f-5qd5w"]
Sep 10 15:45:57.290316 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:57.290280    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3aea63e-68a5-4e23-8e0e-e3bbbb23a1c7" containerName="istio-proxy"
Sep 10 15:45:57.293524 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:57.293508    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-2a38a4/noten-model-2a38a4-kserve-84d6957c9f-5qd5w"
Sep 10 15:45:57.297284 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:57.297255    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-2a38a4\"/\"noten-model-2a38a4-kserve-self-signed-certs\""
Sep 10 15:45:57.297441 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:57.297288    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-2a38a4\"/\"openshift-service-ca.crt\""
Sep 10 15:45:57.297441 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:57.297302    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-2a38a4\"/\"kube-root-ca.crt\""
Sep 10 15:45:57.297441 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:57.297307    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-2a38a4\"/\"default-dockercfg-2z5hq\""
Sep 10 15:45:57.303716 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:57.303694    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-2a38a4/noten-model-2a38a4-kserve-84d6957c9f-5qd5w"]
Sep 10 15:45:57.378435 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:57.378403    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/583770d9-f88f-47df-a939-1b49cd071f0f-dshm\") pod \"noten-model-2a38a4-kserve-84d6957c9f-5qd5w\" (UID: \"583770d9-f88f-47df-a939-1b49cd071f0f\") " pod="ai-tenant-e2e-noten-2a38a4/noten-model-2a38a4-kserve-84d6957c9f-5qd5w"
Sep 10 15:45:57.378435 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:57.378442    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7zfv\" (UniqueName: \"kubernetes.io/projected/583770d9-f88f-47df-a939-1b49cd071f0f-kube-api-access-p7zfv\") pod \"noten-model-2a38a4-kserve-84d6957c9f-5qd5w\" (UID: \"583770d9-f88f-47df-a939-1b49cd071f0f\") " pod="ai-tenant-e2e-noten-2a38a4/noten-model-2a38a4-kserve-84d6957c9f-5qd5w"
Sep 10 15:45:57.378685 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:57.378488    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/583770d9-f88f-47df-a939-1b49cd071f0f-tls-certs\") pod \"noten-model-2a38a4-kserve-84d6957c9f-5qd5w\" (UID: \"583770d9-f88f-47df-a939-1b49cd071f0f\") " pod="ai-tenant-e2e-noten-2a38a4/noten-model-2a38a4-kserve-84d6957c9f-5qd5w"
Sep 10 15:45:57.378685 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:57.378626    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/583770d9-f88f-47df-a939-1b49cd071f0f-home\") pod \"noten-model-2a38a4-kserve-84d6957c9f-5qd5w\" (UID: \"583770d9-f88f-47df-a939-1b49cd071f0f\") " pod="ai-tenant-e2e-noten-2a38a4/noten-model-2a38a4-kserve-84d6957c9f-5qd5w"
Sep 10 15:45:57.378772 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:57.378697    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/583770d9-f88f-47df-a939-1b49cd071f0f-model-cache\") pod \"noten-model-2a38a4-kserve-84d6957c9f-5qd5w\" (UID: \"583770d9-f88f-47df-a939-1b49cd071f0f\") " pod="ai-tenant-e2e-noten-2a38a4/noten-model-2a38a4-kserve-84d6957c9f-5qd5w"
Sep 10 15:45:57.378772 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:57.378733    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/583770d9-f88f-47df-a939-1b49cd071f0f-tmp-dir\") pod \"noten-model-2a38a4-kserve-84d6957c9f-5qd5w\" (UID: \"583770d9-f88f-47df-a939-1b49cd071f0f\") " pod="ai-tenant-e2e-noten-2a38a4/noten-model-2a38a4-kserve-84d6957c9f-5qd5w"
Sep 10 15:45:57.480130 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:57.480097    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/583770d9-f88f-47df-a939-1b49cd071f0f-model-cache\") pod \"noten-model-2a38a4-kserve-84d6957c9f-5qd5w\" (UID: \"583770d9-f88f-47df-a939-1b49cd071f0f\") " pod="ai-tenant-e2e-noten-2a38a4/noten-model-2a38a4-kserve-84d6957c9f-5qd5w"
Sep 10 15:45:57.480301 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:57.480139    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/583770d9-f88f-47df-a939-1b49cd071f0f-tmp-dir\") pod \"noten-model-2a38a4-kserve-84d6957c9f-5qd5w\" (UID: \"583770d9-f88f-47df-a939-1b49cd071f0f\") " pod="ai-tenant-e2e-noten-2a38a4/noten-model-2a38a4-kserve-84d6957c9f-5qd5w"
Sep 10 15:45:57.480301 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:57.480173    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/583770d9-f88f-47df-a939-1b49cd071f0f-dshm\") pod \"noten-model-2a38a4-kserve-84d6957c9f-5qd5w\" (UID: \"583770d9-f88f-47df-a939-1b49cd071f0f\") " pod="ai-tenant-e2e-noten-2a38a4/noten-model-2a38a4-kserve-84d6957c9f-5qd5w"
Sep 10 15:45:57.480301 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:57.480188    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7zfv\" (UniqueName: \"kubernetes.io/projected/583770d9-f88f-47df-a939-1b49cd071f0f-kube-api-access-p7zfv\") pod \"noten-model-2a38a4-kserve-84d6957c9f-5qd5w\" (UID: \"583770d9-f88f-47df-a939-1b49cd071f0f\") " pod="ai-tenant-e2e-noten-2a38a4/noten-model-2a38a4-kserve-84d6957c9f-5qd5w"
Sep 10 15:45:57.480301 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:57.480213    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/583770d9-f88f-47df-a939-1b49cd071f0f-tls-certs\") pod \"noten-model-2a38a4-kserve-84d6957c9f-5qd5w\" (UID: \"583770d9-f88f-47df-a939-1b49cd071f0f\") " pod="ai-tenant-e2e-noten-2a38a4/noten-model-2a38a4-kserve-84d6957c9f-5qd5w"
Sep 10 15:45:57.480494 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:57.480324    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/583770d9-f88f-47df-a939-1b49cd071f0f-home\") pod \"noten-model-2a38a4-kserve-84d6957c9f-5qd5w\" (UID: \"583770d9-f88f-47df-a939-1b49cd071f0f\") " pod="ai-tenant-e2e-noten-2a38a4/noten-model-2a38a4-kserve-84d6957c9f-5qd5w"
Sep 10 15:45:57.480631 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:57.480611    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/583770d9-f88f-47df-a939-1b49cd071f0f-model-cache\") pod \"noten-model-2a38a4-kserve-84d6957c9f-5qd5w\" (UID: \"583770d9-f88f-47df-a939-1b49cd071f0f\") " pod="ai-tenant-e2e-noten-2a38a4/noten-model-2a38a4-kserve-84d6957c9f-5qd5w"
Sep 10 15:45:57.480701 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:57.480637    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/583770d9-f88f-47df-a939-1b49cd071f0f-tmp-dir\") pod \"noten-model-2a38a4-kserve-84d6957c9f-5qd5w\" (UID: \"583770d9-f88f-47df-a939-1b49cd071f0f\") " pod="ai-tenant-e2e-noten-2a38a4/noten-model-2a38a4-kserve-84d6957c9f-5qd5w"
Sep 10 15:45:57.480701 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:57.480679    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/583770d9-f88f-47df-a939-1b49cd071f0f-home\") pod \"noten-model-2a38a4-kserve-84d6957c9f-5qd5w\" (UID: \"583770d9-f88f-47df-a939-1b49cd071f0f\") " pod="ai-tenant-e2e-noten-2a38a4/noten-model-2a38a4-kserve-84d6957c9f-5qd5w"
Sep 10 15:45:57.482465 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:57.482437    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/583770d9-f88f-47df-a939-1b49cd071f0f-dshm\") pod \"noten-model-2a38a4-kserve-84d6957c9f-5qd5w\" (UID: \"583770d9-f88f-47df-a939-1b49cd071f0f\") " pod="ai-tenant-e2e-noten-2a38a4/noten-model-2a38a4-kserve-84d6957c9f-5qd5w"
Sep 10 15:45:57.482748 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:57.482732    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/583770d9-f88f-47df-a939-1b49cd071f0f-tls-certs\") pod \"noten-model-2a38a4-kserve-84d6957c9f-5qd5w\" (UID: \"583770d9-f88f-47df-a939-1b49cd071f0f\") " pod="ai-tenant-e2e-noten-2a38a4/noten-model-2a38a4-kserve-84d6957c9f-5qd5w"
Sep 10 15:45:57.488813 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:57.488793    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7zfv\" (UniqueName: \"kubernetes.io/projected/583770d9-f88f-47df-a939-1b49cd071f0f-kube-api-access-p7zfv\") pod \"noten-model-2a38a4-kserve-84d6957c9f-5qd5w\" (UID: \"583770d9-f88f-47df-a939-1b49cd071f0f\") " pod="ai-tenant-e2e-noten-2a38a4/noten-model-2a38a4-kserve-84d6957c9f-5qd5w"
Sep 10 15:45:57.603430 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:57.603355    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-2a38a4/noten-model-2a38a4-kserve-84d6957c9f-5qd5w"
Sep 10 15:45:57.876204 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:57.876172    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-2a38a4/noten-model-2a38a4-kserve-84d6957c9f-5qd5w"]
Sep 10 15:45:57.877390 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:45:57.877365    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod583770d9_f88f_47df_a939_1b49cd071f0f.slice/crio-227b447d78ace263ddc1260cad63c15c29419006b547f27c50f9c9e87470e4a9 WatchSource:0}: Error finding container 227b447d78ace263ddc1260cad63c15c29419006b547f27c50f9c9e87470e4a9: Status 404 returned error can't find the container with id 227b447d78ace263ddc1260cad63c15c29419006b547f27c50f9c9e87470e4a9
Sep 10 15:45:58.484600 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:58.484535    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-2a38a4/noten-model-2a38a4-kserve-84d6957c9f-5qd5w" event={"ID":"583770d9-f88f-47df-a939-1b49cd071f0f","Type":"ContainerStarted","Data":"f3b01db79ed65c57ce7a3f81d34306948fd990aba1ba590aa5148202975365a3"}
Sep 10 15:45:58.484600 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:58.484600    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-2a38a4/noten-model-2a38a4-kserve-84d6957c9f-5qd5w" event={"ID":"583770d9-f88f-47df-a939-1b49cd071f0f","Type":"ContainerStarted","Data":"227b447d78ace263ddc1260cad63c15c29419006b547f27c50f9c9e87470e4a9"}
Sep 10 15:45:58.503726 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:45:58.503657    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-2a38a4/noten-model-2a38a4-kserve-84d6957c9f-5qd5w" podStartSLOduration=1.503641644 podStartE2EDuration="1.503641644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:45:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:45:58.501393257 +0000 UTC m=+1134.243358363" watchObservedRunningTime="2026-09-10 15:45:58.503641644 +0000 UTC m=+1134.245606806"
Sep 10 15:46:04.416741 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:04.416697    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-2a38a4/noten-model-2a38a4-kserve-84d6957c9f-5qd5w"]
Sep 10 15:46:06.477793 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:06.477743    2559 patch_prober.go:28] interesting pod/e2e-mt-f4a9663c-openshift-default-bf4d48b4d-m9lwx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 10 15:46:06.478216 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:06.477827    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-f4a9663c-openshift-default-bf4d48b4d-m9lwx" podUID="cd88c3dc-642d-4c97-bcfc-80bed1479c0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 10 15:46:07.603755 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:07.603720    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-2a38a4/noten-model-2a38a4-kserve-84d6957c9f-5qd5w"
Sep 10 15:46:07.903327 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:07.903236    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bs7hd"]
Sep 10 15:46:08.540858 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:08.540818    2559 generic.go:356] "Generic (PLEG): container finished" podID="cd88c3dc-642d-4c97-bcfc-80bed1479c0d" containerID="244b67a06f0ed54a2a6812f22f4f6f0c6c42a10162e80cf5774248da0bca9e4d" exitCode=0
Sep 10 15:46:08.541094 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:08.540902    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f4a9663c-openshift-default-bf4d48b4d-m9lwx" event={"ID":"cd88c3dc-642d-4c97-bcfc-80bed1479c0d","Type":"ContainerDied","Data":"244b67a06f0ed54a2a6812f22f4f6f0c6c42a10162e80cf5774248da0bca9e4d"}
Sep 10 15:46:08.541344 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:08.541301    2559 scope.go:117] "RemoveContainer" containerID="244b67a06f0ed54a2a6812f22f4f6f0c6c42a10162e80cf5774248da0bca9e4d"
Sep 10 15:46:08.542438 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:08.542406    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 15:46:08.542521 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:08.542503    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 15:46:08.542564 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:08.542549    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 15:46:09.555499 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:09.555455    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f4a9663c-openshift-default-bf4d48b4d-m9lwx" event={"ID":"cd88c3dc-642d-4c97-bcfc-80bed1479c0d","Type":"ContainerStarted","Data":"0f1a05543e9d40918c76b48fe84d42db9e9cb09e3139740fa6d64aae6f94951a"}
Sep 10 15:46:10.256520 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:10.256482    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-f4a9663c-openshift-default-bf4d48b4d-m9lwx"
Sep 10 15:46:10.256910 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:10.256887    2559 patch_prober.go:28] interesting pod/e2e-mt-f4a9663c-openshift-default-bf4d48b4d-m9lwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Sep 10 15:46:10.257076 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:10.257052    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f4a9663c-openshift-default-bf4d48b4d-m9lwx" podUID="cd88c3dc-642d-4c97-bcfc-80bed1479c0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Sep 10 15:46:11.256113 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:11.256075    2559 patch_prober.go:28] interesting pod/e2e-mt-f4a9663c-openshift-default-bf4d48b4d-m9lwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Sep 10 15:46:11.256732 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:11.256696    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f4a9663c-openshift-default-bf4d48b4d-m9lwx" podUID="cd88c3dc-642d-4c97-bcfc-80bed1479c0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Sep 10 15:46:12.256360 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:12.256318    2559 patch_prober.go:28] interesting pod/e2e-mt-f4a9663c-openshift-default-bf4d48b4d-m9lwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Sep 10 15:46:12.256895 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:12.256383    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f4a9663c-openshift-default-bf4d48b4d-m9lwx" podUID="cd88c3dc-642d-4c97-bcfc-80bed1479c0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Sep 10 15:46:13.256503 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:13.256467    2559 patch_prober.go:28] interesting pod/e2e-mt-f4a9663c-openshift-default-bf4d48b4d-m9lwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Sep 10 15:46:13.257123 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:13.257090    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f4a9663c-openshift-default-bf4d48b4d-m9lwx" podUID="cd88c3dc-642d-4c97-bcfc-80bed1479c0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Sep 10 15:46:14.256491 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:14.256455    2559 patch_prober.go:28] interesting pod/e2e-mt-f4a9663c-openshift-default-bf4d48b4d-m9lwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Sep 10 15:46:14.256796 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:14.256764    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f4a9663c-openshift-default-bf4d48b4d-m9lwx" podUID="cd88c3dc-642d-4c97-bcfc-80bed1479c0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Sep 10 15:46:15.256715 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:15.256673    2559 patch_prober.go:28] interesting pod/e2e-mt-f4a9663c-openshift-default-bf4d48b4d-m9lwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Sep 10 15:46:15.256939 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:15.256738    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f4a9663c-openshift-default-bf4d48b4d-m9lwx" podUID="cd88c3dc-642d-4c97-bcfc-80bed1479c0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Sep 10 15:46:16.259626 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:16.259567    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-f4a9663c-openshift-default-bf4d48b4d-m9lwx"
Sep 10 15:46:16.260063 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:16.259868    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-f4a9663c-openshift-default-bf4d48b4d-m9lwx"
Sep 10 15:46:16.260669 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:16.260645    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-f4a9663c-openshift-default-bf4d48b4d-m9lwx"
Sep 10 15:46:19.436757 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:19.436702    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-2a38a4/noten-model-2a38a4-kserve-84d6957c9f-5qd5w" podUID="583770d9-f88f-47df-a939-1b49cd071f0f" containerName="main" containerID="cri-o://f3b01db79ed65c57ce7a3f81d34306948fd990aba1ba590aa5148202975365a3" gracePeriod=45
Sep 10 15:46:19.618001 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:19.617969    2559 generic.go:356] "Generic (PLEG): container finished" podID="583770d9-f88f-47df-a939-1b49cd071f0f" containerID="f3b01db79ed65c57ce7a3f81d34306948fd990aba1ba590aa5148202975365a3" exitCode=0
Sep 10 15:46:19.618174 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:19.618049    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-2a38a4/noten-model-2a38a4-kserve-84d6957c9f-5qd5w" event={"ID":"583770d9-f88f-47df-a939-1b49cd071f0f","Type":"ContainerDied","Data":"f3b01db79ed65c57ce7a3f81d34306948fd990aba1ba590aa5148202975365a3"}
Sep 10 15:46:19.681704 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:19.681680    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-2a38a4/noten-model-2a38a4-kserve-84d6957c9f-5qd5w"
Sep 10 15:46:19.787849 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:19.787762    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/583770d9-f88f-47df-a939-1b49cd071f0f-tls-certs\") pod \"583770d9-f88f-47df-a939-1b49cd071f0f\" (UID: \"583770d9-f88f-47df-a939-1b49cd071f0f\") "
Sep 10 15:46:19.787849 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:19.787813    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/583770d9-f88f-47df-a939-1b49cd071f0f-tmp-dir\") pod \"583770d9-f88f-47df-a939-1b49cd071f0f\" (UID: \"583770d9-f88f-47df-a939-1b49cd071f0f\") "
Sep 10 15:46:19.788047 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:19.787868    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/583770d9-f88f-47df-a939-1b49cd071f0f-model-cache\") pod \"583770d9-f88f-47df-a939-1b49cd071f0f\" (UID: \"583770d9-f88f-47df-a939-1b49cd071f0f\") "
Sep 10 15:46:19.788047 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:19.787895    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/583770d9-f88f-47df-a939-1b49cd071f0f-dshm\") pod \"583770d9-f88f-47df-a939-1b49cd071f0f\" (UID: \"583770d9-f88f-47df-a939-1b49cd071f0f\") "
Sep 10 15:46:19.788047 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:19.787923    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p7zfv\" (UniqueName: \"kubernetes.io/projected/583770d9-f88f-47df-a939-1b49cd071f0f-kube-api-access-p7zfv\") pod \"583770d9-f88f-47df-a939-1b49cd071f0f\" (UID: \"583770d9-f88f-47df-a939-1b49cd071f0f\") "
Sep 10 15:46:19.788047 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:19.787976    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/583770d9-f88f-47df-a939-1b49cd071f0f-home\") pod \"583770d9-f88f-47df-a939-1b49cd071f0f\" (UID: \"583770d9-f88f-47df-a939-1b49cd071f0f\") "
Sep 10 15:46:19.788245 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:19.788164    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/583770d9-f88f-47df-a939-1b49cd071f0f-model-cache" (OuterVolumeSpecName: "model-cache") pod "583770d9-f88f-47df-a939-1b49cd071f0f" (UID: "583770d9-f88f-47df-a939-1b49cd071f0f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:46:19.788245 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:19.788177    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/583770d9-f88f-47df-a939-1b49cd071f0f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "583770d9-f88f-47df-a939-1b49cd071f0f" (UID: "583770d9-f88f-47df-a939-1b49cd071f0f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:46:19.788354 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:19.788309    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/583770d9-f88f-47df-a939-1b49cd071f0f-home" (OuterVolumeSpecName: "home") pod "583770d9-f88f-47df-a939-1b49cd071f0f" (UID: "583770d9-f88f-47df-a939-1b49cd071f0f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:46:19.788469 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:19.788451    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/583770d9-f88f-47df-a939-1b49cd071f0f-tmp-dir\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:46:19.788531 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:19.788473    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/583770d9-f88f-47df-a939-1b49cd071f0f-model-cache\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:46:19.788531 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:19.788488    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/583770d9-f88f-47df-a939-1b49cd071f0f-home\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:46:19.790238 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:19.790214    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/583770d9-f88f-47df-a939-1b49cd071f0f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "583770d9-f88f-47df-a939-1b49cd071f0f" (UID: "583770d9-f88f-47df-a939-1b49cd071f0f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:46:19.790343 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:19.790243    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/583770d9-f88f-47df-a939-1b49cd071f0f-dshm" (OuterVolumeSpecName: "dshm") pod "583770d9-f88f-47df-a939-1b49cd071f0f" (UID: "583770d9-f88f-47df-a939-1b49cd071f0f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:46:19.790343 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:19.790308    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/583770d9-f88f-47df-a939-1b49cd071f0f-kube-api-access-p7zfv" (OuterVolumeSpecName: "kube-api-access-p7zfv") pod "583770d9-f88f-47df-a939-1b49cd071f0f" (UID: "583770d9-f88f-47df-a939-1b49cd071f0f"). InnerVolumeSpecName "kube-api-access-p7zfv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:46:19.889328 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:19.889293    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/583770d9-f88f-47df-a939-1b49cd071f0f-tls-certs\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:46:19.889328 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:19.889322    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/583770d9-f88f-47df-a939-1b49cd071f0f-dshm\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:46:19.889328 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:19.889332    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7zfv\" (UniqueName: \"kubernetes.io/projected/583770d9-f88f-47df-a939-1b49cd071f0f-kube-api-access-p7zfv\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:46:20.624987 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:20.624952    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-2a38a4/noten-model-2a38a4-kserve-84d6957c9f-5qd5w"
Sep 10 15:46:20.624987 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:20.624978    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-2a38a4/noten-model-2a38a4-kserve-84d6957c9f-5qd5w" event={"ID":"583770d9-f88f-47df-a939-1b49cd071f0f","Type":"ContainerDied","Data":"227b447d78ace263ddc1260cad63c15c29419006b547f27c50f9c9e87470e4a9"}
Sep 10 15:46:20.625505 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:20.625022    2559 scope.go:117] "RemoveContainer" containerID="f3b01db79ed65c57ce7a3f81d34306948fd990aba1ba590aa5148202975365a3"
Sep 10 15:46:20.648186 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:20.648158    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-2a38a4/noten-model-2a38a4-kserve-84d6957c9f-5qd5w"]
Sep 10 15:46:20.654814 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:20.654783    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-2a38a4/noten-model-2a38a4-kserve-84d6957c9f-5qd5w"]
Sep 10 15:46:20.784866 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:20.784831    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="583770d9-f88f-47df-a939-1b49cd071f0f" path="/var/lib/kubelet/pods/583770d9-f88f-47df-a939-1b49cd071f0f/volumes"
Sep 10 15:46:21.502114 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:21.502077    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bs7hd"]
Sep 10 15:46:27.108941 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:27.108854    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bs7hd"]
Sep 10 15:46:29.734331 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:29.734296    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-90ca97-775464f5cc-spv5h"]
Sep 10 15:46:29.735122 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:29.735088    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-90ca97-775464f5cc-spv5h" podUID="a27d3806-fc72-4adf-bd20-fbe020d777f4" containerName="maas-api" containerID="cri-o://15fc9b762ab1c8221e3e8b619e08d6c85c7fc9e988f3d4c0817f84bf1487921c" gracePeriod=30
Sep 10 15:46:29.767415 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:29.767388    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-90ca97-29817585-6mdbm"]
Sep 10 15:46:29.771162 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:29.771140    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-90ca97-29817585-6mdbm"]
Sep 10 15:46:29.988971 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:29.988912    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-90ca97-775464f5cc-spv5h"
Sep 10 15:46:30.081847 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:30.081810    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a27d3806-fc72-4adf-bd20-fbe020d777f4-metrics-certs\") pod \"a27d3806-fc72-4adf-bd20-fbe020d777f4\" (UID: \"a27d3806-fc72-4adf-bd20-fbe020d777f4\") "
Sep 10 15:46:30.082025 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:30.081871    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bhhbp\" (UniqueName: \"kubernetes.io/projected/a27d3806-fc72-4adf-bd20-fbe020d777f4-kube-api-access-bhhbp\") pod \"a27d3806-fc72-4adf-bd20-fbe020d777f4\" (UID: \"a27d3806-fc72-4adf-bd20-fbe020d777f4\") "
Sep 10 15:46:30.082025 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:30.081943    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a27d3806-fc72-4adf-bd20-fbe020d777f4-openshift-service-ca\") pod \"a27d3806-fc72-4adf-bd20-fbe020d777f4\" (UID: \"a27d3806-fc72-4adf-bd20-fbe020d777f4\") "
Sep 10 15:46:30.082025 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:30.081987    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a27d3806-fc72-4adf-bd20-fbe020d777f4-maas-api-tls\") pod \"a27d3806-fc72-4adf-bd20-fbe020d777f4\" (UID: \"a27d3806-fc72-4adf-bd20-fbe020d777f4\") "
Sep 10 15:46:30.082385 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:30.082352    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a27d3806-fc72-4adf-bd20-fbe020d777f4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a27d3806-fc72-4adf-bd20-fbe020d777f4" (UID: "a27d3806-fc72-4adf-bd20-fbe020d777f4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:46:30.084018 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:30.083989    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a27d3806-fc72-4adf-bd20-fbe020d777f4-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a27d3806-fc72-4adf-bd20-fbe020d777f4" (UID: "a27d3806-fc72-4adf-bd20-fbe020d777f4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:46:30.084111 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:30.084070    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a27d3806-fc72-4adf-bd20-fbe020d777f4-kube-api-access-bhhbp" (OuterVolumeSpecName: "kube-api-access-bhhbp") pod "a27d3806-fc72-4adf-bd20-fbe020d777f4" (UID: "a27d3806-fc72-4adf-bd20-fbe020d777f4"). InnerVolumeSpecName "kube-api-access-bhhbp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:46:30.084111 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:30.084077    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a27d3806-fc72-4adf-bd20-fbe020d777f4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a27d3806-fc72-4adf-bd20-fbe020d777f4" (UID: "a27d3806-fc72-4adf-bd20-fbe020d777f4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:46:30.183694 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:30.183655    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a27d3806-fc72-4adf-bd20-fbe020d777f4-maas-api-tls\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:46:30.183694 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:30.183686    2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a27d3806-fc72-4adf-bd20-fbe020d777f4-metrics-certs\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:46:30.183694 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:30.183696    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhhbp\" (UniqueName: \"kubernetes.io/projected/a27d3806-fc72-4adf-bd20-fbe020d777f4-kube-api-access-bhhbp\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:46:30.183694 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:30.183706    2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a27d3806-fc72-4adf-bd20-fbe020d777f4-openshift-service-ca\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:46:30.681849 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:30.681812    2559 generic.go:356] "Generic (PLEG): container finished" podID="a27d3806-fc72-4adf-bd20-fbe020d777f4" containerID="15fc9b762ab1c8221e3e8b619e08d6c85c7fc9e988f3d4c0817f84bf1487921c" exitCode=0
Sep 10 15:46:30.682026 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:30.681860    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-90ca97-775464f5cc-spv5h"
Sep 10 15:46:30.682026 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:30.681903    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-90ca97-775464f5cc-spv5h" event={"ID":"a27d3806-fc72-4adf-bd20-fbe020d777f4","Type":"ContainerDied","Data":"15fc9b762ab1c8221e3e8b619e08d6c85c7fc9e988f3d4c0817f84bf1487921c"}
Sep 10 15:46:30.682026 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:30.681964    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-90ca97-775464f5cc-spv5h" event={"ID":"a27d3806-fc72-4adf-bd20-fbe020d777f4","Type":"ContainerDied","Data":"912829acbb388d526a507a9d5643b9c86e3a72a9c5aee1dff92bf56c7f85c3fc"}
Sep 10 15:46:30.682026 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:30.681987    2559 scope.go:117] "RemoveContainer" containerID="15fc9b762ab1c8221e3e8b619e08d6c85c7fc9e988f3d4c0817f84bf1487921c"
Sep 10 15:46:30.691557 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:30.691540    2559 scope.go:117] "RemoveContainer" containerID="15fc9b762ab1c8221e3e8b619e08d6c85c7fc9e988f3d4c0817f84bf1487921c"
Sep 10 15:46:30.691826 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:46:30.691809    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15fc9b762ab1c8221e3e8b619e08d6c85c7fc9e988f3d4c0817f84bf1487921c\": container with ID starting with 15fc9b762ab1c8221e3e8b619e08d6c85c7fc9e988f3d4c0817f84bf1487921c not found: ID does not exist" containerID="15fc9b762ab1c8221e3e8b619e08d6c85c7fc9e988f3d4c0817f84bf1487921c"
Sep 10 15:46:30.691874 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:30.691834    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15fc9b762ab1c8221e3e8b619e08d6c85c7fc9e988f3d4c0817f84bf1487921c"} err="failed to get container status \"15fc9b762ab1c8221e3e8b619e08d6c85c7fc9e988f3d4c0817f84bf1487921c\": rpc error: code = NotFound desc = could not find container \"15fc9b762ab1c8221e3e8b619e08d6c85c7fc9e988f3d4c0817f84bf1487921c\": container with ID starting with 15fc9b762ab1c8221e3e8b619e08d6c85c7fc9e988f3d4c0817f84bf1487921c not found: ID does not exist"
Sep 10 15:46:30.705512 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:30.705489    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-90ca97-775464f5cc-spv5h"]
Sep 10 15:46:30.709731 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:30.709711    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-90ca97-775464f5cc-spv5h"]
Sep 10 15:46:30.784832 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:30.784804    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a27d3806-fc72-4adf-bd20-fbe020d777f4" path="/var/lib/kubelet/pods/a27d3806-fc72-4adf-bd20-fbe020d777f4/volumes"
Sep 10 15:46:30.785164 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:30.785086    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9f1fcbb-c17a-44eb-9b2c-986fc2869601" path="/var/lib/kubelet/pods/e9f1fcbb-c17a-44eb-9b2c-986fc2869601/volumes"
Sep 10 15:46:31.226203 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:31.226169    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f4a9663c-openshift-default-bf4d48b4d-m9lwx"]
Sep 10 15:46:31.226456 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:31.226424    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-f4a9663c-openshift-default-bf4d48b4d-m9lwx" podUID="cd88c3dc-642d-4c97-bcfc-80bed1479c0d" containerName="istio-proxy" containerID="cri-o://0f1a05543e9d40918c76b48fe84d42db9e9cb09e3139740fa6d64aae6f94951a" gracePeriod=30
Sep 10 15:46:31.261959 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:31.261916    2559 patch_prober.go:28] interesting pod/e2e-mt-f4a9663c-openshift-default-bf4d48b4d-m9lwx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 10 15:46:31.262109 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:31.261980    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-f4a9663c-openshift-default-bf4d48b4d-m9lwx" podUID="cd88c3dc-642d-4c97-bcfc-80bed1479c0d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 10 15:46:35.716748 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:35.716717    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bs7hd"]
Sep 10 15:46:36.481678 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:36.481652    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f4a9663c-openshift-default-bf4d48b4d-m9lwx"
Sep 10 15:46:36.644033 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:36.643939    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd88c3dc-642d-4c97-bcfc-80bed1479c0d-workload-socket\") pod \"cd88c3dc-642d-4c97-bcfc-80bed1479c0d\" (UID: \"cd88c3dc-642d-4c97-bcfc-80bed1479c0d\") "
Sep 10 15:46:36.644033 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:36.643974    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r2s65\" (UniqueName: \"kubernetes.io/projected/cd88c3dc-642d-4c97-bcfc-80bed1479c0d-kube-api-access-r2s65\") pod \"cd88c3dc-642d-4c97-bcfc-80bed1479c0d\" (UID: \"cd88c3dc-642d-4c97-bcfc-80bed1479c0d\") "
Sep 10 15:46:36.644033 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:36.644003    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd88c3dc-642d-4c97-bcfc-80bed1479c0d-istio-envoy\") pod \"cd88c3dc-642d-4c97-bcfc-80bed1479c0d\" (UID: \"cd88c3dc-642d-4c97-bcfc-80bed1479c0d\") "
Sep 10 15:46:36.644033 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:36.644022    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd88c3dc-642d-4c97-bcfc-80bed1479c0d-istiod-ca-cert\") pod \"cd88c3dc-642d-4c97-bcfc-80bed1479c0d\" (UID: \"cd88c3dc-642d-4c97-bcfc-80bed1479c0d\") "
Sep 10 15:46:36.644441 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:36.644049    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd88c3dc-642d-4c97-bcfc-80bed1479c0d-istio-token\") pod \"cd88c3dc-642d-4c97-bcfc-80bed1479c0d\" (UID: \"cd88c3dc-642d-4c97-bcfc-80bed1479c0d\") "
Sep 10 15:46:36.644441 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:36.644078    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd88c3dc-642d-4c97-bcfc-80bed1479c0d-istio-data\") pod \"cd88c3dc-642d-4c97-bcfc-80bed1479c0d\" (UID: \"cd88c3dc-642d-4c97-bcfc-80bed1479c0d\") "
Sep 10 15:46:36.644441 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:36.644122    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd88c3dc-642d-4c97-bcfc-80bed1479c0d-credential-socket\") pod \"cd88c3dc-642d-4c97-bcfc-80bed1479c0d\" (UID: \"cd88c3dc-642d-4c97-bcfc-80bed1479c0d\") "
Sep 10 15:46:36.644441 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:36.644158    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd88c3dc-642d-4c97-bcfc-80bed1479c0d-istio-podinfo\") pod \"cd88c3dc-642d-4c97-bcfc-80bed1479c0d\" (UID: \"cd88c3dc-642d-4c97-bcfc-80bed1479c0d\") "
Sep 10 15:46:36.644441 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:36.644198    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd88c3dc-642d-4c97-bcfc-80bed1479c0d-workload-certs\") pod \"cd88c3dc-642d-4c97-bcfc-80bed1479c0d\" (UID: \"cd88c3dc-642d-4c97-bcfc-80bed1479c0d\") "
Sep 10 15:46:36.644441 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:36.644238    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd88c3dc-642d-4c97-bcfc-80bed1479c0d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "cd88c3dc-642d-4c97-bcfc-80bed1479c0d" (UID: "cd88c3dc-642d-4c97-bcfc-80bed1479c0d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:46:36.644441 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:36.644431    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd88c3dc-642d-4c97-bcfc-80bed1479c0d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "cd88c3dc-642d-4c97-bcfc-80bed1479c0d" (UID: "cd88c3dc-642d-4c97-bcfc-80bed1479c0d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:46:36.644875 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:36.644454    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd88c3dc-642d-4c97-bcfc-80bed1479c0d-istio-data" (OuterVolumeSpecName: "istio-data") pod "cd88c3dc-642d-4c97-bcfc-80bed1479c0d" (UID: "cd88c3dc-642d-4c97-bcfc-80bed1479c0d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:46:36.644875 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:36.644565    2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd88c3dc-642d-4c97-bcfc-80bed1479c0d-workload-socket\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:46:36.644875 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:36.644599    2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd88c3dc-642d-4c97-bcfc-80bed1479c0d-istio-data\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:46:36.644875 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:36.644602    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd88c3dc-642d-4c97-bcfc-80bed1479c0d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "cd88c3dc-642d-4c97-bcfc-80bed1479c0d" (UID: "cd88c3dc-642d-4c97-bcfc-80bed1479c0d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:46:36.644875 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:36.644613    2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd88c3dc-642d-4c97-bcfc-80bed1479c0d-credential-socket\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:46:36.645142 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:36.644967    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd88c3dc-642d-4c97-bcfc-80bed1479c0d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "cd88c3dc-642d-4c97-bcfc-80bed1479c0d" (UID: "cd88c3dc-642d-4c97-bcfc-80bed1479c0d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:46:36.646339 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:36.646306    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd88c3dc-642d-4c97-bcfc-80bed1479c0d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "cd88c3dc-642d-4c97-bcfc-80bed1479c0d" (UID: "cd88c3dc-642d-4c97-bcfc-80bed1479c0d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:46:36.646564 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:36.646535    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cd88c3dc-642d-4c97-bcfc-80bed1479c0d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "cd88c3dc-642d-4c97-bcfc-80bed1479c0d" (UID: "cd88c3dc-642d-4c97-bcfc-80bed1479c0d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 15:46:36.646564 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:36.646539    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd88c3dc-642d-4c97-bcfc-80bed1479c0d-kube-api-access-r2s65" (OuterVolumeSpecName: "kube-api-access-r2s65") pod "cd88c3dc-642d-4c97-bcfc-80bed1479c0d" (UID: "cd88c3dc-642d-4c97-bcfc-80bed1479c0d"). InnerVolumeSpecName "kube-api-access-r2s65". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:46:36.646738 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:36.646678    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd88c3dc-642d-4c97-bcfc-80bed1479c0d-istio-token" (OuterVolumeSpecName: "istio-token") pod "cd88c3dc-642d-4c97-bcfc-80bed1479c0d" (UID: "cd88c3dc-642d-4c97-bcfc-80bed1479c0d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:46:36.717178 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:36.717142    2559 generic.go:356] "Generic (PLEG): container finished" podID="cd88c3dc-642d-4c97-bcfc-80bed1479c0d" containerID="0f1a05543e9d40918c76b48fe84d42db9e9cb09e3139740fa6d64aae6f94951a" exitCode=0
Sep 10 15:46:36.717673 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:36.717190    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f4a9663c-openshift-default-bf4d48b4d-m9lwx"
Sep 10 15:46:36.717673 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:36.717230    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f4a9663c-openshift-default-bf4d48b4d-m9lwx" event={"ID":"cd88c3dc-642d-4c97-bcfc-80bed1479c0d","Type":"ContainerDied","Data":"0f1a05543e9d40918c76b48fe84d42db9e9cb09e3139740fa6d64aae6f94951a"}
Sep 10 15:46:36.717673 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:36.717301    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f4a9663c-openshift-default-bf4d48b4d-m9lwx" event={"ID":"cd88c3dc-642d-4c97-bcfc-80bed1479c0d","Type":"ContainerDied","Data":"ee4107f806f5dc30a323f021872cb161c19a01e8a85743bfd443e78a83a1619b"}
Sep 10 15:46:36.717673 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:36.717337    2559 scope.go:117] "RemoveContainer" containerID="0f1a05543e9d40918c76b48fe84d42db9e9cb09e3139740fa6d64aae6f94951a"
Sep 10 15:46:36.727116 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:36.727094    2559 scope.go:117] "RemoveContainer" containerID="244b67a06f0ed54a2a6812f22f4f6f0c6c42a10162e80cf5774248da0bca9e4d"
Sep 10 15:46:36.735242 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:36.735222    2559 scope.go:117] "RemoveContainer" containerID="0f1a05543e9d40918c76b48fe84d42db9e9cb09e3139740fa6d64aae6f94951a"
Sep 10 15:46:36.735505 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:46:36.735486    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f1a05543e9d40918c76b48fe84d42db9e9cb09e3139740fa6d64aae6f94951a\": container with ID starting with 0f1a05543e9d40918c76b48fe84d42db9e9cb09e3139740fa6d64aae6f94951a not found: ID does not exist" containerID="0f1a05543e9d40918c76b48fe84d42db9e9cb09e3139740fa6d64aae6f94951a"
Sep 10 15:46:36.735571 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:36.735511    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f1a05543e9d40918c76b48fe84d42db9e9cb09e3139740fa6d64aae6f94951a"} err="failed to get container status \"0f1a05543e9d40918c76b48fe84d42db9e9cb09e3139740fa6d64aae6f94951a\": rpc error: code = NotFound desc = could not find container \"0f1a05543e9d40918c76b48fe84d42db9e9cb09e3139740fa6d64aae6f94951a\": container with ID starting with 0f1a05543e9d40918c76b48fe84d42db9e9cb09e3139740fa6d64aae6f94951a not found: ID does not exist"
Sep 10 15:46:36.735571 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:36.735528    2559 scope.go:117] "RemoveContainer" containerID="244b67a06f0ed54a2a6812f22f4f6f0c6c42a10162e80cf5774248da0bca9e4d"
Sep 10 15:46:36.735813 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:46:36.735796    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"244b67a06f0ed54a2a6812f22f4f6f0c6c42a10162e80cf5774248da0bca9e4d\": container with ID starting with 244b67a06f0ed54a2a6812f22f4f6f0c6c42a10162e80cf5774248da0bca9e4d not found: ID does not exist" containerID="244b67a06f0ed54a2a6812f22f4f6f0c6c42a10162e80cf5774248da0bca9e4d"
Sep 10 15:46:36.735851 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:36.735820    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"244b67a06f0ed54a2a6812f22f4f6f0c6c42a10162e80cf5774248da0bca9e4d"} err="failed to get container status \"244b67a06f0ed54a2a6812f22f4f6f0c6c42a10162e80cf5774248da0bca9e4d\": rpc error: code = NotFound desc = could not find container \"244b67a06f0ed54a2a6812f22f4f6f0c6c42a10162e80cf5774248da0bca9e4d\": container with ID starting with 244b67a06f0ed54a2a6812f22f4f6f0c6c42a10162e80cf5774248da0bca9e4d not found: ID does not exist"
Sep 10 15:46:36.744431 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:36.744404    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f4a9663c-openshift-default-bf4d48b4d-m9lwx"]
Sep 10 15:46:36.745128 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:36.745100    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2s65\" (UniqueName: \"kubernetes.io/projected/cd88c3dc-642d-4c97-bcfc-80bed1479c0d-kube-api-access-r2s65\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:46:36.745234 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:36.745133    2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd88c3dc-642d-4c97-bcfc-80bed1479c0d-istio-envoy\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:46:36.745234 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:36.745147    2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd88c3dc-642d-4c97-bcfc-80bed1479c0d-istiod-ca-cert\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:46:36.745234 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:36.745161    2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd88c3dc-642d-4c97-bcfc-80bed1479c0d-istio-token\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:46:36.745234 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:36.745173    2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd88c3dc-642d-4c97-bcfc-80bed1479c0d-istio-podinfo\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:46:36.745234 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:36.745186    2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd88c3dc-642d-4c97-bcfc-80bed1479c0d-workload-certs\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:46:36.747322 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:36.747303    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-f4a9663c-openshift-default-bf4d48b4d-m9lwx"]
Sep 10 15:46:36.784880 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:36.784850    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd88c3dc-642d-4c97-bcfc-80bed1479c0d" path="/var/lib/kubelet/pods/cd88c3dc-642d-4c97-bcfc-80bed1479c0d/volumes"
Sep 10 15:46:36.854623 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:36.854567    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-90ca97-openshift-default-659f84d46b-9l4kj"]
Sep 10 15:46:36.854851 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:36.854824    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-90ca97-openshift-default-659f84d46b-9l4kj" podUID="f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af" containerName="istio-proxy" containerID="cri-o://30103093e03447cf221d184225d9a3d3187f2f1131c0cb61cddf9ed65c3389a3" gracePeriod=30
Sep 10 15:46:42.110823 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:42.110800    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-90ca97-openshift-default-659f84d46b-9l4kj"
Sep 10 15:46:42.189553 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:42.189521    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af-istiod-ca-cert\") pod \"f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af\" (UID: \"f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af\") "
Sep 10 15:46:42.189553 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:42.189558    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af-workload-certs\") pod \"f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af\" (UID: \"f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af\") "
Sep 10 15:46:42.189773 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:42.189589    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af-credential-socket\") pod \"f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af\" (UID: \"f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af\") "
Sep 10 15:46:42.189773 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:42.189674    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af-istio-data\") pod \"f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af\" (UID: \"f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af\") "
Sep 10 15:46:42.189773 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:42.189720    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af-istio-envoy\") pod \"f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af\" (UID: \"f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af\") "
Sep 10 15:46:42.189773 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:42.189772    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af-workload-socket\") pod \"f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af\" (UID: \"f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af\") "
Sep 10 15:46:42.189981 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:42.189828    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af-istio-podinfo\") pod \"f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af\" (UID: \"f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af\") "
Sep 10 15:46:42.189981 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:42.189858    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-75ks8\" (UniqueName: \"kubernetes.io/projected/f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af-kube-api-access-75ks8\") pod \"f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af\" (UID: \"f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af\") "
Sep 10 15:46:42.189981 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:42.189881    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af-istio-token\") pod \"f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af\" (UID: \"f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af\") "
Sep 10 15:46:42.190171 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:42.189877    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af" (UID: "f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:46:42.190265 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:42.189893    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af" (UID: "f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:46:42.190265 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:42.190038    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af-istio-data" (OuterVolumeSpecName: "istio-data") pod "f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af" (UID: "f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:46:42.190265 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:42.190057    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af" (UID: "f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:46:42.190265 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:42.190232    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af" (UID: "f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:46:42.192114 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:42.192081    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af" (UID: "f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:46:42.192218 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:42.192087    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af-istio-token" (OuterVolumeSpecName: "istio-token") pod "f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af" (UID: "f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:46:42.192359 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:42.192334    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af" (UID: "f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 15:46:42.192473 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:42.192432    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af-kube-api-access-75ks8" (OuterVolumeSpecName: "kube-api-access-75ks8") pod "f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af" (UID: "f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af"). InnerVolumeSpecName "kube-api-access-75ks8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:46:42.291172 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:42.291120    2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af-workload-socket\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:46:42.291172 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:42.291163    2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af-istio-podinfo\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:46:42.291172 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:42.291173    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75ks8\" (UniqueName: \"kubernetes.io/projected/f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af-kube-api-access-75ks8\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:46:42.291172 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:42.291183    2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af-istio-token\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:46:42.291172 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:42.291192    2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af-istiod-ca-cert\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:46:42.291481 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:42.291200    2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af-workload-certs\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:46:42.291481 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:42.291208    2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af-credential-socket\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:46:42.291481 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:42.291215    2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af-istio-data\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:46:42.291481 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:42.291224    2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af-istio-envoy\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:46:42.759468 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:42.759435    2559 generic.go:356] "Generic (PLEG): container finished" podID="f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af" containerID="30103093e03447cf221d184225d9a3d3187f2f1131c0cb61cddf9ed65c3389a3" exitCode=0
Sep 10 15:46:42.759692 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:42.759494    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-90ca97-openshift-default-659f84d46b-9l4kj"
Sep 10 15:46:42.759692 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:42.759520    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-90ca97-openshift-default-659f84d46b-9l4kj" event={"ID":"f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af","Type":"ContainerDied","Data":"30103093e03447cf221d184225d9a3d3187f2f1131c0cb61cddf9ed65c3389a3"}
Sep 10 15:46:42.759692 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:42.759595    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-90ca97-openshift-default-659f84d46b-9l4kj" event={"ID":"f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af","Type":"ContainerDied","Data":"149bf37e1f5ee477d66639d7cb3c2f5507b81a6d5549147fde960165e48b62c5"}
Sep 10 15:46:42.759692 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:42.759620    2559 scope.go:117] "RemoveContainer" containerID="30103093e03447cf221d184225d9a3d3187f2f1131c0cb61cddf9ed65c3389a3"
Sep 10 15:46:42.770704 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:42.770686    2559 scope.go:117] "RemoveContainer" containerID="29bfb5137b99e6953e3e53c39f44311f7e0e2ab67ead091e0c3c09c5f8a351f7"
Sep 10 15:46:42.779732 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:42.779712    2559 scope.go:117] "RemoveContainer" containerID="30103093e03447cf221d184225d9a3d3187f2f1131c0cb61cddf9ed65c3389a3"
Sep 10 15:46:42.780065 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:46:42.780046    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30103093e03447cf221d184225d9a3d3187f2f1131c0cb61cddf9ed65c3389a3\": container with ID starting with 30103093e03447cf221d184225d9a3d3187f2f1131c0cb61cddf9ed65c3389a3 not found: ID does not exist" containerID="30103093e03447cf221d184225d9a3d3187f2f1131c0cb61cddf9ed65c3389a3"
Sep 10 15:46:42.780132 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:42.780073    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30103093e03447cf221d184225d9a3d3187f2f1131c0cb61cddf9ed65c3389a3"} err="failed to get container status \"30103093e03447cf221d184225d9a3d3187f2f1131c0cb61cddf9ed65c3389a3\": rpc error: code = NotFound desc = could not find container \"30103093e03447cf221d184225d9a3d3187f2f1131c0cb61cddf9ed65c3389a3\": container with ID starting with 30103093e03447cf221d184225d9a3d3187f2f1131c0cb61cddf9ed65c3389a3 not found: ID does not exist"
Sep 10 15:46:42.780132 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:42.780091    2559 scope.go:117] "RemoveContainer" containerID="29bfb5137b99e6953e3e53c39f44311f7e0e2ab67ead091e0c3c09c5f8a351f7"
Sep 10 15:46:42.780347 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:46:42.780328    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29bfb5137b99e6953e3e53c39f44311f7e0e2ab67ead091e0c3c09c5f8a351f7\": container with ID starting with 29bfb5137b99e6953e3e53c39f44311f7e0e2ab67ead091e0c3c09c5f8a351f7 not found: ID does not exist" containerID="29bfb5137b99e6953e3e53c39f44311f7e0e2ab67ead091e0c3c09c5f8a351f7"
Sep 10 15:46:42.780402 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:42.780355    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29bfb5137b99e6953e3e53c39f44311f7e0e2ab67ead091e0c3c09c5f8a351f7"} err="failed to get container status \"29bfb5137b99e6953e3e53c39f44311f7e0e2ab67ead091e0c3c09c5f8a351f7\": rpc error: code = NotFound desc = could not find container \"29bfb5137b99e6953e3e53c39f44311f7e0e2ab67ead091e0c3c09c5f8a351f7\": container with ID starting with 29bfb5137b99e6953e3e53c39f44311f7e0e2ab67ead091e0c3c09c5f8a351f7 not found: ID does not exist"
Sep 10 15:46:42.789248 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:42.789225    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-90ca97-openshift-default-659f84d46b-9l4kj"]
Sep 10 15:46:42.801789 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:42.801762    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-90ca97-openshift-default-659f84d46b-9l4kj"]
Sep 10 15:46:44.785083 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:44.785048    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af" path="/var/lib/kubelet/pods/f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af/volumes"
Sep 10 15:46:45.107947 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:45.107865    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bs7hd"]
Sep 10 15:46:46.719610 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:46.719562    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-30d70c-openshift-default-786f7fff56-8sqxw"]
Sep 10 15:46:46.719996 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:46.719964    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd88c3dc-642d-4c97-bcfc-80bed1479c0d" containerName="istio-proxy"
Sep 10 15:46:46.719996 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:46.719978    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd88c3dc-642d-4c97-bcfc-80bed1479c0d" containerName="istio-proxy"
Sep 10 15:46:46.719996 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:46.719995    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="583770d9-f88f-47df-a939-1b49cd071f0f" containerName="main"
Sep 10 15:46:46.720114 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:46.720000    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="583770d9-f88f-47df-a939-1b49cd071f0f" containerName="main"
Sep 10 15:46:46.720114 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:46.720012    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a27d3806-fc72-4adf-bd20-fbe020d777f4" containerName="maas-api"
Sep 10 15:46:46.720114 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:46.720018    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a27d3806-fc72-4adf-bd20-fbe020d777f4" containerName="maas-api"
Sep 10 15:46:46.720114 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:46.720025    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af" containerName="istio-proxy"
Sep 10 15:46:46.720114 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:46.720031    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af" containerName="istio-proxy"
Sep 10 15:46:46.720114 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:46.720038    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af" containerName="istio-proxy"
Sep 10 15:46:46.720114 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:46.720043    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af" containerName="istio-proxy"
Sep 10 15:46:46.720114 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:46.720100    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af" containerName="istio-proxy"
Sep 10 15:46:46.720114 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:46.720106    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd88c3dc-642d-4c97-bcfc-80bed1479c0d" containerName="istio-proxy"
Sep 10 15:46:46.720114 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:46.720112    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="583770d9-f88f-47df-a939-1b49cd071f0f" containerName="main"
Sep 10 15:46:46.720114 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:46.720118    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd88c3dc-642d-4c97-bcfc-80bed1479c0d" containerName="istio-proxy"
Sep 10 15:46:46.720465 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:46.720125    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="a27d3806-fc72-4adf-bd20-fbe020d777f4" containerName="maas-api"
Sep 10 15:46:46.724497 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:46.724475    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-30d70c-openshift-default-786f7fff56-8sqxw"
Sep 10 15:46:46.727748 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:46.727719    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 10 15:46:46.727895 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:46.727794    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-30d70c-openshift-default-dockercfg-kmg96\""
Sep 10 15:46:46.744066 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:46.744037    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-30d70c-openshift-default-786f7fff56-8sqxw"]
Sep 10 15:46:46.828000 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:46.827964    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b514c76-21e4-43ac-a01f-9f5a44f3e773-istio-token\") pod \"e2e-shared-a-w2-30d70c-openshift-default-786f7fff56-8sqxw\" (UID: \"0b514c76-21e4-43ac-a01f-9f5a44f3e773\") " pod="openshift-ingress/e2e-shared-a-w2-30d70c-openshift-default-786f7fff56-8sqxw"
Sep 10 15:46:46.828000 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:46.828004    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b514c76-21e4-43ac-a01f-9f5a44f3e773-istio-envoy\") pod \"e2e-shared-a-w2-30d70c-openshift-default-786f7fff56-8sqxw\" (UID: \"0b514c76-21e4-43ac-a01f-9f5a44f3e773\") " pod="openshift-ingress/e2e-shared-a-w2-30d70c-openshift-default-786f7fff56-8sqxw"
Sep 10 15:46:46.828221 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:46.828097    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b514c76-21e4-43ac-a01f-9f5a44f3e773-workload-certs\") pod \"e2e-shared-a-w2-30d70c-openshift-default-786f7fff56-8sqxw\" (UID: \"0b514c76-21e4-43ac-a01f-9f5a44f3e773\") " pod="openshift-ingress/e2e-shared-a-w2-30d70c-openshift-default-786f7fff56-8sqxw"
Sep 10 15:46:46.828221 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:46.828128    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b514c76-21e4-43ac-a01f-9f5a44f3e773-istiod-ca-cert\") pod \"e2e-shared-a-w2-30d70c-openshift-default-786f7fff56-8sqxw\" (UID: \"0b514c76-21e4-43ac-a01f-9f5a44f3e773\") " pod="openshift-ingress/e2e-shared-a-w2-30d70c-openshift-default-786f7fff56-8sqxw"
Sep 10 15:46:46.828221 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:46.828150    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8d8zn\" (UniqueName: \"kubernetes.io/projected/0b514c76-21e4-43ac-a01f-9f5a44f3e773-kube-api-access-8d8zn\") pod \"e2e-shared-a-w2-30d70c-openshift-default-786f7fff56-8sqxw\" (UID: \"0b514c76-21e4-43ac-a01f-9f5a44f3e773\") " pod="openshift-ingress/e2e-shared-a-w2-30d70c-openshift-default-786f7fff56-8sqxw"
Sep 10 15:46:46.828221 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:46.828206    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b514c76-21e4-43ac-a01f-9f5a44f3e773-credential-socket\") pod \"e2e-shared-a-w2-30d70c-openshift-default-786f7fff56-8sqxw\" (UID: \"0b514c76-21e4-43ac-a01f-9f5a44f3e773\") " pod="openshift-ingress/e2e-shared-a-w2-30d70c-openshift-default-786f7fff56-8sqxw"
Sep 10 15:46:46.828354 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:46.828228    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b514c76-21e4-43ac-a01f-9f5a44f3e773-workload-socket\") pod \"e2e-shared-a-w2-30d70c-openshift-default-786f7fff56-8sqxw\" (UID: \"0b514c76-21e4-43ac-a01f-9f5a44f3e773\") " pod="openshift-ingress/e2e-shared-a-w2-30d70c-openshift-default-786f7fff56-8sqxw"
Sep 10 15:46:46.828354 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:46.828289    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b514c76-21e4-43ac-a01f-9f5a44f3e773-istio-data\") pod \"e2e-shared-a-w2-30d70c-openshift-default-786f7fff56-8sqxw\" (UID: \"0b514c76-21e4-43ac-a01f-9f5a44f3e773\") " pod="openshift-ingress/e2e-shared-a-w2-30d70c-openshift-default-786f7fff56-8sqxw"
Sep 10 15:46:46.828354 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:46.828318    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b514c76-21e4-43ac-a01f-9f5a44f3e773-istio-podinfo\") pod \"e2e-shared-a-w2-30d70c-openshift-default-786f7fff56-8sqxw\" (UID: \"0b514c76-21e4-43ac-a01f-9f5a44f3e773\") " pod="openshift-ingress/e2e-shared-a-w2-30d70c-openshift-default-786f7fff56-8sqxw"
Sep 10 15:46:46.929047 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:46.929006    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b514c76-21e4-43ac-a01f-9f5a44f3e773-istio-token\") pod \"e2e-shared-a-w2-30d70c-openshift-default-786f7fff56-8sqxw\" (UID: \"0b514c76-21e4-43ac-a01f-9f5a44f3e773\") " pod="openshift-ingress/e2e-shared-a-w2-30d70c-openshift-default-786f7fff56-8sqxw"
Sep 10 15:46:46.929236 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:46.929057    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b514c76-21e4-43ac-a01f-9f5a44f3e773-istio-envoy\") pod \"e2e-shared-a-w2-30d70c-openshift-default-786f7fff56-8sqxw\" (UID: \"0b514c76-21e4-43ac-a01f-9f5a44f3e773\") " pod="openshift-ingress/e2e-shared-a-w2-30d70c-openshift-default-786f7fff56-8sqxw"
Sep 10 15:46:46.929236 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:46.929120    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b514c76-21e4-43ac-a01f-9f5a44f3e773-workload-certs\") pod \"e2e-shared-a-w2-30d70c-openshift-default-786f7fff56-8sqxw\" (UID: \"0b514c76-21e4-43ac-a01f-9f5a44f3e773\") " pod="openshift-ingress/e2e-shared-a-w2-30d70c-openshift-default-786f7fff56-8sqxw"
Sep 10 15:46:46.929236 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:46.929195    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b514c76-21e4-43ac-a01f-9f5a44f3e773-istiod-ca-cert\") pod \"e2e-shared-a-w2-30d70c-openshift-default-786f7fff56-8sqxw\" (UID: \"0b514c76-21e4-43ac-a01f-9f5a44f3e773\") " pod="openshift-ingress/e2e-shared-a-w2-30d70c-openshift-default-786f7fff56-8sqxw"
Sep 10 15:46:46.929346 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:46.929236    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8d8zn\" (UniqueName: \"kubernetes.io/projected/0b514c76-21e4-43ac-a01f-9f5a44f3e773-kube-api-access-8d8zn\") pod \"e2e-shared-a-w2-30d70c-openshift-default-786f7fff56-8sqxw\" (UID: \"0b514c76-21e4-43ac-a01f-9f5a44f3e773\") " pod="openshift-ingress/e2e-shared-a-w2-30d70c-openshift-default-786f7fff56-8sqxw"
Sep 10 15:46:46.929346 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:46.929286    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b514c76-21e4-43ac-a01f-9f5a44f3e773-credential-socket\") pod \"e2e-shared-a-w2-30d70c-openshift-default-786f7fff56-8sqxw\" (UID: \"0b514c76-21e4-43ac-a01f-9f5a44f3e773\") " pod="openshift-ingress/e2e-shared-a-w2-30d70c-openshift-default-786f7fff56-8sqxw"
Sep 10 15:46:46.929346 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:46.929313    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b514c76-21e4-43ac-a01f-9f5a44f3e773-workload-socket\") pod \"e2e-shared-a-w2-30d70c-openshift-default-786f7fff56-8sqxw\" (UID: \"0b514c76-21e4-43ac-a01f-9f5a44f3e773\") " pod="openshift-ingress/e2e-shared-a-w2-30d70c-openshift-default-786f7fff56-8sqxw"
Sep 10 15:46:46.929501 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:46.929354    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b514c76-21e4-43ac-a01f-9f5a44f3e773-istio-data\") pod \"e2e-shared-a-w2-30d70c-openshift-default-786f7fff56-8sqxw\" (UID: \"0b514c76-21e4-43ac-a01f-9f5a44f3e773\") " pod="openshift-ingress/e2e-shared-a-w2-30d70c-openshift-default-786f7fff56-8sqxw"
Sep 10 15:46:46.929501 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:46.929385    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b514c76-21e4-43ac-a01f-9f5a44f3e773-istio-podinfo\") pod \"e2e-shared-a-w2-30d70c-openshift-default-786f7fff56-8sqxw\" (UID: \"0b514c76-21e4-43ac-a01f-9f5a44f3e773\") " pod="openshift-ingress/e2e-shared-a-w2-30d70c-openshift-default-786f7fff56-8sqxw"
Sep 10 15:46:46.929635 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:46.929531    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b514c76-21e4-43ac-a01f-9f5a44f3e773-workload-certs\") pod \"e2e-shared-a-w2-30d70c-openshift-default-786f7fff56-8sqxw\" (UID: \"0b514c76-21e4-43ac-a01f-9f5a44f3e773\") " pod="openshift-ingress/e2e-shared-a-w2-30d70c-openshift-default-786f7fff56-8sqxw"
Sep 10 15:46:46.929718 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:46.929694    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b514c76-21e4-43ac-a01f-9f5a44f3e773-workload-socket\") pod \"e2e-shared-a-w2-30d70c-openshift-default-786f7fff56-8sqxw\" (UID: \"0b514c76-21e4-43ac-a01f-9f5a44f3e773\") " pod="openshift-ingress/e2e-shared-a-w2-30d70c-openshift-default-786f7fff56-8sqxw"
Sep 10 15:46:46.929819 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:46.929786    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b514c76-21e4-43ac-a01f-9f5a44f3e773-credential-socket\") pod \"e2e-shared-a-w2-30d70c-openshift-default-786f7fff56-8sqxw\" (UID: \"0b514c76-21e4-43ac-a01f-9f5a44f3e773\") " pod="openshift-ingress/e2e-shared-a-w2-30d70c-openshift-default-786f7fff56-8sqxw"
Sep 10 15:46:46.929949 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:46.929820    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b514c76-21e4-43ac-a01f-9f5a44f3e773-istio-data\") pod \"e2e-shared-a-w2-30d70c-openshift-default-786f7fff56-8sqxw\" (UID: \"0b514c76-21e4-43ac-a01f-9f5a44f3e773\") " pod="openshift-ingress/e2e-shared-a-w2-30d70c-openshift-default-786f7fff56-8sqxw"
Sep 10 15:46:46.930015 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:46.929961    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b514c76-21e4-43ac-a01f-9f5a44f3e773-istiod-ca-cert\") pod \"e2e-shared-a-w2-30d70c-openshift-default-786f7fff56-8sqxw\" (UID: \"0b514c76-21e4-43ac-a01f-9f5a44f3e773\") " pod="openshift-ingress/e2e-shared-a-w2-30d70c-openshift-default-786f7fff56-8sqxw"
Sep 10 15:46:46.931492 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:46.931471    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b514c76-21e4-43ac-a01f-9f5a44f3e773-istio-envoy\") pod \"e2e-shared-a-w2-30d70c-openshift-default-786f7fff56-8sqxw\" (UID: \"0b514c76-21e4-43ac-a01f-9f5a44f3e773\") " pod="openshift-ingress/e2e-shared-a-w2-30d70c-openshift-default-786f7fff56-8sqxw"
Sep 10 15:46:46.931919 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:46.931899    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b514c76-21e4-43ac-a01f-9f5a44f3e773-istio-podinfo\") pod \"e2e-shared-a-w2-30d70c-openshift-default-786f7fff56-8sqxw\" (UID: \"0b514c76-21e4-43ac-a01f-9f5a44f3e773\") " pod="openshift-ingress/e2e-shared-a-w2-30d70c-openshift-default-786f7fff56-8sqxw"
Sep 10 15:46:46.937830 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:46.937807    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8d8zn\" (UniqueName: \"kubernetes.io/projected/0b514c76-21e4-43ac-a01f-9f5a44f3e773-kube-api-access-8d8zn\") pod \"e2e-shared-a-w2-30d70c-openshift-default-786f7fff56-8sqxw\" (UID: \"0b514c76-21e4-43ac-a01f-9f5a44f3e773\") " pod="openshift-ingress/e2e-shared-a-w2-30d70c-openshift-default-786f7fff56-8sqxw"
Sep 10 15:46:46.938132 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:46.938110    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b514c76-21e4-43ac-a01f-9f5a44f3e773-istio-token\") pod \"e2e-shared-a-w2-30d70c-openshift-default-786f7fff56-8sqxw\" (UID: \"0b514c76-21e4-43ac-a01f-9f5a44f3e773\") " pod="openshift-ingress/e2e-shared-a-w2-30d70c-openshift-default-786f7fff56-8sqxw"
Sep 10 15:46:47.035696 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:47.035611    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-30d70c-openshift-default-786f7fff56-8sqxw"
Sep 10 15:46:47.316967 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:47.316916    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-30d70c-openshift-default-786f7fff56-8sqxw"]
Sep 10 15:46:47.319210 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:46:47.319181    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b514c76_21e4_43ac_a01f_9f5a44f3e773.slice/crio-942f4883f80c1503675f6fcd7e7866e9c6f8fe15e785e38e1ac60602d42aaab3 WatchSource:0}: Error finding container 942f4883f80c1503675f6fcd7e7866e9c6f8fe15e785e38e1ac60602d42aaab3: Status 404 returned error can't find the container with id 942f4883f80c1503675f6fcd7e7866e9c6f8fe15e785e38e1ac60602d42aaab3
Sep 10 15:46:47.321376 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:47.321347    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 15:46:47.321466 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:47.321412    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 15:46:47.321466 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:47.321442    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 15:46:47.787981 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:47.787945    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-30d70c-openshift-default-786f7fff56-8sqxw" event={"ID":"0b514c76-21e4-43ac-a01f-9f5a44f3e773","Type":"ContainerStarted","Data":"1227fdcb190e4d6a9429b01f358d7474501e53fdcf1f85e2e87dcabc326add10"}
Sep 10 15:46:47.788434 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:47.787994    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-30d70c-openshift-default-786f7fff56-8sqxw" event={"ID":"0b514c76-21e4-43ac-a01f-9f5a44f3e773","Type":"ContainerStarted","Data":"942f4883f80c1503675f6fcd7e7866e9c6f8fe15e785e38e1ac60602d42aaab3"}
Sep 10 15:46:47.812974 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:47.812917    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-30d70c-openshift-default-786f7fff56-8sqxw" podStartSLOduration=1.812902115 podStartE2EDuration="1.812902115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:46:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:46:47.810361808 +0000 UTC m=+1183.552326927" watchObservedRunningTime="2026-09-10 15:46:47.812902115 +0000 UTC m=+1183.554867217"
Sep 10 15:46:48.036240 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:48.036196    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-30d70c-openshift-default-786f7fff56-8sqxw"
Sep 10 15:46:48.037452 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:48.037428    2559 patch_prober.go:28] interesting pod/e2e-shared-a-w2-30d70c-openshift-default-786f7fff56-8sqxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Sep 10 15:46:48.037571 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:48.037479    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-30d70c-openshift-default-786f7fff56-8sqxw" podUID="0b514c76-21e4-43ac-a01f-9f5a44f3e773" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Sep 10 15:46:49.041745 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:49.041713    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-30d70c-openshift-default-786f7fff56-8sqxw"
Sep 10 15:46:49.798801 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:49.798767    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-30d70c-openshift-default-786f7fff56-8sqxw"
Sep 10 15:46:49.799671 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:49.799646    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-30d70c-openshift-default-786f7fff56-8sqxw"
Sep 10 15:46:55.006011 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:55.005977    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bs7hd"]
Sep 10 15:46:56.276983 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:56.276948    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-3650b0-openshift-default-5b6b46d878-fcvqh"]
Sep 10 15:46:56.277608 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:56.277565    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd88c3dc-642d-4c97-bcfc-80bed1479c0d" containerName="istio-proxy"
Sep 10 15:46:56.277608 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:56.277605    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd88c3dc-642d-4c97-bcfc-80bed1479c0d" containerName="istio-proxy"
Sep 10 15:46:56.277762 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:56.277739    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="f318062b-f0b7-4baa-9c3c-3ec9d4f9f5af" containerName="istio-proxy"
Sep 10 15:46:56.281103 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:56.281081    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-3650b0-openshift-default-5b6b46d878-fcvqh"
Sep 10 15:46:56.283817 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:56.283794    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-3650b0-openshift-default-dockercfg-k7nmf\""
Sep 10 15:46:56.294470 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:56.294448    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-3650b0-openshift-default-5b6b46d878-fcvqh"]
Sep 10 15:46:56.315663 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:56.315634    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bh26t\" (UniqueName: \"kubernetes.io/projected/4c8c63e9-de0d-4181-ae91-f9a5ba79c89b-kube-api-access-bh26t\") pod \"e2e-shared-b-w3-3650b0-openshift-default-5b6b46d878-fcvqh\" (UID: \"4c8c63e9-de0d-4181-ae91-f9a5ba79c89b\") " pod="openshift-ingress/e2e-shared-b-w3-3650b0-openshift-default-5b6b46d878-fcvqh"
Sep 10 15:46:56.315809 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:56.315686    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c8c63e9-de0d-4181-ae91-f9a5ba79c89b-workload-certs\") pod \"e2e-shared-b-w3-3650b0-openshift-default-5b6b46d878-fcvqh\" (UID: \"4c8c63e9-de0d-4181-ae91-f9a5ba79c89b\") " pod="openshift-ingress/e2e-shared-b-w3-3650b0-openshift-default-5b6b46d878-fcvqh"
Sep 10 15:46:56.315809 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:56.315767    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c8c63e9-de0d-4181-ae91-f9a5ba79c89b-istiod-ca-cert\") pod \"e2e-shared-b-w3-3650b0-openshift-default-5b6b46d878-fcvqh\" (UID: \"4c8c63e9-de0d-4181-ae91-f9a5ba79c89b\") " pod="openshift-ingress/e2e-shared-b-w3-3650b0-openshift-default-5b6b46d878-fcvqh"
Sep 10 15:46:56.315894 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:56.315817    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c8c63e9-de0d-4181-ae91-f9a5ba79c89b-istio-podinfo\") pod \"e2e-shared-b-w3-3650b0-openshift-default-5b6b46d878-fcvqh\" (UID: \"4c8c63e9-de0d-4181-ae91-f9a5ba79c89b\") " pod="openshift-ingress/e2e-shared-b-w3-3650b0-openshift-default-5b6b46d878-fcvqh"
Sep 10 15:46:56.315894 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:56.315875    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c8c63e9-de0d-4181-ae91-f9a5ba79c89b-istio-data\") pod \"e2e-shared-b-w3-3650b0-openshift-default-5b6b46d878-fcvqh\" (UID: \"4c8c63e9-de0d-4181-ae91-f9a5ba79c89b\") " pod="openshift-ingress/e2e-shared-b-w3-3650b0-openshift-default-5b6b46d878-fcvqh"
Sep 10 15:46:56.315972 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:56.315907    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c8c63e9-de0d-4181-ae91-f9a5ba79c89b-workload-socket\") pod \"e2e-shared-b-w3-3650b0-openshift-default-5b6b46d878-fcvqh\" (UID: \"4c8c63e9-de0d-4181-ae91-f9a5ba79c89b\") " pod="openshift-ingress/e2e-shared-b-w3-3650b0-openshift-default-5b6b46d878-fcvqh"
Sep 10 15:46:56.315972 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:56.315940    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c8c63e9-de0d-4181-ae91-f9a5ba79c89b-istio-envoy\") pod \"e2e-shared-b-w3-3650b0-openshift-default-5b6b46d878-fcvqh\" (UID: \"4c8c63e9-de0d-4181-ae91-f9a5ba79c89b\") " pod="openshift-ingress/e2e-shared-b-w3-3650b0-openshift-default-5b6b46d878-fcvqh"
Sep 10 15:46:56.315972 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:56.315957    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c8c63e9-de0d-4181-ae91-f9a5ba79c89b-istio-token\") pod \"e2e-shared-b-w3-3650b0-openshift-default-5b6b46d878-fcvqh\" (UID: \"4c8c63e9-de0d-4181-ae91-f9a5ba79c89b\") " pod="openshift-ingress/e2e-shared-b-w3-3650b0-openshift-default-5b6b46d878-fcvqh"
Sep 10 15:46:56.316078 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:56.316026    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c8c63e9-de0d-4181-ae91-f9a5ba79c89b-credential-socket\") pod \"e2e-shared-b-w3-3650b0-openshift-default-5b6b46d878-fcvqh\" (UID: \"4c8c63e9-de0d-4181-ae91-f9a5ba79c89b\") " pod="openshift-ingress/e2e-shared-b-w3-3650b0-openshift-default-5b6b46d878-fcvqh"
Sep 10 15:46:56.416765 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:56.416731    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bh26t\" (UniqueName: \"kubernetes.io/projected/4c8c63e9-de0d-4181-ae91-f9a5ba79c89b-kube-api-access-bh26t\") pod \"e2e-shared-b-w3-3650b0-openshift-default-5b6b46d878-fcvqh\" (UID: \"4c8c63e9-de0d-4181-ae91-f9a5ba79c89b\") " pod="openshift-ingress/e2e-shared-b-w3-3650b0-openshift-default-5b6b46d878-fcvqh"
Sep 10 15:46:56.416765 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:56.416772    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c8c63e9-de0d-4181-ae91-f9a5ba79c89b-workload-certs\") pod \"e2e-shared-b-w3-3650b0-openshift-default-5b6b46d878-fcvqh\" (UID: \"4c8c63e9-de0d-4181-ae91-f9a5ba79c89b\") " pod="openshift-ingress/e2e-shared-b-w3-3650b0-openshift-default-5b6b46d878-fcvqh"
Sep 10 15:46:56.417009 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:56.416806    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c8c63e9-de0d-4181-ae91-f9a5ba79c89b-istiod-ca-cert\") pod \"e2e-shared-b-w3-3650b0-openshift-default-5b6b46d878-fcvqh\" (UID: \"4c8c63e9-de0d-4181-ae91-f9a5ba79c89b\") " pod="openshift-ingress/e2e-shared-b-w3-3650b0-openshift-default-5b6b46d878-fcvqh"
Sep 10 15:46:56.417009 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:56.416826    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c8c63e9-de0d-4181-ae91-f9a5ba79c89b-istio-podinfo\") pod \"e2e-shared-b-w3-3650b0-openshift-default-5b6b46d878-fcvqh\" (UID: \"4c8c63e9-de0d-4181-ae91-f9a5ba79c89b\") " pod="openshift-ingress/e2e-shared-b-w3-3650b0-openshift-default-5b6b46d878-fcvqh"
Sep 10 15:46:56.417009 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:56.416896    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c8c63e9-de0d-4181-ae91-f9a5ba79c89b-istio-data\") pod \"e2e-shared-b-w3-3650b0-openshift-default-5b6b46d878-fcvqh\" (UID: \"4c8c63e9-de0d-4181-ae91-f9a5ba79c89b\") " pod="openshift-ingress/e2e-shared-b-w3-3650b0-openshift-default-5b6b46d878-fcvqh"
Sep 10 15:46:56.417009 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:56.416952    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c8c63e9-de0d-4181-ae91-f9a5ba79c89b-workload-socket\") pod \"e2e-shared-b-w3-3650b0-openshift-default-5b6b46d878-fcvqh\" (UID: \"4c8c63e9-de0d-4181-ae91-f9a5ba79c89b\") " pod="openshift-ingress/e2e-shared-b-w3-3650b0-openshift-default-5b6b46d878-fcvqh"
Sep 10 15:46:56.417009 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:56.417000    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c8c63e9-de0d-4181-ae91-f9a5ba79c89b-istio-envoy\") pod \"e2e-shared-b-w3-3650b0-openshift-default-5b6b46d878-fcvqh\" (UID: \"4c8c63e9-de0d-4181-ae91-f9a5ba79c89b\") " pod="openshift-ingress/e2e-shared-b-w3-3650b0-openshift-default-5b6b46d878-fcvqh"
Sep 10 15:46:56.417223 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:56.417022    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c8c63e9-de0d-4181-ae91-f9a5ba79c89b-istio-token\") pod \"e2e-shared-b-w3-3650b0-openshift-default-5b6b46d878-fcvqh\" (UID: \"4c8c63e9-de0d-4181-ae91-f9a5ba79c89b\") " pod="openshift-ingress/e2e-shared-b-w3-3650b0-openshift-default-5b6b46d878-fcvqh"
Sep 10 15:46:56.417223 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:56.417047    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c8c63e9-de0d-4181-ae91-f9a5ba79c89b-credential-socket\") pod \"e2e-shared-b-w3-3650b0-openshift-default-5b6b46d878-fcvqh\" (UID: \"4c8c63e9-de0d-4181-ae91-f9a5ba79c89b\") " pod="openshift-ingress/e2e-shared-b-w3-3650b0-openshift-default-5b6b46d878-fcvqh"
Sep 10 15:46:56.417407 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:56.417380    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c8c63e9-de0d-4181-ae91-f9a5ba79c89b-istio-data\") pod \"e2e-shared-b-w3-3650b0-openshift-default-5b6b46d878-fcvqh\" (UID: \"4c8c63e9-de0d-4181-ae91-f9a5ba79c89b\") " pod="openshift-ingress/e2e-shared-b-w3-3650b0-openshift-default-5b6b46d878-fcvqh"
Sep 10 15:46:56.417495 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:56.417411    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c8c63e9-de0d-4181-ae91-f9a5ba79c89b-workload-socket\") pod \"e2e-shared-b-w3-3650b0-openshift-default-5b6b46d878-fcvqh\" (UID: \"4c8c63e9-de0d-4181-ae91-f9a5ba79c89b\") " pod="openshift-ingress/e2e-shared-b-w3-3650b0-openshift-default-5b6b46d878-fcvqh"
Sep 10 15:46:56.417551 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:56.417519    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c8c63e9-de0d-4181-ae91-f9a5ba79c89b-workload-certs\") pod \"e2e-shared-b-w3-3650b0-openshift-default-5b6b46d878-fcvqh\" (UID: \"4c8c63e9-de0d-4181-ae91-f9a5ba79c89b\") " pod="openshift-ingress/e2e-shared-b-w3-3650b0-openshift-default-5b6b46d878-fcvqh"
Sep 10 15:46:56.417737 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:56.417721    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c8c63e9-de0d-4181-ae91-f9a5ba79c89b-credential-socket\") pod \"e2e-shared-b-w3-3650b0-openshift-default-5b6b46d878-fcvqh\" (UID: \"4c8c63e9-de0d-4181-ae91-f9a5ba79c89b\") " pod="openshift-ingress/e2e-shared-b-w3-3650b0-openshift-default-5b6b46d878-fcvqh"
Sep 10 15:46:56.417796 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:56.417778    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c8c63e9-de0d-4181-ae91-f9a5ba79c89b-istiod-ca-cert\") pod \"e2e-shared-b-w3-3650b0-openshift-default-5b6b46d878-fcvqh\" (UID: \"4c8c63e9-de0d-4181-ae91-f9a5ba79c89b\") " pod="openshift-ingress/e2e-shared-b-w3-3650b0-openshift-default-5b6b46d878-fcvqh"
Sep 10 15:46:56.419653 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:56.419634    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c8c63e9-de0d-4181-ae91-f9a5ba79c89b-istio-envoy\") pod \"e2e-shared-b-w3-3650b0-openshift-default-5b6b46d878-fcvqh\" (UID: \"4c8c63e9-de0d-4181-ae91-f9a5ba79c89b\") " pod="openshift-ingress/e2e-shared-b-w3-3650b0-openshift-default-5b6b46d878-fcvqh"
Sep 10 15:46:56.420000 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:56.419981    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c8c63e9-de0d-4181-ae91-f9a5ba79c89b-istio-podinfo\") pod \"e2e-shared-b-w3-3650b0-openshift-default-5b6b46d878-fcvqh\" (UID: \"4c8c63e9-de0d-4181-ae91-f9a5ba79c89b\") " pod="openshift-ingress/e2e-shared-b-w3-3650b0-openshift-default-5b6b46d878-fcvqh"
Sep 10 15:46:56.425879 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:56.425859    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bh26t\" (UniqueName: \"kubernetes.io/projected/4c8c63e9-de0d-4181-ae91-f9a5ba79c89b-kube-api-access-bh26t\") pod \"e2e-shared-b-w3-3650b0-openshift-default-5b6b46d878-fcvqh\" (UID: \"4c8c63e9-de0d-4181-ae91-f9a5ba79c89b\") " pod="openshift-ingress/e2e-shared-b-w3-3650b0-openshift-default-5b6b46d878-fcvqh"
Sep 10 15:46:56.425958 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:56.425878    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c8c63e9-de0d-4181-ae91-f9a5ba79c89b-istio-token\") pod \"e2e-shared-b-w3-3650b0-openshift-default-5b6b46d878-fcvqh\" (UID: \"4c8c63e9-de0d-4181-ae91-f9a5ba79c89b\") " pod="openshift-ingress/e2e-shared-b-w3-3650b0-openshift-default-5b6b46d878-fcvqh"
Sep 10 15:46:56.592386 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:56.592292    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-3650b0-openshift-default-5b6b46d878-fcvqh"
Sep 10 15:46:56.872290 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:56.872244    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-3650b0-openshift-default-5b6b46d878-fcvqh"]
Sep 10 15:46:56.875724 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:56.875678    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 15:46:56.875851 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:56.875824    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 15:46:56.875937 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:56.875917    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 15:46:57.846117 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:57.846067    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-3650b0-openshift-default-5b6b46d878-fcvqh" event={"ID":"4c8c63e9-de0d-4181-ae91-f9a5ba79c89b","Type":"ContainerStarted","Data":"3415437d37340eefa951ab8b9a3f7f8caa543151cfd743b12f834e86a770dddb"}
Sep 10 15:46:57.846680 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:57.846145    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-3650b0-openshift-default-5b6b46d878-fcvqh" event={"ID":"4c8c63e9-de0d-4181-ae91-f9a5ba79c89b","Type":"ContainerStarted","Data":"3f9d40d9140213abd4e8bc99a6ba7e9036399af2fc680a9a12a43dc5dfa3b03a"}
Sep 10 15:46:57.877461 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:57.877392    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-3650b0-openshift-default-5b6b46d878-fcvqh" podStartSLOduration=1.877371123 podStartE2EDuration="1.877371123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:46:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:46:57.87686974 +0000 UTC m=+1193.618834847" watchObservedRunningTime="2026-09-10 15:46:57.877371123 +0000 UTC m=+1193.619336226"
Sep 10 15:46:58.592911 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:58.592878    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-3650b0-openshift-default-5b6b46d878-fcvqh"
Sep 10 15:46:58.597620 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:58.597570    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-3650b0-openshift-default-5b6b46d878-fcvqh"
Sep 10 15:46:58.855513 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:58.855416    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-3650b0-openshift-default-5b6b46d878-fcvqh"
Sep 10 15:46:58.857043 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:46:58.856800    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-3650b0-openshift-default-5b6b46d878-fcvqh"
Sep 10 15:47:05.006520 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:05.006467    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ec08ac-openshift-default-55ff657f88-tddkk"]
Sep 10 15:47:05.011428 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:05.011404    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ec08ac-openshift-default-55ff657f88-tddkk"
Sep 10 15:47:05.015540 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:05.015512    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-ec08ac-openshift-default-dockercfg-hq9s8\""
Sep 10 15:47:05.024728 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:05.024703    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ec08ac-openshift-default-55ff657f88-tddkk"]
Sep 10 15:47:05.104040 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:05.104007    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23f9cdab-2ba9-4e1b-a322-7b6d5b09409c-workload-socket\") pod \"e2e-shared-b-w2-ec08ac-openshift-default-55ff657f88-tddkk\" (UID: \"23f9cdab-2ba9-4e1b-a322-7b6d5b09409c\") " pod="openshift-ingress/e2e-shared-b-w2-ec08ac-openshift-default-55ff657f88-tddkk"
Sep 10 15:47:05.104040 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:05.104043    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23f9cdab-2ba9-4e1b-a322-7b6d5b09409c-credential-socket\") pod \"e2e-shared-b-w2-ec08ac-openshift-default-55ff657f88-tddkk\" (UID: \"23f9cdab-2ba9-4e1b-a322-7b6d5b09409c\") " pod="openshift-ingress/e2e-shared-b-w2-ec08ac-openshift-default-55ff657f88-tddkk"
Sep 10 15:47:05.104298 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:05.104098    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23f9cdab-2ba9-4e1b-a322-7b6d5b09409c-istio-token\") pod \"e2e-shared-b-w2-ec08ac-openshift-default-55ff657f88-tddkk\" (UID: \"23f9cdab-2ba9-4e1b-a322-7b6d5b09409c\") " pod="openshift-ingress/e2e-shared-b-w2-ec08ac-openshift-default-55ff657f88-tddkk"
Sep 10 15:47:05.104298 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:05.104120    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23f9cdab-2ba9-4e1b-a322-7b6d5b09409c-istio-data\") pod \"e2e-shared-b-w2-ec08ac-openshift-default-55ff657f88-tddkk\" (UID: \"23f9cdab-2ba9-4e1b-a322-7b6d5b09409c\") " pod="openshift-ingress/e2e-shared-b-w2-ec08ac-openshift-default-55ff657f88-tddkk"
Sep 10 15:47:05.104298 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:05.104139    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23f9cdab-2ba9-4e1b-a322-7b6d5b09409c-istiod-ca-cert\") pod \"e2e-shared-b-w2-ec08ac-openshift-default-55ff657f88-tddkk\" (UID: \"23f9cdab-2ba9-4e1b-a322-7b6d5b09409c\") " pod="openshift-ingress/e2e-shared-b-w2-ec08ac-openshift-default-55ff657f88-tddkk"
Sep 10 15:47:05.104298 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:05.104220    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9j75\" (UniqueName: \"kubernetes.io/projected/23f9cdab-2ba9-4e1b-a322-7b6d5b09409c-kube-api-access-z9j75\") pod \"e2e-shared-b-w2-ec08ac-openshift-default-55ff657f88-tddkk\" (UID: \"23f9cdab-2ba9-4e1b-a322-7b6d5b09409c\") " pod="openshift-ingress/e2e-shared-b-w2-ec08ac-openshift-default-55ff657f88-tddkk"
Sep 10 15:47:05.104298 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:05.104265    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23f9cdab-2ba9-4e1b-a322-7b6d5b09409c-workload-certs\") pod \"e2e-shared-b-w2-ec08ac-openshift-default-55ff657f88-tddkk\" (UID: \"23f9cdab-2ba9-4e1b-a322-7b6d5b09409c\") " pod="openshift-ingress/e2e-shared-b-w2-ec08ac-openshift-default-55ff657f88-tddkk"
Sep 10 15:47:05.104298 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:05.104297    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23f9cdab-2ba9-4e1b-a322-7b6d5b09409c-istio-podinfo\") pod \"e2e-shared-b-w2-ec08ac-openshift-default-55ff657f88-tddkk\" (UID: \"23f9cdab-2ba9-4e1b-a322-7b6d5b09409c\") " pod="openshift-ingress/e2e-shared-b-w2-ec08ac-openshift-default-55ff657f88-tddkk"
Sep 10 15:47:05.104564 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:05.104370    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23f9cdab-2ba9-4e1b-a322-7b6d5b09409c-istio-envoy\") pod \"e2e-shared-b-w2-ec08ac-openshift-default-55ff657f88-tddkk\" (UID: \"23f9cdab-2ba9-4e1b-a322-7b6d5b09409c\") " pod="openshift-ingress/e2e-shared-b-w2-ec08ac-openshift-default-55ff657f88-tddkk"
Sep 10 15:47:05.205294 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:05.205255    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23f9cdab-2ba9-4e1b-a322-7b6d5b09409c-istio-data\") pod \"e2e-shared-b-w2-ec08ac-openshift-default-55ff657f88-tddkk\" (UID: \"23f9cdab-2ba9-4e1b-a322-7b6d5b09409c\") " pod="openshift-ingress/e2e-shared-b-w2-ec08ac-openshift-default-55ff657f88-tddkk"
Sep 10 15:47:05.205294 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:05.205296    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23f9cdab-2ba9-4e1b-a322-7b6d5b09409c-istiod-ca-cert\") pod \"e2e-shared-b-w2-ec08ac-openshift-default-55ff657f88-tddkk\" (UID: \"23f9cdab-2ba9-4e1b-a322-7b6d5b09409c\") " pod="openshift-ingress/e2e-shared-b-w2-ec08ac-openshift-default-55ff657f88-tddkk"
Sep 10 15:47:05.205548 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:05.205422    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9j75\" (UniqueName: \"kubernetes.io/projected/23f9cdab-2ba9-4e1b-a322-7b6d5b09409c-kube-api-access-z9j75\") pod \"e2e-shared-b-w2-ec08ac-openshift-default-55ff657f88-tddkk\" (UID: \"23f9cdab-2ba9-4e1b-a322-7b6d5b09409c\") " pod="openshift-ingress/e2e-shared-b-w2-ec08ac-openshift-default-55ff657f88-tddkk"
Sep 10 15:47:05.205548 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:05.205477    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23f9cdab-2ba9-4e1b-a322-7b6d5b09409c-workload-certs\") pod \"e2e-shared-b-w2-ec08ac-openshift-default-55ff657f88-tddkk\" (UID: \"23f9cdab-2ba9-4e1b-a322-7b6d5b09409c\") " pod="openshift-ingress/e2e-shared-b-w2-ec08ac-openshift-default-55ff657f88-tddkk"
Sep 10 15:47:05.205548 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:05.205511    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23f9cdab-2ba9-4e1b-a322-7b6d5b09409c-istio-podinfo\") pod \"e2e-shared-b-w2-ec08ac-openshift-default-55ff657f88-tddkk\" (UID: \"23f9cdab-2ba9-4e1b-a322-7b6d5b09409c\") " pod="openshift-ingress/e2e-shared-b-w2-ec08ac-openshift-default-55ff657f88-tddkk"
Sep 10 15:47:05.205548 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:05.205544    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23f9cdab-2ba9-4e1b-a322-7b6d5b09409c-istio-envoy\") pod \"e2e-shared-b-w2-ec08ac-openshift-default-55ff657f88-tddkk\" (UID: \"23f9cdab-2ba9-4e1b-a322-7b6d5b09409c\") " pod="openshift-ingress/e2e-shared-b-w2-ec08ac-openshift-default-55ff657f88-tddkk"
Sep 10 15:47:05.205830 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:05.205739    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23f9cdab-2ba9-4e1b-a322-7b6d5b09409c-istio-data\") pod \"e2e-shared-b-w2-ec08ac-openshift-default-55ff657f88-tddkk\" (UID: \"23f9cdab-2ba9-4e1b-a322-7b6d5b09409c\") " pod="openshift-ingress/e2e-shared-b-w2-ec08ac-openshift-default-55ff657f88-tddkk"
Sep 10 15:47:05.205830 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:05.205825    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23f9cdab-2ba9-4e1b-a322-7b6d5b09409c-workload-socket\") pod \"e2e-shared-b-w2-ec08ac-openshift-default-55ff657f88-tddkk\" (UID: \"23f9cdab-2ba9-4e1b-a322-7b6d5b09409c\") " pod="openshift-ingress/e2e-shared-b-w2-ec08ac-openshift-default-55ff657f88-tddkk"
Sep 10 15:47:05.205939 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:05.205854    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23f9cdab-2ba9-4e1b-a322-7b6d5b09409c-credential-socket\") pod \"e2e-shared-b-w2-ec08ac-openshift-default-55ff657f88-tddkk\" (UID: \"23f9cdab-2ba9-4e1b-a322-7b6d5b09409c\") " pod="openshift-ingress/e2e-shared-b-w2-ec08ac-openshift-default-55ff657f88-tddkk"
Sep 10 15:47:05.205939 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:05.205914    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23f9cdab-2ba9-4e1b-a322-7b6d5b09409c-istio-token\") pod \"e2e-shared-b-w2-ec08ac-openshift-default-55ff657f88-tddkk\" (UID: \"23f9cdab-2ba9-4e1b-a322-7b6d5b09409c\") " pod="openshift-ingress/e2e-shared-b-w2-ec08ac-openshift-default-55ff657f88-tddkk"
Sep 10 15:47:05.206060 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:05.205991    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23f9cdab-2ba9-4e1b-a322-7b6d5b09409c-workload-certs\") pod \"e2e-shared-b-w2-ec08ac-openshift-default-55ff657f88-tddkk\" (UID: \"23f9cdab-2ba9-4e1b-a322-7b6d5b09409c\") " pod="openshift-ingress/e2e-shared-b-w2-ec08ac-openshift-default-55ff657f88-tddkk"
Sep 10 15:47:05.206228 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:05.206202    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23f9cdab-2ba9-4e1b-a322-7b6d5b09409c-workload-socket\") pod \"e2e-shared-b-w2-ec08ac-openshift-default-55ff657f88-tddkk\" (UID: \"23f9cdab-2ba9-4e1b-a322-7b6d5b09409c\") " pod="openshift-ingress/e2e-shared-b-w2-ec08ac-openshift-default-55ff657f88-tddkk"
Sep 10 15:47:05.206308 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:05.206215    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23f9cdab-2ba9-4e1b-a322-7b6d5b09409c-istiod-ca-cert\") pod \"e2e-shared-b-w2-ec08ac-openshift-default-55ff657f88-tddkk\" (UID: \"23f9cdab-2ba9-4e1b-a322-7b6d5b09409c\") " pod="openshift-ingress/e2e-shared-b-w2-ec08ac-openshift-default-55ff657f88-tddkk"
Sep 10 15:47:05.206352 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:05.206302    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23f9cdab-2ba9-4e1b-a322-7b6d5b09409c-credential-socket\") pod \"e2e-shared-b-w2-ec08ac-openshift-default-55ff657f88-tddkk\" (UID: \"23f9cdab-2ba9-4e1b-a322-7b6d5b09409c\") " pod="openshift-ingress/e2e-shared-b-w2-ec08ac-openshift-default-55ff657f88-tddkk"
Sep 10 15:47:05.208213 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:05.208183    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23f9cdab-2ba9-4e1b-a322-7b6d5b09409c-istio-envoy\") pod \"e2e-shared-b-w2-ec08ac-openshift-default-55ff657f88-tddkk\" (UID: \"23f9cdab-2ba9-4e1b-a322-7b6d5b09409c\") " pod="openshift-ingress/e2e-shared-b-w2-ec08ac-openshift-default-55ff657f88-tddkk"
Sep 10 15:47:05.208325 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:05.208249    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23f9cdab-2ba9-4e1b-a322-7b6d5b09409c-istio-podinfo\") pod \"e2e-shared-b-w2-ec08ac-openshift-default-55ff657f88-tddkk\" (UID: \"23f9cdab-2ba9-4e1b-a322-7b6d5b09409c\") " pod="openshift-ingress/e2e-shared-b-w2-ec08ac-openshift-default-55ff657f88-tddkk"
Sep 10 15:47:05.215365 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:05.215333    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23f9cdab-2ba9-4e1b-a322-7b6d5b09409c-istio-token\") pod \"e2e-shared-b-w2-ec08ac-openshift-default-55ff657f88-tddkk\" (UID: \"23f9cdab-2ba9-4e1b-a322-7b6d5b09409c\") " pod="openshift-ingress/e2e-shared-b-w2-ec08ac-openshift-default-55ff657f88-tddkk"
Sep 10 15:47:05.216523 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:05.216496    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9j75\" (UniqueName: \"kubernetes.io/projected/23f9cdab-2ba9-4e1b-a322-7b6d5b09409c-kube-api-access-z9j75\") pod \"e2e-shared-b-w2-ec08ac-openshift-default-55ff657f88-tddkk\" (UID: \"23f9cdab-2ba9-4e1b-a322-7b6d5b09409c\") " pod="openshift-ingress/e2e-shared-b-w2-ec08ac-openshift-default-55ff657f88-tddkk"
Sep 10 15:47:05.322412 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:05.322369    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ec08ac-openshift-default-55ff657f88-tddkk"
Sep 10 15:47:05.605318 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:05.605288    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ec08ac-openshift-default-55ff657f88-tddkk"]
Sep 10 15:47:05.607248 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:47:05.607221    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23f9cdab_2ba9_4e1b_a322_7b6d5b09409c.slice/crio-d0621d61fb82187292328f8abb55843050c862cb4e99acb88bedcbf01cfa934c WatchSource:0}: Error finding container d0621d61fb82187292328f8abb55843050c862cb4e99acb88bedcbf01cfa934c: Status 404 returned error can't find the container with id d0621d61fb82187292328f8abb55843050c862cb4e99acb88bedcbf01cfa934c
Sep 10 15:47:05.609871 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:05.609836    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 15:47:05.609980 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:05.609938    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 15:47:05.610046 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:05.609980    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 15:47:05.755005 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:05.754968    2559 patch_prober.go:28] interesting pod/e2e-shared-a-w2-30d70c-openshift-default-786f7fff56-8sqxw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": read tcp 10.133.0.2:46738->10.133.0.58:15021: read: connection reset by peer" start-of-body=
Sep 10 15:47:05.755191 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:05.755043    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-30d70c-openshift-default-786f7fff56-8sqxw" podUID="0b514c76-21e4-43ac-a01f-9f5a44f3e773" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": read tcp 10.133.0.2:46738->10.133.0.58:15021: read: connection reset by peer"
Sep 10 15:47:05.902066 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:05.901969    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ec08ac-openshift-default-55ff657f88-tddkk" event={"ID":"23f9cdab-2ba9-4e1b-a322-7b6d5b09409c","Type":"ContainerStarted","Data":"ae069168d1e1d52d67ec180abbfc4217bf193db1f60ce8f9987c4867558b1f11"}
Sep 10 15:47:05.902066 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:05.902029    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ec08ac-openshift-default-55ff657f88-tddkk" event={"ID":"23f9cdab-2ba9-4e1b-a322-7b6d5b09409c","Type":"ContainerStarted","Data":"d0621d61fb82187292328f8abb55843050c862cb4e99acb88bedcbf01cfa934c"}
Sep 10 15:47:05.903932 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:05.903904    2559 generic.go:356] "Generic (PLEG): container finished" podID="0b514c76-21e4-43ac-a01f-9f5a44f3e773" containerID="1227fdcb190e4d6a9429b01f358d7474501e53fdcf1f85e2e87dcabc326add10" exitCode=0
Sep 10 15:47:05.904071 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:05.903946    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-30d70c-openshift-default-786f7fff56-8sqxw" event={"ID":"0b514c76-21e4-43ac-a01f-9f5a44f3e773","Type":"ContainerDied","Data":"1227fdcb190e4d6a9429b01f358d7474501e53fdcf1f85e2e87dcabc326add10"}
Sep 10 15:47:05.904351 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:05.904334    2559 scope.go:117] "RemoveContainer" containerID="1227fdcb190e4d6a9429b01f358d7474501e53fdcf1f85e2e87dcabc326add10"
Sep 10 15:47:05.905500 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:05.905469    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 15:47:05.905594 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:05.905556    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 15:47:05.905646 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:05.905614    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 15:47:05.938850 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:05.938802    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-ec08ac-openshift-default-55ff657f88-tddkk" podStartSLOduration=1.938785017 podStartE2EDuration="1.938785017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:47:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:47:05.93396896 +0000 UTC m=+1201.675934062" watchObservedRunningTime="2026-09-10 15:47:05.938785017 +0000 UTC m=+1201.680750120"
Sep 10 15:47:06.323154 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:06.323113    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-ec08ac-openshift-default-55ff657f88-tddkk"
Sep 10 15:47:06.324750 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:06.324725    2559 patch_prober.go:28] interesting pod/e2e-shared-b-w2-ec08ac-openshift-default-55ff657f88-tddkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Sep 10 15:47:06.324888 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:06.324773    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ec08ac-openshift-default-55ff657f88-tddkk" podUID="23f9cdab-2ba9-4e1b-a322-7b6d5b09409c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Sep 10 15:47:06.911413 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:06.911378    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-30d70c-openshift-default-786f7fff56-8sqxw" event={"ID":"0b514c76-21e4-43ac-a01f-9f5a44f3e773","Type":"ContainerStarted","Data":"1329e99e26268019e553eabd003d841bad0da5a2dff6c15e0c8f4ff52392aead"}
Sep 10 15:47:07.036731 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:07.036701    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-30d70c-openshift-default-786f7fff56-8sqxw"
Sep 10 15:47:07.036976 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:07.036956    2559 patch_prober.go:28] interesting pod/e2e-shared-a-w2-30d70c-openshift-default-786f7fff56-8sqxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Sep 10 15:47:07.037028 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:07.036999    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-30d70c-openshift-default-786f7fff56-8sqxw" podUID="0b514c76-21e4-43ac-a01f-9f5a44f3e773" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Sep 10 15:47:07.327739 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:07.327654    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-ec08ac-openshift-default-55ff657f88-tddkk"
Sep 10 15:47:07.917594 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:07.917548    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-ec08ac-openshift-default-55ff657f88-tddkk"
Sep 10 15:47:07.918678 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:07.918658    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-ec08ac-openshift-default-55ff657f88-tddkk"
Sep 10 15:47:08.036722 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:08.036686    2559 patch_prober.go:28] interesting pod/e2e-shared-a-w2-30d70c-openshift-default-786f7fff56-8sqxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Sep 10 15:47:08.036884 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:08.036759    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-30d70c-openshift-default-786f7fff56-8sqxw" podUID="0b514c76-21e4-43ac-a01f-9f5a44f3e773" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Sep 10 15:47:09.036597 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:09.036553    2559 patch_prober.go:28] interesting pod/e2e-shared-a-w2-30d70c-openshift-default-786f7fff56-8sqxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Sep 10 15:47:09.036993 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:09.036632    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-30d70c-openshift-default-786f7fff56-8sqxw" podUID="0b514c76-21e4-43ac-a01f-9f5a44f3e773" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Sep 10 15:47:09.920141 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:09.920112    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bs7hd"]
Sep 10 15:47:10.036213 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:10.036176    2559 patch_prober.go:28] interesting pod/e2e-shared-a-w2-30d70c-openshift-default-786f7fff56-8sqxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Sep 10 15:47:10.036411 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:10.036239    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-30d70c-openshift-default-786f7fff56-8sqxw" podUID="0b514c76-21e4-43ac-a01f-9f5a44f3e773" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Sep 10 15:47:11.036278 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:11.036241    2559 patch_prober.go:28] interesting pod/e2e-shared-a-w2-30d70c-openshift-default-786f7fff56-8sqxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Sep 10 15:47:11.036717 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:11.036300    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-30d70c-openshift-default-786f7fff56-8sqxw" podUID="0b514c76-21e4-43ac-a01f-9f5a44f3e773" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Sep 10 15:47:11.226447 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:11.226395    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ec08ac-5c45cc5d-zr65m"]
Sep 10 15:47:11.229113 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:11.229091    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ec08ac-5c45cc5d-zr65m"
Sep 10 15:47:11.232194 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:11.232172    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-ec08ac-serving-cert\""
Sep 10 15:47:11.233958 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:11.233833    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 10 15:47:11.233958 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:11.233853    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-5qzmv\""
Sep 10 15:47:11.234125 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:11.233949    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 15:47:11.234125 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:11.233982    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 10 15:47:11.258618 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:11.252455    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ec08ac-5c45cc5d-zr65m"]
Sep 10 15:47:11.370325 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:11.370229    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tj65q\" (UniqueName: \"kubernetes.io/projected/5cc8c332-932a-4bc8-9bc4-5483c84f4a3d-kube-api-access-tj65q\") pod \"maas-api-e2e-shared-b-w2-ec08ac-5c45cc5d-zr65m\" (UID: \"5cc8c332-932a-4bc8-9bc4-5483c84f4a3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ec08ac-5c45cc5d-zr65m"
Sep 10 15:47:11.370325 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:11.370275    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5cc8c332-932a-4bc8-9bc4-5483c84f4a3d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-ec08ac-5c45cc5d-zr65m\" (UID: \"5cc8c332-932a-4bc8-9bc4-5483c84f4a3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ec08ac-5c45cc5d-zr65m"
Sep 10 15:47:11.370325 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:11.370324    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cc8c332-932a-4bc8-9bc4-5483c84f4a3d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-ec08ac-5c45cc5d-zr65m\" (UID: \"5cc8c332-932a-4bc8-9bc4-5483c84f4a3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ec08ac-5c45cc5d-zr65m"
Sep 10 15:47:11.370593 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:11.370365    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cc8c332-932a-4bc8-9bc4-5483c84f4a3d-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-ec08ac-5c45cc5d-zr65m\" (UID: \"5cc8c332-932a-4bc8-9bc4-5483c84f4a3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ec08ac-5c45cc5d-zr65m"
Sep 10 15:47:11.471112 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:11.471076    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cc8c332-932a-4bc8-9bc4-5483c84f4a3d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-ec08ac-5c45cc5d-zr65m\" (UID: \"5cc8c332-932a-4bc8-9bc4-5483c84f4a3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ec08ac-5c45cc5d-zr65m"
Sep 10 15:47:11.471375 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:11.471122    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cc8c332-932a-4bc8-9bc4-5483c84f4a3d-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-ec08ac-5c45cc5d-zr65m\" (UID: \"5cc8c332-932a-4bc8-9bc4-5483c84f4a3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ec08ac-5c45cc5d-zr65m"
Sep 10 15:47:11.471375 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:11.471224    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tj65q\" (UniqueName: \"kubernetes.io/projected/5cc8c332-932a-4bc8-9bc4-5483c84f4a3d-kube-api-access-tj65q\") pod \"maas-api-e2e-shared-b-w2-ec08ac-5c45cc5d-zr65m\" (UID: \"5cc8c332-932a-4bc8-9bc4-5483c84f4a3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ec08ac-5c45cc5d-zr65m"
Sep 10 15:47:11.471375 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:11.471260    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5cc8c332-932a-4bc8-9bc4-5483c84f4a3d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-ec08ac-5c45cc5d-zr65m\" (UID: \"5cc8c332-932a-4bc8-9bc4-5483c84f4a3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ec08ac-5c45cc5d-zr65m"
Sep 10 15:47:11.471857 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:11.471832    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cc8c332-932a-4bc8-9bc4-5483c84f4a3d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-ec08ac-5c45cc5d-zr65m\" (UID: \"5cc8c332-932a-4bc8-9bc4-5483c84f4a3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ec08ac-5c45cc5d-zr65m"
Sep 10 15:47:11.473547 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:11.473524    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cc8c332-932a-4bc8-9bc4-5483c84f4a3d-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-ec08ac-5c45cc5d-zr65m\" (UID: \"5cc8c332-932a-4bc8-9bc4-5483c84f4a3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ec08ac-5c45cc5d-zr65m"
Sep 10 15:47:11.473661 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:11.473611    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5cc8c332-932a-4bc8-9bc4-5483c84f4a3d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-ec08ac-5c45cc5d-zr65m\" (UID: \"5cc8c332-932a-4bc8-9bc4-5483c84f4a3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ec08ac-5c45cc5d-zr65m"
Sep 10 15:47:11.490325 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:11.490305    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tj65q\" (UniqueName: \"kubernetes.io/projected/5cc8c332-932a-4bc8-9bc4-5483c84f4a3d-kube-api-access-tj65q\") pod \"maas-api-e2e-shared-b-w2-ec08ac-5c45cc5d-zr65m\" (UID: \"5cc8c332-932a-4bc8-9bc4-5483c84f4a3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ec08ac-5c45cc5d-zr65m"
Sep 10 15:47:11.553337 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:11.553295    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ec08ac-5c45cc5d-zr65m"
Sep 10 15:47:11.835628 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:11.835604    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ec08ac-5c45cc5d-zr65m"]
Sep 10 15:47:11.838517 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:47:11.838489    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cc8c332_932a_4bc8_9bc4_5483c84f4a3d.slice/crio-581a0d2f3cc88e2087be10660b6d04856a0d70999e77ecc83d80db4b97b1ad28 WatchSource:0}: Error finding container 581a0d2f3cc88e2087be10660b6d04856a0d70999e77ecc83d80db4b97b1ad28: Status 404 returned error can't find the container with id 581a0d2f3cc88e2087be10660b6d04856a0d70999e77ecc83d80db4b97b1ad28
Sep 10 15:47:11.967600 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:11.956647    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ec08ac-5c45cc5d-zr65m" event={"ID":"5cc8c332-932a-4bc8-9bc4-5483c84f4a3d","Type":"ContainerStarted","Data":"581a0d2f3cc88e2087be10660b6d04856a0d70999e77ecc83d80db4b97b1ad28"}
Sep 10 15:47:12.036828 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:12.036792    2559 patch_prober.go:28] interesting pod/e2e-shared-a-w2-30d70c-openshift-default-786f7fff56-8sqxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Sep 10 15:47:12.037256 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:12.036845    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-30d70c-openshift-default-786f7fff56-8sqxw" podUID="0b514c76-21e4-43ac-a01f-9f5a44f3e773" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Sep 10 15:47:12.969041 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:12.968986    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ec08ac-5c45cc5d-zr65m" event={"ID":"5cc8c332-932a-4bc8-9bc4-5483c84f4a3d","Type":"ContainerStarted","Data":"ddd9cff474768d9e2beb515aae0cf70d38ad8215b555c783fab1a3c6924ee969"}
Sep 10 15:47:12.969235 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:12.969068    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ec08ac-5c45cc5d-zr65m"
Sep 10 15:47:12.993132 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:12.993081    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ec08ac-5c45cc5d-zr65m" podStartSLOduration=1.583010264 podStartE2EDuration="1.993064655s" totalImagesPullingTime="410.054391ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:47:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:47:12.991191155 +0000 UTC m=+1208.733156283" watchObservedRunningTime="2026-09-10 15:47:12.993064655 +0000 UTC m=+1208.735029771"
Sep 10 15:47:13.039560 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:13.039525    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-30d70c-openshift-default-786f7fff56-8sqxw"
Sep 10 15:47:13.044665 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:13.040542    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-30d70c-openshift-default-786f7fff56-8sqxw"
Sep 10 15:47:13.049942 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:13.049919    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-30d70c-openshift-default-786f7fff56-8sqxw"
Sep 10 15:47:18.980438 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:18.980406    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ec08ac-5c45cc5d-zr65m"
Sep 10 15:47:21.016572 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:21.016538    2559 generic.go:356] "Generic (PLEG): container finished" podID="4c8c63e9-de0d-4181-ae91-f9a5ba79c89b" containerID="3415437d37340eefa951ab8b9a3f7f8caa543151cfd743b12f834e86a770dddb" exitCode=0
Sep 10 15:47:21.017048 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:21.016619    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-3650b0-openshift-default-5b6b46d878-fcvqh" event={"ID":"4c8c63e9-de0d-4181-ae91-f9a5ba79c89b","Type":"ContainerDied","Data":"3415437d37340eefa951ab8b9a3f7f8caa543151cfd743b12f834e86a770dddb"}
Sep 10 15:47:21.017048 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:21.017028    2559 scope.go:117] "RemoveContainer" containerID="3415437d37340eefa951ab8b9a3f7f8caa543151cfd743b12f834e86a770dddb"
Sep 10 15:47:21.018129 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:21.018103    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 15:47:21.018189 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:21.018167    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 15:47:21.018229 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:21.018197    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 15:47:22.024719 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:22.024686    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-3650b0-openshift-default-5b6b46d878-fcvqh" event={"ID":"4c8c63e9-de0d-4181-ae91-f9a5ba79c89b","Type":"ContainerStarted","Data":"a075235a0a77580834e2742720fe67b7c92229511da8b1275ec91fec13e84a50"}
Sep 10 15:47:22.593144 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:22.593100    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-3650b0-openshift-default-5b6b46d878-fcvqh"
Sep 10 15:47:22.593334 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:22.593146    2559 patch_prober.go:28] interesting pod/e2e-shared-b-w3-3650b0-openshift-default-5b6b46d878-fcvqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Sep 10 15:47:22.593334 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:22.593218    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-3650b0-openshift-default-5b6b46d878-fcvqh" podUID="4c8c63e9-de0d-4181-ae91-f9a5ba79c89b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Sep 10 15:47:23.592958 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:23.592924    2559 patch_prober.go:28] interesting pod/e2e-shared-b-w3-3650b0-openshift-default-5b6b46d878-fcvqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Sep 10 15:47:23.593454 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:23.593411    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-3650b0-openshift-default-5b6b46d878-fcvqh" podUID="4c8c63e9-de0d-4181-ae91-f9a5ba79c89b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Sep 10 15:47:24.593133 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:24.593097    2559 patch_prober.go:28] interesting pod/e2e-shared-b-w3-3650b0-openshift-default-5b6b46d878-fcvqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Sep 10 15:47:24.593648 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:24.593160    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-3650b0-openshift-default-5b6b46d878-fcvqh" podUID="4c8c63e9-de0d-4181-ae91-f9a5ba79c89b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Sep 10 15:47:25.593208 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:25.593168    2559 patch_prober.go:28] interesting pod/e2e-shared-b-w3-3650b0-openshift-default-5b6b46d878-fcvqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Sep 10 15:47:25.593765 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:25.593727    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-3650b0-openshift-default-5b6b46d878-fcvqh" podUID="4c8c63e9-de0d-4181-ae91-f9a5ba79c89b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Sep 10 15:47:26.593198 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:26.593161    2559 patch_prober.go:28] interesting pod/e2e-shared-b-w3-3650b0-openshift-default-5b6b46d878-fcvqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Sep 10 15:47:26.593377 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:26.593221    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-3650b0-openshift-default-5b6b46d878-fcvqh" podUID="4c8c63e9-de0d-4181-ae91-f9a5ba79c89b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Sep 10 15:47:26.905024 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:26.904928    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bs7hd"]
Sep 10 15:47:27.593656 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:27.593618    2559 patch_prober.go:28] interesting pod/e2e-shared-b-w3-3650b0-openshift-default-5b6b46d878-fcvqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Sep 10 15:47:27.594130 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:27.593709    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-3650b0-openshift-default-5b6b46d878-fcvqh" podUID="4c8c63e9-de0d-4181-ae91-f9a5ba79c89b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Sep 10 15:47:28.439669 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:28.439636    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7749-e4c233"]
Sep 10 15:47:28.442592 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:28.442552    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7749-e4c233"
Sep 10 15:47:28.445036 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:28.445016    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-wtngs\""
Sep 10 15:47:28.450536 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:28.450515    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7749-e4c233"]
Sep 10 15:47:28.542849 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:28.542814    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vm2gh\" (UniqueName: \"kubernetes.io/projected/569ae829-2e76-465a-ac73-dfdb77572b72-kube-api-access-vm2gh\") pod \"test-curl-7749-e4c233\" (UID: \"569ae829-2e76-465a-ac73-dfdb77572b72\") " pod="opendatahub/test-curl-7749-e4c233"
Sep 10 15:47:28.598260 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:28.598233    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-3650b0-openshift-default-5b6b46d878-fcvqh"
Sep 10 15:47:28.598706 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:28.598692    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-3650b0-openshift-default-5b6b46d878-fcvqh"
Sep 10 15:47:28.599362 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:28.599341    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-3650b0-openshift-default-5b6b46d878-fcvqh"
Sep 10 15:47:28.643764 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:28.643732    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vm2gh\" (UniqueName: \"kubernetes.io/projected/569ae829-2e76-465a-ac73-dfdb77572b72-kube-api-access-vm2gh\") pod \"test-curl-7749-e4c233\" (UID: \"569ae829-2e76-465a-ac73-dfdb77572b72\") " pod="opendatahub/test-curl-7749-e4c233"
Sep 10 15:47:28.654437 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:28.654392    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vm2gh\" (UniqueName: \"kubernetes.io/projected/569ae829-2e76-465a-ac73-dfdb77572b72-kube-api-access-vm2gh\") pod \"test-curl-7749-e4c233\" (UID: \"569ae829-2e76-465a-ac73-dfdb77572b72\") " pod="opendatahub/test-curl-7749-e4c233"
Sep 10 15:47:28.754214 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:28.754120    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7749-e4c233"
Sep 10 15:47:29.030832 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:29.030807    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7749-e4c233"]
Sep 10 15:47:29.095121 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:29.095093    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7749-e4c233" event={"ID":"569ae829-2e76-465a-ac73-dfdb77572b72","Type":"ContainerStarted","Data":"92871e98babe1325823497cd5de40452aebeb8ba3a152b73e3071bb989e826c0"}
Sep 10 15:47:30.102695 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:30.102660    2559 generic.go:356] "Generic (PLEG): container finished" podID="23f9cdab-2ba9-4e1b-a322-7b6d5b09409c" containerID="ae069168d1e1d52d67ec180abbfc4217bf193db1f60ce8f9987c4867558b1f11" exitCode=0
Sep 10 15:47:30.103231 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:30.102737    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ec08ac-openshift-default-55ff657f88-tddkk" event={"ID":"23f9cdab-2ba9-4e1b-a322-7b6d5b09409c","Type":"ContainerDied","Data":"ae069168d1e1d52d67ec180abbfc4217bf193db1f60ce8f9987c4867558b1f11"}
Sep 10 15:47:30.103231 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:30.103223    2559 scope.go:117] "RemoveContainer" containerID="ae069168d1e1d52d67ec180abbfc4217bf193db1f60ce8f9987c4867558b1f11"
Sep 10 15:47:30.104309 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:30.104287    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7749-e4c233" event={"ID":"569ae829-2e76-465a-ac73-dfdb77572b72","Type":"ContainerStarted","Data":"a90b78ebba781653ba1881a72e66bda88a7f7bfbdd2810a42b0bcac58d47068b"}
Sep 10 15:47:30.104432 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:30.104393    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 15:47:30.104501 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:30.104473    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 15:47:30.104559 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:30.104521    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 15:47:30.140518 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:30.140473    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7749-e4c233" podStartSLOduration=2.140458574 podStartE2EDuration="2.140458574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:47:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:47:30.138365672 +0000 UTC m=+1225.880330793" watchObservedRunningTime="2026-09-10 15:47:30.140458574 +0000 UTC m=+1225.882423695"
Sep 10 15:47:30.785671 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:30.785634    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7749-e4c233"]
Sep 10 15:47:30.788924 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:30.788899    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7749-e4c233"]
Sep 10 15:47:30.905558 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:30.905465    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7749-cce0e2"]
Sep 10 15:47:30.905961 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:30.905941    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="569ae829-2e76-465a-ac73-dfdb77572b72" containerName="test-curl-7749-e4c233"
Sep 10 15:47:30.906057 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:30.905964    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="569ae829-2e76-465a-ac73-dfdb77572b72" containerName="test-curl-7749-e4c233"
Sep 10 15:47:30.906125 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:30.906111    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="569ae829-2e76-465a-ac73-dfdb77572b72" containerName="test-curl-7749-e4c233"
Sep 10 15:47:30.908273 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:30.908253    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7749-cce0e2"
Sep 10 15:47:30.915098 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:30.915078    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7749-cce0e2"]
Sep 10 15:47:30.968462 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:30.968264    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-js27g\" (UniqueName: \"kubernetes.io/projected/e269c7bd-d4b6-4a1d-90ec-e9437abeeb8a-kube-api-access-js27g\") pod \"test-curl-7749-cce0e2\" (UID: \"e269c7bd-d4b6-4a1d-90ec-e9437abeeb8a\") " pod="opendatahub/test-curl-7749-cce0e2"
Sep 10 15:47:31.069442 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:31.069301    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-js27g\" (UniqueName: \"kubernetes.io/projected/e269c7bd-d4b6-4a1d-90ec-e9437abeeb8a-kube-api-access-js27g\") pod \"test-curl-7749-cce0e2\" (UID: \"e269c7bd-d4b6-4a1d-90ec-e9437abeeb8a\") " pod="opendatahub/test-curl-7749-cce0e2"
Sep 10 15:47:31.081587 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:31.081545    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-js27g\" (UniqueName: \"kubernetes.io/projected/e269c7bd-d4b6-4a1d-90ec-e9437abeeb8a-kube-api-access-js27g\") pod \"test-curl-7749-cce0e2\" (UID: \"e269c7bd-d4b6-4a1d-90ec-e9437abeeb8a\") " pod="opendatahub/test-curl-7749-cce0e2"
Sep 10 15:47:31.111833 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:31.111803    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ec08ac-openshift-default-55ff657f88-tddkk" event={"ID":"23f9cdab-2ba9-4e1b-a322-7b6d5b09409c","Type":"ContainerStarted","Data":"9711ff3626e63d4681c61fe7b21d19cf1f693d84e4c5305517924ce8b7175d8f"}
Sep 10 15:47:31.112179 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:47:31.111994    2559 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7749-e4c233" container="test-curl-7749-e4c233"
Sep 10 15:47:31.219140 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:31.219107    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7749-cce0e2"
Sep 10 15:47:31.322719 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:31.322693    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-ec08ac-openshift-default-55ff657f88-tddkk"
Sep 10 15:47:31.322937 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:31.322913    2559 patch_prober.go:28] interesting pod/e2e-shared-b-w2-ec08ac-openshift-default-55ff657f88-tddkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Sep 10 15:47:31.322987 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:31.322962    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ec08ac-openshift-default-55ff657f88-tddkk" podUID="23f9cdab-2ba9-4e1b-a322-7b6d5b09409c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Sep 10 15:47:31.514259 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:31.514231    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7749-cce0e2"]
Sep 10 15:47:31.515273 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:47:31.515241    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode269c7bd_d4b6_4a1d_90ec_e9437abeeb8a.slice/crio-2f539cafe4a49af957e0ee88762cd5d9879e2a0da3d3ed8415c60bbe1a6cc391 WatchSource:0}: Error finding container 2f539cafe4a49af957e0ee88762cd5d9879e2a0da3d3ed8415c60bbe1a6cc391: Status 404 returned error can't find the container with id 2f539cafe4a49af957e0ee88762cd5d9879e2a0da3d3ed8415c60bbe1a6cc391
Sep 10 15:47:31.725459 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:31.725421    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-c7257b/auth-test-model-c7257b-kserve-7dcfccff79-zxdpt"]
Sep 10 15:47:31.728286 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:31.728265    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-c7257b/auth-test-model-c7257b-kserve-7dcfccff79-zxdpt"
Sep 10 15:47:31.731120 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:31.731093    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-c7257b\"/\"kube-root-ca.crt\""
Sep 10 15:47:31.731271 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:31.731137    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-c7257b\"/\"auth-test-model-c7257b-kserve-self-signed-certs\""
Sep 10 15:47:31.731271 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:31.731100    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-c7257b\"/\"openshift-service-ca.crt\""
Sep 10 15:47:31.731271 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:31.731234    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-c7257b\"/\"default-dockercfg-s5vj9\""
Sep 10 15:47:31.758603 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:31.749673    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-c7257b/auth-test-model-c7257b-kserve-7dcfccff79-zxdpt"]
Sep 10 15:47:31.774743 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:31.774653    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3d20bc8-5feb-4a2f-bfef-6f08c06d5114-home\") pod \"auth-test-model-c7257b-kserve-7dcfccff79-zxdpt\" (UID: \"a3d20bc8-5feb-4a2f-bfef-6f08c06d5114\") " pod="ai-tenant-e2e-shared-a-w3-c7257b/auth-test-model-c7257b-kserve-7dcfccff79-zxdpt"
Sep 10 15:47:31.774743 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:31.774710    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3d20bc8-5feb-4a2f-bfef-6f08c06d5114-tmp-dir\") pod \"auth-test-model-c7257b-kserve-7dcfccff79-zxdpt\" (UID: \"a3d20bc8-5feb-4a2f-bfef-6f08c06d5114\") " pod="ai-tenant-e2e-shared-a-w3-c7257b/auth-test-model-c7257b-kserve-7dcfccff79-zxdpt"
Sep 10 15:47:31.774934 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:31.774799    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3d20bc8-5feb-4a2f-bfef-6f08c06d5114-model-cache\") pod \"auth-test-model-c7257b-kserve-7dcfccff79-zxdpt\" (UID: \"a3d20bc8-5feb-4a2f-bfef-6f08c06d5114\") " pod="ai-tenant-e2e-shared-a-w3-c7257b/auth-test-model-c7257b-kserve-7dcfccff79-zxdpt"
Sep 10 15:47:31.774934 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:31.774843    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3d20bc8-5feb-4a2f-bfef-6f08c06d5114-dshm\") pod \"auth-test-model-c7257b-kserve-7dcfccff79-zxdpt\" (UID: \"a3d20bc8-5feb-4a2f-bfef-6f08c06d5114\") " pod="ai-tenant-e2e-shared-a-w3-c7257b/auth-test-model-c7257b-kserve-7dcfccff79-zxdpt"
Sep 10 15:47:31.774934 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:31.774914    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4t6wc\" (UniqueName: \"kubernetes.io/projected/a3d20bc8-5feb-4a2f-bfef-6f08c06d5114-kube-api-access-4t6wc\") pod \"auth-test-model-c7257b-kserve-7dcfccff79-zxdpt\" (UID: \"a3d20bc8-5feb-4a2f-bfef-6f08c06d5114\") " pod="ai-tenant-e2e-shared-a-w3-c7257b/auth-test-model-c7257b-kserve-7dcfccff79-zxdpt"
Sep 10 15:47:31.775059 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:31.774954    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3d20bc8-5feb-4a2f-bfef-6f08c06d5114-tls-certs\") pod \"auth-test-model-c7257b-kserve-7dcfccff79-zxdpt\" (UID: \"a3d20bc8-5feb-4a2f-bfef-6f08c06d5114\") " pod="ai-tenant-e2e-shared-a-w3-c7257b/auth-test-model-c7257b-kserve-7dcfccff79-zxdpt"
Sep 10 15:47:31.876236 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:31.876195    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4t6wc\" (UniqueName: \"kubernetes.io/projected/a3d20bc8-5feb-4a2f-bfef-6f08c06d5114-kube-api-access-4t6wc\") pod \"auth-test-model-c7257b-kserve-7dcfccff79-zxdpt\" (UID: \"a3d20bc8-5feb-4a2f-bfef-6f08c06d5114\") " pod="ai-tenant-e2e-shared-a-w3-c7257b/auth-test-model-c7257b-kserve-7dcfccff79-zxdpt"
Sep 10 15:47:31.876432 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:31.876269    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3d20bc8-5feb-4a2f-bfef-6f08c06d5114-tls-certs\") pod \"auth-test-model-c7257b-kserve-7dcfccff79-zxdpt\" (UID: \"a3d20bc8-5feb-4a2f-bfef-6f08c06d5114\") " pod="ai-tenant-e2e-shared-a-w3-c7257b/auth-test-model-c7257b-kserve-7dcfccff79-zxdpt"
Sep 10 15:47:31.876432 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:31.876307    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3d20bc8-5feb-4a2f-bfef-6f08c06d5114-home\") pod \"auth-test-model-c7257b-kserve-7dcfccff79-zxdpt\" (UID: \"a3d20bc8-5feb-4a2f-bfef-6f08c06d5114\") " pod="ai-tenant-e2e-shared-a-w3-c7257b/auth-test-model-c7257b-kserve-7dcfccff79-zxdpt"
Sep 10 15:47:31.876432 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:31.876387    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3d20bc8-5feb-4a2f-bfef-6f08c06d5114-tmp-dir\") pod \"auth-test-model-c7257b-kserve-7dcfccff79-zxdpt\" (UID: \"a3d20bc8-5feb-4a2f-bfef-6f08c06d5114\") " pod="ai-tenant-e2e-shared-a-w3-c7257b/auth-test-model-c7257b-kserve-7dcfccff79-zxdpt"
Sep 10 15:47:31.876684 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:31.876442    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3d20bc8-5feb-4a2f-bfef-6f08c06d5114-model-cache\") pod \"auth-test-model-c7257b-kserve-7dcfccff79-zxdpt\" (UID: \"a3d20bc8-5feb-4a2f-bfef-6f08c06d5114\") " pod="ai-tenant-e2e-shared-a-w3-c7257b/auth-test-model-c7257b-kserve-7dcfccff79-zxdpt"
Sep 10 15:47:31.876684 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:31.876506    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3d20bc8-5feb-4a2f-bfef-6f08c06d5114-dshm\") pod \"auth-test-model-c7257b-kserve-7dcfccff79-zxdpt\" (UID: \"a3d20bc8-5feb-4a2f-bfef-6f08c06d5114\") " pod="ai-tenant-e2e-shared-a-w3-c7257b/auth-test-model-c7257b-kserve-7dcfccff79-zxdpt"
Sep 10 15:47:31.876988 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:31.876957    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3d20bc8-5feb-4a2f-bfef-6f08c06d5114-home\") pod \"auth-test-model-c7257b-kserve-7dcfccff79-zxdpt\" (UID: \"a3d20bc8-5feb-4a2f-bfef-6f08c06d5114\") " pod="ai-tenant-e2e-shared-a-w3-c7257b/auth-test-model-c7257b-kserve-7dcfccff79-zxdpt"
Sep 10 15:47:31.877117 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:31.877086    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3d20bc8-5feb-4a2f-bfef-6f08c06d5114-tmp-dir\") pod \"auth-test-model-c7257b-kserve-7dcfccff79-zxdpt\" (UID: \"a3d20bc8-5feb-4a2f-bfef-6f08c06d5114\") " pod="ai-tenant-e2e-shared-a-w3-c7257b/auth-test-model-c7257b-kserve-7dcfccff79-zxdpt"
Sep 10 15:47:31.877117 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:31.877106    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3d20bc8-5feb-4a2f-bfef-6f08c06d5114-model-cache\") pod \"auth-test-model-c7257b-kserve-7dcfccff79-zxdpt\" (UID: \"a3d20bc8-5feb-4a2f-bfef-6f08c06d5114\") " pod="ai-tenant-e2e-shared-a-w3-c7257b/auth-test-model-c7257b-kserve-7dcfccff79-zxdpt"
Sep 10 15:47:31.878759 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:31.878733    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3d20bc8-5feb-4a2f-bfef-6f08c06d5114-dshm\") pod \"auth-test-model-c7257b-kserve-7dcfccff79-zxdpt\" (UID: \"a3d20bc8-5feb-4a2f-bfef-6f08c06d5114\") " pod="ai-tenant-e2e-shared-a-w3-c7257b/auth-test-model-c7257b-kserve-7dcfccff79-zxdpt"
Sep 10 15:47:31.878951 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:31.878936    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3d20bc8-5feb-4a2f-bfef-6f08c06d5114-tls-certs\") pod \"auth-test-model-c7257b-kserve-7dcfccff79-zxdpt\" (UID: \"a3d20bc8-5feb-4a2f-bfef-6f08c06d5114\") " pod="ai-tenant-e2e-shared-a-w3-c7257b/auth-test-model-c7257b-kserve-7dcfccff79-zxdpt"
Sep 10 15:47:31.887308 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:31.887288    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4t6wc\" (UniqueName: \"kubernetes.io/projected/a3d20bc8-5feb-4a2f-bfef-6f08c06d5114-kube-api-access-4t6wc\") pod \"auth-test-model-c7257b-kserve-7dcfccff79-zxdpt\" (UID: \"a3d20bc8-5feb-4a2f-bfef-6f08c06d5114\") " pod="ai-tenant-e2e-shared-a-w3-c7257b/auth-test-model-c7257b-kserve-7dcfccff79-zxdpt"
Sep 10 15:47:32.039917 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:32.039611    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-c7257b/auth-test-model-c7257b-kserve-7dcfccff79-zxdpt"
Sep 10 15:47:32.121644 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:32.121614    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7749-cce0e2" event={"ID":"e269c7bd-d4b6-4a1d-90ec-e9437abeeb8a","Type":"ContainerStarted","Data":"0814c92fb9fdb05c1cc6502383564f79398b46dbd41bad4bc1eb12702885488e"}
Sep 10 15:47:32.122035 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:32.121665    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7749-cce0e2" event={"ID":"e269c7bd-d4b6-4a1d-90ec-e9437abeeb8a","Type":"ContainerStarted","Data":"2f539cafe4a49af957e0ee88762cd5d9879e2a0da3d3ed8415c60bbe1a6cc391"}
Sep 10 15:47:32.122096 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:32.122017    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7749-e4c233" podUID="569ae829-2e76-465a-ac73-dfdb77572b72" containerName="test-curl-7749-e4c233" containerID="cri-o://a90b78ebba781653ba1881a72e66bda88a7f7bfbdd2810a42b0bcac58d47068b" gracePeriod=2
Sep 10 15:47:32.155600 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:47:32.153813    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7749-e4c233\" is forbidden: User \"system:node:ip-10-0-130-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-30.ec2.internal' and this object" podUID="569ae829-2e76-465a-ac73-dfdb77572b72" pod="opendatahub/test-curl-7749-e4c233"
Sep 10 15:47:32.159598 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:32.157225    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7749-cce0e2" podStartSLOduration=2.157203322 podStartE2EDuration="2.157203322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:47:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:47:32.138597979 +0000 UTC m=+1227.880563082" watchObservedRunningTime="2026-09-10 15:47:32.157203322 +0000 UTC m=+1227.899168428"
Sep 10 15:47:32.322774 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:32.322697    2559 patch_prober.go:28] interesting pod/e2e-shared-b-w2-ec08ac-openshift-default-55ff657f88-tddkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Sep 10 15:47:32.322774 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:32.322743    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ec08ac-openshift-default-55ff657f88-tddkk" podUID="23f9cdab-2ba9-4e1b-a322-7b6d5b09409c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Sep 10 15:47:32.348859 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:32.348804    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-c7257b/auth-test-model-c7257b-kserve-7dcfccff79-zxdpt"]
Sep 10 15:47:32.360933 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:47:32.356732    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda3d20bc8_5feb_4a2f_bfef_6f08c06d5114.slice/crio-1f8157a37194bdddaf5aea5f482c73aa25ffcdea4c749abc3935b8bf23289046 WatchSource:0}: Error finding container 1f8157a37194bdddaf5aea5f482c73aa25ffcdea4c749abc3935b8bf23289046: Status 404 returned error can't find the container with id 1f8157a37194bdddaf5aea5f482c73aa25ffcdea4c749abc3935b8bf23289046
Sep 10 15:47:32.785507 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:32.785470    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7749-cce0e2"]
Sep 10 15:47:32.787009 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:32.786983    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7749-cce0e2"]
Sep 10 15:47:33.129644 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:33.129541    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-c7257b/auth-test-model-c7257b-kserve-7dcfccff79-zxdpt" event={"ID":"a3d20bc8-5feb-4a2f-bfef-6f08c06d5114","Type":"ContainerStarted","Data":"0992cf65eae36ec5a379899d2bfc5439fc5e2b6ffa9ff773095346fecfd5c9a3"}
Sep 10 15:47:33.129644 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:33.129611    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-c7257b/auth-test-model-c7257b-kserve-7dcfccff79-zxdpt" event={"ID":"a3d20bc8-5feb-4a2f-bfef-6f08c06d5114","Type":"ContainerStarted","Data":"1f8157a37194bdddaf5aea5f482c73aa25ffcdea4c749abc3935b8bf23289046"}
Sep 10 15:47:33.130057 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:47:33.129728    2559 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7749-cce0e2" container="test-curl-7749-cce0e2"
Sep 10 15:47:33.148318 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:33.148260    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-c7257b/auth-test-model-c7257b-kserve-7dcfccff79-zxdpt" podStartSLOduration=2.148240418 podStartE2EDuration="2.148240418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:47:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:47:33.146786765 +0000 UTC m=+1228.888751869" watchObservedRunningTime="2026-09-10 15:47:33.148240418 +0000 UTC m=+1228.890205522"
Sep 10 15:47:33.323281 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:33.323245    2559 patch_prober.go:28] interesting pod/e2e-shared-b-w2-ec08ac-openshift-default-55ff657f88-tddkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Sep 10 15:47:33.323431 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:33.323301    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ec08ac-openshift-default-55ff657f88-tddkk" podUID="23f9cdab-2ba9-4e1b-a322-7b6d5b09409c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Sep 10 15:47:34.135921 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:34.135889    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7749-cce0e2" podUID="e269c7bd-d4b6-4a1d-90ec-e9437abeeb8a" containerName="test-curl-7749-cce0e2" containerID="cri-o://0814c92fb9fdb05c1cc6502383564f79398b46dbd41bad4bc1eb12702885488e" gracePeriod=2
Sep 10 15:47:34.138270 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:47:34.138238    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7749-cce0e2\" is forbidden: User \"system:node:ip-10-0-130-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-30.ec2.internal' and this object" podUID="e269c7bd-d4b6-4a1d-90ec-e9437abeeb8a" pod="opendatahub/test-curl-7749-cce0e2"
Sep 10 15:47:34.299626 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:34.299598    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7749-e4c233"
Sep 10 15:47:34.302040 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:47:34.301999    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7749-cce0e2\" is forbidden: User \"system:node:ip-10-0-130-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-30.ec2.internal' and this object" podUID="e269c7bd-d4b6-4a1d-90ec-e9437abeeb8a" pod="opendatahub/test-curl-7749-cce0e2"
Sep 10 15:47:34.323457 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:34.323430    2559 patch_prober.go:28] interesting pod/e2e-shared-b-w2-ec08ac-openshift-default-55ff657f88-tddkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Sep 10 15:47:34.323550 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:34.323477    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ec08ac-openshift-default-55ff657f88-tddkk" podUID="23f9cdab-2ba9-4e1b-a322-7b6d5b09409c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Sep 10 15:47:34.400972 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:34.400885    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vm2gh\" (UniqueName: \"kubernetes.io/projected/569ae829-2e76-465a-ac73-dfdb77572b72-kube-api-access-vm2gh\") pod \"569ae829-2e76-465a-ac73-dfdb77572b72\" (UID: \"569ae829-2e76-465a-ac73-dfdb77572b72\") "
Sep 10 15:47:34.403043 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:34.403009    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/569ae829-2e76-465a-ac73-dfdb77572b72-kube-api-access-vm2gh" (OuterVolumeSpecName: "kube-api-access-vm2gh") pod "569ae829-2e76-465a-ac73-dfdb77572b72" (UID: "569ae829-2e76-465a-ac73-dfdb77572b72"). InnerVolumeSpecName "kube-api-access-vm2gh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:47:34.502135 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:34.502105    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vm2gh\" (UniqueName: \"kubernetes.io/projected/569ae829-2e76-465a-ac73-dfdb77572b72-kube-api-access-vm2gh\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:47:34.784617 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:47:34.784560    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7749-cce0e2\" is forbidden: User \"system:node:ip-10-0-130-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-30.ec2.internal' and this object" podUID="e269c7bd-d4b6-4a1d-90ec-e9437abeeb8a" pod="opendatahub/test-curl-7749-cce0e2"
Sep 10 15:47:34.785899 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:34.785879    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="569ae829-2e76-465a-ac73-dfdb77572b72" path="/var/lib/kubelet/pods/569ae829-2e76-465a-ac73-dfdb77572b72/volumes"
Sep 10 15:47:35.151026 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:35.150927    2559 generic.go:356] "Generic (PLEG): container finished" podID="569ae829-2e76-465a-ac73-dfdb77572b72" containerID="a90b78ebba781653ba1881a72e66bda88a7f7bfbdd2810a42b0bcac58d47068b" exitCode=137
Sep 10 15:47:35.151559 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:35.150980    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7749-e4c233"
Sep 10 15:47:35.151838 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:35.151028    2559 scope.go:117] "RemoveContainer" containerID="a90b78ebba781653ba1881a72e66bda88a7f7bfbdd2810a42b0bcac58d47068b"
Sep 10 15:47:35.171031 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:35.171004    2559 scope.go:117] "RemoveContainer" containerID="a90b78ebba781653ba1881a72e66bda88a7f7bfbdd2810a42b0bcac58d47068b"
Sep 10 15:47:35.171701 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:47:35.171673    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a90b78ebba781653ba1881a72e66bda88a7f7bfbdd2810a42b0bcac58d47068b\": container with ID starting with a90b78ebba781653ba1881a72e66bda88a7f7bfbdd2810a42b0bcac58d47068b not found: ID does not exist" containerID="a90b78ebba781653ba1881a72e66bda88a7f7bfbdd2810a42b0bcac58d47068b"
Sep 10 15:47:35.171838 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:35.171718    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a90b78ebba781653ba1881a72e66bda88a7f7bfbdd2810a42b0bcac58d47068b"} err="failed to get container status \"a90b78ebba781653ba1881a72e66bda88a7f7bfbdd2810a42b0bcac58d47068b\": rpc error: code = NotFound desc = could not find container \"a90b78ebba781653ba1881a72e66bda88a7f7bfbdd2810a42b0bcac58d47068b\": container with ID starting with a90b78ebba781653ba1881a72e66bda88a7f7bfbdd2810a42b0bcac58d47068b not found: ID does not exist"
Sep 10 15:47:35.323723 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:35.323686    2559 patch_prober.go:28] interesting pod/e2e-shared-b-w2-ec08ac-openshift-default-55ff657f88-tddkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Sep 10 15:47:35.323905 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:35.323746    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ec08ac-openshift-default-55ff657f88-tddkk" podUID="23f9cdab-2ba9-4e1b-a322-7b6d5b09409c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Sep 10 15:47:36.290743 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:36.290719    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7749-cce0e2"
Sep 10 15:47:36.318962 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:36.318936    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-js27g\" (UniqueName: \"kubernetes.io/projected/e269c7bd-d4b6-4a1d-90ec-e9437abeeb8a-kube-api-access-js27g\") pod \"e269c7bd-d4b6-4a1d-90ec-e9437abeeb8a\" (UID: \"e269c7bd-d4b6-4a1d-90ec-e9437abeeb8a\") "
Sep 10 15:47:36.320948 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:36.320924    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e269c7bd-d4b6-4a1d-90ec-e9437abeeb8a-kube-api-access-js27g" (OuterVolumeSpecName: "kube-api-access-js27g") pod "e269c7bd-d4b6-4a1d-90ec-e9437abeeb8a" (UID: "e269c7bd-d4b6-4a1d-90ec-e9437abeeb8a"). InnerVolumeSpecName "kube-api-access-js27g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:47:36.327331 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:36.327311    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-ec08ac-openshift-default-55ff657f88-tddkk"
Sep 10 15:47:36.327607 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:36.327587    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-ec08ac-openshift-default-55ff657f88-tddkk"
Sep 10 15:47:36.328351 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:36.328333    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-ec08ac-openshift-default-55ff657f88-tddkk"
Sep 10 15:47:36.420015 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:36.419975    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-js27g\" (UniqueName: \"kubernetes.io/projected/e269c7bd-d4b6-4a1d-90ec-e9437abeeb8a-kube-api-access-js27g\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:47:36.785128 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:36.785047    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e269c7bd-d4b6-4a1d-90ec-e9437abeeb8a" path="/var/lib/kubelet/pods/e269c7bd-d4b6-4a1d-90ec-e9437abeeb8a/volumes"
Sep 10 15:47:37.165249 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:37.165216    2559 generic.go:356] "Generic (PLEG): container finished" podID="e269c7bd-d4b6-4a1d-90ec-e9437abeeb8a" containerID="0814c92fb9fdb05c1cc6502383564f79398b46dbd41bad4bc1eb12702885488e" exitCode=137
Sep 10 15:47:37.165427 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:37.165257    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7749-cce0e2"
Sep 10 15:47:37.165427 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:37.165321    2559 scope.go:117] "RemoveContainer" containerID="0814c92fb9fdb05c1cc6502383564f79398b46dbd41bad4bc1eb12702885488e"
Sep 10 15:47:37.176278 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:37.174458    2559 scope.go:117] "RemoveContainer" containerID="0814c92fb9fdb05c1cc6502383564f79398b46dbd41bad4bc1eb12702885488e"
Sep 10 15:47:37.176673 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:47:37.176653    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0814c92fb9fdb05c1cc6502383564f79398b46dbd41bad4bc1eb12702885488e\": container with ID starting with 0814c92fb9fdb05c1cc6502383564f79398b46dbd41bad4bc1eb12702885488e not found: ID does not exist" containerID="0814c92fb9fdb05c1cc6502383564f79398b46dbd41bad4bc1eb12702885488e"
Sep 10 15:47:37.176748 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:37.176681    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0814c92fb9fdb05c1cc6502383564f79398b46dbd41bad4bc1eb12702885488e"} err="failed to get container status \"0814c92fb9fdb05c1cc6502383564f79398b46dbd41bad4bc1eb12702885488e\": rpc error: code = NotFound desc = could not find container \"0814c92fb9fdb05c1cc6502383564f79398b46dbd41bad4bc1eb12702885488e\": container with ID starting with 0814c92fb9fdb05c1cc6502383564f79398b46dbd41bad4bc1eb12702885488e not found: ID does not exist"
Sep 10 15:47:39.402324 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:39.402287    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bs7hd"]
Sep 10 15:47:42.039699 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:42.039667    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-c7257b/auth-test-model-c7257b-kserve-7dcfccff79-zxdpt"
Sep 10 15:47:42.040202 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:42.039990    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-c7257b/auth-test-model-c7257b-kserve-7dcfccff79-zxdpt"
Sep 10 15:47:42.053174 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:42.053146    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-c7257b/auth-test-model-c7257b-kserve-7dcfccff79-zxdpt"
Sep 10 15:47:42.207556 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:42.207524    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-c7257b/auth-test-model-c7257b-kserve-7dcfccff79-zxdpt"
Sep 10 15:47:47.020159 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:47.020125    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-81f538c6-7969557d6f-b9k4h"]
Sep 10 15:47:47.020741 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:47.020724    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e269c7bd-d4b6-4a1d-90ec-e9437abeeb8a" containerName="test-curl-7749-cce0e2"
Sep 10 15:47:47.020803 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:47.020744    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e269c7bd-d4b6-4a1d-90ec-e9437abeeb8a" containerName="test-curl-7749-cce0e2"
Sep 10 15:47:47.020859 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:47.020848    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="e269c7bd-d4b6-4a1d-90ec-e9437abeeb8a" containerName="test-curl-7749-cce0e2"
Sep 10 15:47:47.025414 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:47.025391    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-81f538c6-7969557d6f-b9k4h"
Sep 10 15:47:47.030687 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:47.030667    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-81f538c6-serving-cert\""
Sep 10 15:47:47.047661 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:47.047631    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-81f538c6-7969557d6f-b9k4h"]
Sep 10 15:47:47.120996 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:47.120957    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7df4d38-c459-48d6-a4d5-567ac624fd1a-metrics-certs\") pod \"maas-api-e2e-mt-81f538c6-7969557d6f-b9k4h\" (UID: \"c7df4d38-c459-48d6-a4d5-567ac624fd1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-81f538c6-7969557d6f-b9k4h"
Sep 10 15:47:47.121205 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:47.121021    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7df4d38-c459-48d6-a4d5-567ac624fd1a-maas-api-tls\") pod \"maas-api-e2e-mt-81f538c6-7969557d6f-b9k4h\" (UID: \"c7df4d38-c459-48d6-a4d5-567ac624fd1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-81f538c6-7969557d6f-b9k4h"
Sep 10 15:47:47.121205 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:47.121048    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nb59n\" (UniqueName: \"kubernetes.io/projected/c7df4d38-c459-48d6-a4d5-567ac624fd1a-kube-api-access-nb59n\") pod \"maas-api-e2e-mt-81f538c6-7969557d6f-b9k4h\" (UID: \"c7df4d38-c459-48d6-a4d5-567ac624fd1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-81f538c6-7969557d6f-b9k4h"
Sep 10 15:47:47.121205 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:47.121070    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7df4d38-c459-48d6-a4d5-567ac624fd1a-openshift-service-ca\") pod \"maas-api-e2e-mt-81f538c6-7969557d6f-b9k4h\" (UID: \"c7df4d38-c459-48d6-a4d5-567ac624fd1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-81f538c6-7969557d6f-b9k4h"
Sep 10 15:47:47.222157 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:47.222127    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7df4d38-c459-48d6-a4d5-567ac624fd1a-metrics-certs\") pod \"maas-api-e2e-mt-81f538c6-7969557d6f-b9k4h\" (UID: \"c7df4d38-c459-48d6-a4d5-567ac624fd1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-81f538c6-7969557d6f-b9k4h"
Sep 10 15:47:47.222382 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:47.222173    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7df4d38-c459-48d6-a4d5-567ac624fd1a-maas-api-tls\") pod \"maas-api-e2e-mt-81f538c6-7969557d6f-b9k4h\" (UID: \"c7df4d38-c459-48d6-a4d5-567ac624fd1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-81f538c6-7969557d6f-b9k4h"
Sep 10 15:47:47.222382 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:47.222191    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nb59n\" (UniqueName: \"kubernetes.io/projected/c7df4d38-c459-48d6-a4d5-567ac624fd1a-kube-api-access-nb59n\") pod \"maas-api-e2e-mt-81f538c6-7969557d6f-b9k4h\" (UID: \"c7df4d38-c459-48d6-a4d5-567ac624fd1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-81f538c6-7969557d6f-b9k4h"
Sep 10 15:47:47.222382 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:47.222209    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7df4d38-c459-48d6-a4d5-567ac624fd1a-openshift-service-ca\") pod \"maas-api-e2e-mt-81f538c6-7969557d6f-b9k4h\" (UID: \"c7df4d38-c459-48d6-a4d5-567ac624fd1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-81f538c6-7969557d6f-b9k4h"
Sep 10 15:47:47.222382 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:47:47.222292    2559 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-81f538c6-serving-cert: secret "maas-api-e2e-mt-81f538c6-serving-cert" not found
Sep 10 15:47:47.222382 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:47:47.222350    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7df4d38-c459-48d6-a4d5-567ac624fd1a-maas-api-tls podName:c7df4d38-c459-48d6-a4d5-567ac624fd1a nodeName:}" failed. No retries permitted until 2026-09-10 15:47:47.722334032 +0000 UTC m=+1243.464299113 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c7df4d38-c459-48d6-a4d5-567ac624fd1a-maas-api-tls") pod "maas-api-e2e-mt-81f538c6-7969557d6f-b9k4h" (UID: "c7df4d38-c459-48d6-a4d5-567ac624fd1a") : secret "maas-api-e2e-mt-81f538c6-serving-cert" not found
Sep 10 15:47:47.222894 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:47.222877    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7df4d38-c459-48d6-a4d5-567ac624fd1a-openshift-service-ca\") pod \"maas-api-e2e-mt-81f538c6-7969557d6f-b9k4h\" (UID: \"c7df4d38-c459-48d6-a4d5-567ac624fd1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-81f538c6-7969557d6f-b9k4h"
Sep 10 15:47:47.224669 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:47.224647    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7df4d38-c459-48d6-a4d5-567ac624fd1a-metrics-certs\") pod \"maas-api-e2e-mt-81f538c6-7969557d6f-b9k4h\" (UID: \"c7df4d38-c459-48d6-a4d5-567ac624fd1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-81f538c6-7969557d6f-b9k4h"
Sep 10 15:47:47.245644 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:47.245610    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nb59n\" (UniqueName: \"kubernetes.io/projected/c7df4d38-c459-48d6-a4d5-567ac624fd1a-kube-api-access-nb59n\") pod \"maas-api-e2e-mt-81f538c6-7969557d6f-b9k4h\" (UID: \"c7df4d38-c459-48d6-a4d5-567ac624fd1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-81f538c6-7969557d6f-b9k4h"
Sep 10 15:47:47.726957 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:47.726917    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7df4d38-c459-48d6-a4d5-567ac624fd1a-maas-api-tls\") pod \"maas-api-e2e-mt-81f538c6-7969557d6f-b9k4h\" (UID: \"c7df4d38-c459-48d6-a4d5-567ac624fd1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-81f538c6-7969557d6f-b9k4h"
Sep 10 15:47:47.729311 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:47.729281    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7df4d38-c459-48d6-a4d5-567ac624fd1a-maas-api-tls\") pod \"maas-api-e2e-mt-81f538c6-7969557d6f-b9k4h\" (UID: \"c7df4d38-c459-48d6-a4d5-567ac624fd1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-81f538c6-7969557d6f-b9k4h"
Sep 10 15:47:47.936268 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:47.936235    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-81f538c6-7969557d6f-b9k4h"
Sep 10 15:47:48.218742 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:48.218715    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-81f538c6-7969557d6f-b9k4h"]
Sep 10 15:47:48.219931 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:47:48.219905    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7df4d38_c459_48d6_a4d5_567ac624fd1a.slice/crio-1b07472eff02af6704e754f2c2ea0d3f0c9579784fdb304703702c766e7ee7b1 WatchSource:0}: Error finding container 1b07472eff02af6704e754f2c2ea0d3f0c9579784fdb304703702c766e7ee7b1: Status 404 returned error can't find the container with id 1b07472eff02af6704e754f2c2ea0d3f0c9579784fdb304703702c766e7ee7b1
Sep 10 15:47:48.229637 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:48.229606    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-81f538c6-7969557d6f-b9k4h" event={"ID":"c7df4d38-c459-48d6-a4d5-567ac624fd1a","Type":"ContainerStarted","Data":"1b07472eff02af6704e754f2c2ea0d3f0c9579784fdb304703702c766e7ee7b1"}
Sep 10 15:47:49.237103 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:49.237067    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-81f538c6-7969557d6f-b9k4h" event={"ID":"c7df4d38-c459-48d6-a4d5-567ac624fd1a","Type":"ContainerStarted","Data":"034d305df44aa8c1f56b66a6a14d1c1838f48a8b6c2196fe9e753ac7ce9850f4"}
Sep 10 15:47:49.237518 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:49.237127    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-81f538c6-7969557d6f-b9k4h"
Sep 10 15:47:49.257073 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:49.257018    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-81f538c6-7969557d6f-b9k4h" podStartSLOduration=2.623065521 podStartE2EDuration="3.257001755s" totalImagesPullingTime="633.936234ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:47:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:47:49.255639136 +0000 UTC m=+1244.997604240" watchObservedRunningTime="2026-09-10 15:47:49.257001755 +0000 UTC m=+1244.998966858"
Sep 10 15:47:53.207953 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:53.207917    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bs7hd"]
Sep 10 15:47:55.248606 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:55.248565    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-81f538c6-7969557d6f-b9k4h"
Sep 10 15:47:59.103805 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:59.103724    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-3650b0/auth-test-model-3650b0-kserve-75fc488795-psgzt"]
Sep 10 15:47:59.107398 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:59.107374    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-3650b0/auth-test-model-3650b0-kserve-75fc488795-psgzt"
Sep 10 15:47:59.110368 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:59.110338    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-3650b0\"/\"kube-root-ca.crt\""
Sep 10 15:47:59.110510 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:59.110349    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-3650b0\"/\"openshift-service-ca.crt\""
Sep 10 15:47:59.111470 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:59.111448    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-3650b0\"/\"default-dockercfg-jk265\""
Sep 10 15:47:59.111611 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:59.111522    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-3650b0\"/\"auth-test-model-3650b0-kserve-self-signed-certs\""
Sep 10 15:47:59.117706 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:59.117684    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3650b0/auth-test-model-3650b0-kserve-75fc488795-psgzt"]
Sep 10 15:47:59.245962 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:59.245922    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0358a9e7-fc9e-47ab-bde3-932fb43124f5-dshm\") pod \"auth-test-model-3650b0-kserve-75fc488795-psgzt\" (UID: \"0358a9e7-fc9e-47ab-bde3-932fb43124f5\") " pod="ai-tenant-e2e-shared-b-w3-3650b0/auth-test-model-3650b0-kserve-75fc488795-psgzt"
Sep 10 15:47:59.246137 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:59.245969    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0358a9e7-fc9e-47ab-bde3-932fb43124f5-tls-certs\") pod \"auth-test-model-3650b0-kserve-75fc488795-psgzt\" (UID: \"0358a9e7-fc9e-47ab-bde3-932fb43124f5\") " pod="ai-tenant-e2e-shared-b-w3-3650b0/auth-test-model-3650b0-kserve-75fc488795-psgzt"
Sep 10 15:47:59.246137 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:59.246105    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0358a9e7-fc9e-47ab-bde3-932fb43124f5-home\") pod \"auth-test-model-3650b0-kserve-75fc488795-psgzt\" (UID: \"0358a9e7-fc9e-47ab-bde3-932fb43124f5\") " pod="ai-tenant-e2e-shared-b-w3-3650b0/auth-test-model-3650b0-kserve-75fc488795-psgzt"
Sep 10 15:47:59.246225 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:59.246138    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0358a9e7-fc9e-47ab-bde3-932fb43124f5-tmp-dir\") pod \"auth-test-model-3650b0-kserve-75fc488795-psgzt\" (UID: \"0358a9e7-fc9e-47ab-bde3-932fb43124f5\") " pod="ai-tenant-e2e-shared-b-w3-3650b0/auth-test-model-3650b0-kserve-75fc488795-psgzt"
Sep 10 15:47:59.246225 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:59.246213    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0358a9e7-fc9e-47ab-bde3-932fb43124f5-model-cache\") pod \"auth-test-model-3650b0-kserve-75fc488795-psgzt\" (UID: \"0358a9e7-fc9e-47ab-bde3-932fb43124f5\") " pod="ai-tenant-e2e-shared-b-w3-3650b0/auth-test-model-3650b0-kserve-75fc488795-psgzt"
Sep 10 15:47:59.246311 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:59.246266    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fpmr\" (UniqueName: \"kubernetes.io/projected/0358a9e7-fc9e-47ab-bde3-932fb43124f5-kube-api-access-4fpmr\") pod \"auth-test-model-3650b0-kserve-75fc488795-psgzt\" (UID: \"0358a9e7-fc9e-47ab-bde3-932fb43124f5\") " pod="ai-tenant-e2e-shared-b-w3-3650b0/auth-test-model-3650b0-kserve-75fc488795-psgzt"
Sep 10 15:47:59.347805 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:59.347762    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fpmr\" (UniqueName: \"kubernetes.io/projected/0358a9e7-fc9e-47ab-bde3-932fb43124f5-kube-api-access-4fpmr\") pod \"auth-test-model-3650b0-kserve-75fc488795-psgzt\" (UID: \"0358a9e7-fc9e-47ab-bde3-932fb43124f5\") " pod="ai-tenant-e2e-shared-b-w3-3650b0/auth-test-model-3650b0-kserve-75fc488795-psgzt"
Sep 10 15:47:59.348003 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:59.347834    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0358a9e7-fc9e-47ab-bde3-932fb43124f5-dshm\") pod \"auth-test-model-3650b0-kserve-75fc488795-psgzt\" (UID: \"0358a9e7-fc9e-47ab-bde3-932fb43124f5\") " pod="ai-tenant-e2e-shared-b-w3-3650b0/auth-test-model-3650b0-kserve-75fc488795-psgzt"
Sep 10 15:47:59.348003 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:59.347859    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0358a9e7-fc9e-47ab-bde3-932fb43124f5-tls-certs\") pod \"auth-test-model-3650b0-kserve-75fc488795-psgzt\" (UID: \"0358a9e7-fc9e-47ab-bde3-932fb43124f5\") " pod="ai-tenant-e2e-shared-b-w3-3650b0/auth-test-model-3650b0-kserve-75fc488795-psgzt"
Sep 10 15:47:59.348003 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:59.347934    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0358a9e7-fc9e-47ab-bde3-932fb43124f5-home\") pod \"auth-test-model-3650b0-kserve-75fc488795-psgzt\" (UID: \"0358a9e7-fc9e-47ab-bde3-932fb43124f5\") " pod="ai-tenant-e2e-shared-b-w3-3650b0/auth-test-model-3650b0-kserve-75fc488795-psgzt"
Sep 10 15:47:59.348003 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:59.347964    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0358a9e7-fc9e-47ab-bde3-932fb43124f5-tmp-dir\") pod \"auth-test-model-3650b0-kserve-75fc488795-psgzt\" (UID: \"0358a9e7-fc9e-47ab-bde3-932fb43124f5\") " pod="ai-tenant-e2e-shared-b-w3-3650b0/auth-test-model-3650b0-kserve-75fc488795-psgzt"
Sep 10 15:47:59.348228 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:59.348026    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0358a9e7-fc9e-47ab-bde3-932fb43124f5-model-cache\") pod \"auth-test-model-3650b0-kserve-75fc488795-psgzt\" (UID: \"0358a9e7-fc9e-47ab-bde3-932fb43124f5\") " pod="ai-tenant-e2e-shared-b-w3-3650b0/auth-test-model-3650b0-kserve-75fc488795-psgzt"
Sep 10 15:47:59.348456 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:59.348404    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0358a9e7-fc9e-47ab-bde3-932fb43124f5-home\") pod \"auth-test-model-3650b0-kserve-75fc488795-psgzt\" (UID: \"0358a9e7-fc9e-47ab-bde3-932fb43124f5\") " pod="ai-tenant-e2e-shared-b-w3-3650b0/auth-test-model-3650b0-kserve-75fc488795-psgzt"
Sep 10 15:47:59.348456 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:59.348433    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0358a9e7-fc9e-47ab-bde3-932fb43124f5-model-cache\") pod \"auth-test-model-3650b0-kserve-75fc488795-psgzt\" (UID: \"0358a9e7-fc9e-47ab-bde3-932fb43124f5\") " pod="ai-tenant-e2e-shared-b-w3-3650b0/auth-test-model-3650b0-kserve-75fc488795-psgzt"
Sep 10 15:47:59.348688 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:59.348475    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0358a9e7-fc9e-47ab-bde3-932fb43124f5-tmp-dir\") pod \"auth-test-model-3650b0-kserve-75fc488795-psgzt\" (UID: \"0358a9e7-fc9e-47ab-bde3-932fb43124f5\") " pod="ai-tenant-e2e-shared-b-w3-3650b0/auth-test-model-3650b0-kserve-75fc488795-psgzt"
Sep 10 15:47:59.350261 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:59.350238    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0358a9e7-fc9e-47ab-bde3-932fb43124f5-dshm\") pod \"auth-test-model-3650b0-kserve-75fc488795-psgzt\" (UID: \"0358a9e7-fc9e-47ab-bde3-932fb43124f5\") " pod="ai-tenant-e2e-shared-b-w3-3650b0/auth-test-model-3650b0-kserve-75fc488795-psgzt"
Sep 10 15:47:59.350555 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:59.350537    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0358a9e7-fc9e-47ab-bde3-932fb43124f5-tls-certs\") pod \"auth-test-model-3650b0-kserve-75fc488795-psgzt\" (UID: \"0358a9e7-fc9e-47ab-bde3-932fb43124f5\") " pod="ai-tenant-e2e-shared-b-w3-3650b0/auth-test-model-3650b0-kserve-75fc488795-psgzt"
Sep 10 15:47:59.356936 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:59.356881    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fpmr\" (UniqueName: \"kubernetes.io/projected/0358a9e7-fc9e-47ab-bde3-932fb43124f5-kube-api-access-4fpmr\") pod \"auth-test-model-3650b0-kserve-75fc488795-psgzt\" (UID: \"0358a9e7-fc9e-47ab-bde3-932fb43124f5\") " pod="ai-tenant-e2e-shared-b-w3-3650b0/auth-test-model-3650b0-kserve-75fc488795-psgzt"
Sep 10 15:47:59.417572 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:59.417536    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-3650b0/auth-test-model-3650b0-kserve-75fc488795-psgzt"
Sep 10 15:47:59.695853 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:47:59.695822    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3650b0/auth-test-model-3650b0-kserve-75fc488795-psgzt"]
Sep 10 15:47:59.697482 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:47:59.697455    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0358a9e7_fc9e_47ab_bde3_932fb43124f5.slice/crio-ec01d3ed4dd66481780711e960d063e7ebdedb542c26e6f682c54ad041c82429 WatchSource:0}: Error finding container ec01d3ed4dd66481780711e960d063e7ebdedb542c26e6f682c54ad041c82429: Status 404 returned error can't find the container with id ec01d3ed4dd66481780711e960d063e7ebdedb542c26e6f682c54ad041c82429
Sep 10 15:48:00.317991 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:48:00.317948    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-3650b0/auth-test-model-3650b0-kserve-75fc488795-psgzt" event={"ID":"0358a9e7-fc9e-47ab-bde3-932fb43124f5","Type":"ContainerStarted","Data":"bbe0e779f521783e2e744dd85deb52975a530ccf9da55552e78aa13bdbb8384f"}
Sep 10 15:48:00.318398 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:48:00.318002    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-3650b0/auth-test-model-3650b0-kserve-75fc488795-psgzt" event={"ID":"0358a9e7-fc9e-47ab-bde3-932fb43124f5","Type":"ContainerStarted","Data":"ec01d3ed4dd66481780711e960d063e7ebdedb542c26e6f682c54ad041c82429"}
Sep 10 15:48:00.337349 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:48:00.337291    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-3650b0/auth-test-model-3650b0-kserve-75fc488795-psgzt" podStartSLOduration=1.33727565 podStartE2EDuration="1.33727565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:47:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:48:00.335795061 +0000 UTC m=+1256.077760191" watchObservedRunningTime="2026-09-10 15:48:00.33727565 +0000 UTC m=+1256.079240786"
Sep 10 15:48:06.525737 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:48:06.525688    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bs7hd"]
Sep 10 15:48:09.418532 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:48:09.418498    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-3650b0/auth-test-model-3650b0-kserve-75fc488795-psgzt"
Sep 10 15:48:09.419019 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:48:09.418554    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-3650b0/auth-test-model-3650b0-kserve-75fc488795-psgzt"
Sep 10 15:48:09.430948 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:48:09.430922    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-3650b0/auth-test-model-3650b0-kserve-75fc488795-psgzt"
Sep 10 15:48:10.387081 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:48:10.387054    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-3650b0/auth-test-model-3650b0-kserve-75fc488795-psgzt"
Sep 10 15:48:25.003088 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:48:25.003053    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bs7hd"]
Sep 10 15:48:43.208351 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:48:43.208309    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bs7hd"]
Sep 10 15:49:02.008714 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:49:02.008674    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bs7hd"]
Sep 10 15:49:13.747923 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:49:13.747887    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-81f538c6-7969557d6f-b9k4h"]
Sep 10 15:49:13.748487 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:49:13.748127    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-81f538c6-7969557d6f-b9k4h" podUID="c7df4d38-c459-48d6-a4d5-567ac624fd1a" containerName="maas-api" containerID="cri-o://034d305df44aa8c1f56b66a6a14d1c1838f48a8b6c2196fe9e753ac7ce9850f4" gracePeriod=30
Sep 10 15:49:14.001025 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:49:14.000965    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-81f538c6-7969557d6f-b9k4h"
Sep 10 15:49:14.187553 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:49:14.187522    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nb59n\" (UniqueName: \"kubernetes.io/projected/c7df4d38-c459-48d6-a4d5-567ac624fd1a-kube-api-access-nb59n\") pod \"c7df4d38-c459-48d6-a4d5-567ac624fd1a\" (UID: \"c7df4d38-c459-48d6-a4d5-567ac624fd1a\") "
Sep 10 15:49:14.187760 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:49:14.187572    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7df4d38-c459-48d6-a4d5-567ac624fd1a-openshift-service-ca\") pod \"c7df4d38-c459-48d6-a4d5-567ac624fd1a\" (UID: \"c7df4d38-c459-48d6-a4d5-567ac624fd1a\") "
Sep 10 15:49:14.187760 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:49:14.187638    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7df4d38-c459-48d6-a4d5-567ac624fd1a-maas-api-tls\") pod \"c7df4d38-c459-48d6-a4d5-567ac624fd1a\" (UID: \"c7df4d38-c459-48d6-a4d5-567ac624fd1a\") "
Sep 10 15:49:14.187760 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:49:14.187710    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7df4d38-c459-48d6-a4d5-567ac624fd1a-metrics-certs\") pod \"c7df4d38-c459-48d6-a4d5-567ac624fd1a\" (UID: \"c7df4d38-c459-48d6-a4d5-567ac624fd1a\") "
Sep 10 15:49:14.187999 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:49:14.187975    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7df4d38-c459-48d6-a4d5-567ac624fd1a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c7df4d38-c459-48d6-a4d5-567ac624fd1a" (UID: "c7df4d38-c459-48d6-a4d5-567ac624fd1a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:49:14.189783 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:49:14.189757    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7df4d38-c459-48d6-a4d5-567ac624fd1a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c7df4d38-c459-48d6-a4d5-567ac624fd1a" (UID: "c7df4d38-c459-48d6-a4d5-567ac624fd1a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:49:14.189877 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:49:14.189829    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7df4d38-c459-48d6-a4d5-567ac624fd1a-kube-api-access-nb59n" (OuterVolumeSpecName: "kube-api-access-nb59n") pod "c7df4d38-c459-48d6-a4d5-567ac624fd1a" (UID: "c7df4d38-c459-48d6-a4d5-567ac624fd1a"). InnerVolumeSpecName "kube-api-access-nb59n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:49:14.189877 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:49:14.189849    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7df4d38-c459-48d6-a4d5-567ac624fd1a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c7df4d38-c459-48d6-a4d5-567ac624fd1a" (UID: "c7df4d38-c459-48d6-a4d5-567ac624fd1a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:49:14.289619 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:49:14.289518    2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7df4d38-c459-48d6-a4d5-567ac624fd1a-metrics-certs\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:49:14.289619 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:49:14.289548    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nb59n\" (UniqueName: \"kubernetes.io/projected/c7df4d38-c459-48d6-a4d5-567ac624fd1a-kube-api-access-nb59n\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:49:14.289619 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:49:14.289557    2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7df4d38-c459-48d6-a4d5-567ac624fd1a-openshift-service-ca\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:49:14.289619 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:49:14.289566    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7df4d38-c459-48d6-a4d5-567ac624fd1a-maas-api-tls\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:49:14.752948 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:49:14.752910    2559 generic.go:356] "Generic (PLEG): container finished" podID="c7df4d38-c459-48d6-a4d5-567ac624fd1a" containerID="034d305df44aa8c1f56b66a6a14d1c1838f48a8b6c2196fe9e753ac7ce9850f4" exitCode=0
Sep 10 15:49:14.753403 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:49:14.752959    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-81f538c6-7969557d6f-b9k4h"
Sep 10 15:49:14.753474 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:49:14.753416    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-81f538c6-7969557d6f-b9k4h" event={"ID":"c7df4d38-c459-48d6-a4d5-567ac624fd1a","Type":"ContainerDied","Data":"034d305df44aa8c1f56b66a6a14d1c1838f48a8b6c2196fe9e753ac7ce9850f4"}
Sep 10 15:49:14.753533 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:49:14.753506    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-81f538c6-7969557d6f-b9k4h" event={"ID":"c7df4d38-c459-48d6-a4d5-567ac624fd1a","Type":"ContainerDied","Data":"1b07472eff02af6704e754f2c2ea0d3f0c9579784fdb304703702c766e7ee7b1"}
Sep 10 15:49:14.753616 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:49:14.753552    2559 scope.go:117] "RemoveContainer" containerID="034d305df44aa8c1f56b66a6a14d1c1838f48a8b6c2196fe9e753ac7ce9850f4"
Sep 10 15:49:14.770696 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:49:14.770676    2559 scope.go:117] "RemoveContainer" containerID="034d305df44aa8c1f56b66a6a14d1c1838f48a8b6c2196fe9e753ac7ce9850f4"
Sep 10 15:49:14.770952 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:49:14.770932    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"034d305df44aa8c1f56b66a6a14d1c1838f48a8b6c2196fe9e753ac7ce9850f4\": container with ID starting with 034d305df44aa8c1f56b66a6a14d1c1838f48a8b6c2196fe9e753ac7ce9850f4 not found: ID does not exist" containerID="034d305df44aa8c1f56b66a6a14d1c1838f48a8b6c2196fe9e753ac7ce9850f4"
Sep 10 15:49:14.770991 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:49:14.770969    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"034d305df44aa8c1f56b66a6a14d1c1838f48a8b6c2196fe9e753ac7ce9850f4"} err="failed to get container status \"034d305df44aa8c1f56b66a6a14d1c1838f48a8b6c2196fe9e753ac7ce9850f4\": rpc error: code = NotFound desc = could not find container \"034d305df44aa8c1f56b66a6a14d1c1838f48a8b6c2196fe9e753ac7ce9850f4\": container with ID starting with 034d305df44aa8c1f56b66a6a14d1c1838f48a8b6c2196fe9e753ac7ce9850f4 not found: ID does not exist"
Sep 10 15:49:14.786811 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:49:14.786789    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-81f538c6-7969557d6f-b9k4h"]
Sep 10 15:49:14.786935 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:49:14.786817    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-81f538c6-7969557d6f-b9k4h"]
Sep 10 15:49:16.785357 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:49:16.785315    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c7df4d38-c459-48d6-a4d5-567ac624fd1a" path="/var/lib/kubelet/pods/c7df4d38-c459-48d6-a4d5-567ac624fd1a/volumes"
Sep 10 15:49:18.810890 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:49:18.810859    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bs7hd"]
Sep 10 15:49:36.602931 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:49:36.602843    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bs7hd"]
Sep 10 15:49:52.710820 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:49:52.710785    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bs7hd"]
Sep 10 15:49:57.304305 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:49:57.304266    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-685b4d8546-h2x7r"]
Sep 10 15:50:12.103712 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:12.103672    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bs7hd"]
Sep 10 15:50:17.313720 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:17.313687    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-c7257b/auth-test-model-c7257b-kserve-7dcfccff79-zxdpt"]
Sep 10 15:50:18.294425 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:18.294374    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3650b0/auth-test-model-3650b0-kserve-75fc488795-psgzt"]
Sep 10 15:50:22.324126 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:22.324087    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-685b4d8546-h2x7r" podUID="e2cacf38-f42a-4dcf-b350-2209b839209e" containerName="console" containerID="cri-o://66d50222b2402938a41201d83daaecfa698f1d923ca799515ddf70cbfbc92987" gracePeriod=15
Sep 10 15:50:22.578431 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:22.578361    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-685b4d8546-h2x7r"
Sep 10 15:50:22.626086 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:22.626052    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e2cacf38-f42a-4dcf-b350-2209b839209e-console-config\") pod \"e2cacf38-f42a-4dcf-b350-2209b839209e\" (UID: \"e2cacf38-f42a-4dcf-b350-2209b839209e\") "
Sep 10 15:50:22.626317 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:22.626118    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e2cacf38-f42a-4dcf-b350-2209b839209e-console-oauth-config\") pod \"e2cacf38-f42a-4dcf-b350-2209b839209e\" (UID: \"e2cacf38-f42a-4dcf-b350-2209b839209e\") "
Sep 10 15:50:22.626317 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:22.626151    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fv96c\" (UniqueName: \"kubernetes.io/projected/e2cacf38-f42a-4dcf-b350-2209b839209e-kube-api-access-fv96c\") pod \"e2cacf38-f42a-4dcf-b350-2209b839209e\" (UID: \"e2cacf38-f42a-4dcf-b350-2209b839209e\") "
Sep 10 15:50:22.626317 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:22.626173    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e2cacf38-f42a-4dcf-b350-2209b839209e-oauth-serving-cert\") pod \"e2cacf38-f42a-4dcf-b350-2209b839209e\" (UID: \"e2cacf38-f42a-4dcf-b350-2209b839209e\") "
Sep 10 15:50:22.626317 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:22.626192    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e2cacf38-f42a-4dcf-b350-2209b839209e-console-serving-cert\") pod \"e2cacf38-f42a-4dcf-b350-2209b839209e\" (UID: \"e2cacf38-f42a-4dcf-b350-2209b839209e\") "
Sep 10 15:50:22.626317 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:22.626234    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e2cacf38-f42a-4dcf-b350-2209b839209e-service-ca\") pod \"e2cacf38-f42a-4dcf-b350-2209b839209e\" (UID: \"e2cacf38-f42a-4dcf-b350-2209b839209e\") "
Sep 10 15:50:22.626317 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:22.626264    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2cacf38-f42a-4dcf-b350-2209b839209e-trusted-ca-bundle\") pod \"e2cacf38-f42a-4dcf-b350-2209b839209e\" (UID: \"e2cacf38-f42a-4dcf-b350-2209b839209e\") "
Sep 10 15:50:22.626685 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:22.626543    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2cacf38-f42a-4dcf-b350-2209b839209e-console-config" (OuterVolumeSpecName: "console-config") pod "e2cacf38-f42a-4dcf-b350-2209b839209e" (UID: "e2cacf38-f42a-4dcf-b350-2209b839209e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:50:22.626685 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:22.626656    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2cacf38-f42a-4dcf-b350-2209b839209e-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "e2cacf38-f42a-4dcf-b350-2209b839209e" (UID: "e2cacf38-f42a-4dcf-b350-2209b839209e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:50:22.626848 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:22.626816    2559 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e2cacf38-f42a-4dcf-b350-2209b839209e-console-config\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:50:22.626960 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:22.626937    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2cacf38-f42a-4dcf-b350-2209b839209e-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "e2cacf38-f42a-4dcf-b350-2209b839209e" (UID: "e2cacf38-f42a-4dcf-b350-2209b839209e"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:50:22.627039 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:22.627004    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2cacf38-f42a-4dcf-b350-2209b839209e-service-ca" (OuterVolumeSpecName: "service-ca") pod "e2cacf38-f42a-4dcf-b350-2209b839209e" (UID: "e2cacf38-f42a-4dcf-b350-2209b839209e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:50:22.628387 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:22.628362    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2cacf38-f42a-4dcf-b350-2209b839209e-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "e2cacf38-f42a-4dcf-b350-2209b839209e" (UID: "e2cacf38-f42a-4dcf-b350-2209b839209e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:50:22.628518 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:22.628408    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2cacf38-f42a-4dcf-b350-2209b839209e-kube-api-access-fv96c" (OuterVolumeSpecName: "kube-api-access-fv96c") pod "e2cacf38-f42a-4dcf-b350-2209b839209e" (UID: "e2cacf38-f42a-4dcf-b350-2209b839209e"). InnerVolumeSpecName "kube-api-access-fv96c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:50:22.628637 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:22.628619    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2cacf38-f42a-4dcf-b350-2209b839209e-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "e2cacf38-f42a-4dcf-b350-2209b839209e" (UID: "e2cacf38-f42a-4dcf-b350-2209b839209e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:50:22.727445 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:22.727408    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fv96c\" (UniqueName: \"kubernetes.io/projected/e2cacf38-f42a-4dcf-b350-2209b839209e-kube-api-access-fv96c\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:50:22.727445 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:22.727438    2559 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e2cacf38-f42a-4dcf-b350-2209b839209e-oauth-serving-cert\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:50:22.727445 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:22.727448    2559 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e2cacf38-f42a-4dcf-b350-2209b839209e-console-serving-cert\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:50:22.727726 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:22.727463    2559 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e2cacf38-f42a-4dcf-b350-2209b839209e-service-ca\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:50:22.727726 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:22.727472    2559 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2cacf38-f42a-4dcf-b350-2209b839209e-trusted-ca-bundle\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:50:22.727726 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:22.727480    2559 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e2cacf38-f42a-4dcf-b350-2209b839209e-console-oauth-config\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:50:23.160430 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:23.160396    2559 generic.go:356] "Generic (PLEG): container finished" podID="e2cacf38-f42a-4dcf-b350-2209b839209e" containerID="66d50222b2402938a41201d83daaecfa698f1d923ca799515ddf70cbfbc92987" exitCode=0
Sep 10 15:50:23.160630 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:23.160446    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-685b4d8546-h2x7r"
Sep 10 15:50:23.160630 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:23.160481    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-685b4d8546-h2x7r" event={"ID":"e2cacf38-f42a-4dcf-b350-2209b839209e","Type":"ContainerDied","Data":"66d50222b2402938a41201d83daaecfa698f1d923ca799515ddf70cbfbc92987"}
Sep 10 15:50:23.160630 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:23.160534    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-685b4d8546-h2x7r" event={"ID":"e2cacf38-f42a-4dcf-b350-2209b839209e","Type":"ContainerDied","Data":"381316f4087c552558b17c321748362ff5009f3c1099542f3a4af3a5b15ed4b5"}
Sep 10 15:50:23.160630 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:23.160559    2559 scope.go:117] "RemoveContainer" containerID="66d50222b2402938a41201d83daaecfa698f1d923ca799515ddf70cbfbc92987"
Sep 10 15:50:23.169549 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:23.169514    2559 scope.go:117] "RemoveContainer" containerID="66d50222b2402938a41201d83daaecfa698f1d923ca799515ddf70cbfbc92987"
Sep 10 15:50:23.169812 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:50:23.169793    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66d50222b2402938a41201d83daaecfa698f1d923ca799515ddf70cbfbc92987\": container with ID starting with 66d50222b2402938a41201d83daaecfa698f1d923ca799515ddf70cbfbc92987 not found: ID does not exist" containerID="66d50222b2402938a41201d83daaecfa698f1d923ca799515ddf70cbfbc92987"
Sep 10 15:50:23.169896 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:23.169818    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66d50222b2402938a41201d83daaecfa698f1d923ca799515ddf70cbfbc92987"} err="failed to get container status \"66d50222b2402938a41201d83daaecfa698f1d923ca799515ddf70cbfbc92987\": rpc error: code = NotFound desc = could not find container \"66d50222b2402938a41201d83daaecfa698f1d923ca799515ddf70cbfbc92987\": container with ID starting with 66d50222b2402938a41201d83daaecfa698f1d923ca799515ddf70cbfbc92987 not found: ID does not exist"
Sep 10 15:50:23.180955 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:23.180931    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-685b4d8546-h2x7r"]
Sep 10 15:50:23.185563 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:23.185540    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-685b4d8546-h2x7r"]
Sep 10 15:50:24.786747 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:24.786714    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2cacf38-f42a-4dcf-b350-2209b839209e" path="/var/lib/kubelet/pods/e2cacf38-f42a-4dcf-b350-2209b839209e/volumes"
Sep 10 15:50:28.004413 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:28.004374    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bs7hd"]
Sep 10 15:50:32.333793 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:32.333752    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-c7257b/auth-test-model-c7257b-kserve-7dcfccff79-zxdpt" podUID="a3d20bc8-5feb-4a2f-bfef-6f08c06d5114" containerName="main" containerID="cri-o://0992cf65eae36ec5a379899d2bfc5439fc5e2b6ffa9ff773095346fecfd5c9a3" gracePeriod=45
Sep 10 15:50:32.586564 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:32.586503    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-c7257b/auth-test-model-c7257b-kserve-7dcfccff79-zxdpt"
Sep 10 15:50:32.731036 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:32.731001    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3d20bc8-5feb-4a2f-bfef-6f08c06d5114-dshm\") pod \"a3d20bc8-5feb-4a2f-bfef-6f08c06d5114\" (UID: \"a3d20bc8-5feb-4a2f-bfef-6f08c06d5114\") "
Sep 10 15:50:32.731233 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:32.731055    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4t6wc\" (UniqueName: \"kubernetes.io/projected/a3d20bc8-5feb-4a2f-bfef-6f08c06d5114-kube-api-access-4t6wc\") pod \"a3d20bc8-5feb-4a2f-bfef-6f08c06d5114\" (UID: \"a3d20bc8-5feb-4a2f-bfef-6f08c06d5114\") "
Sep 10 15:50:32.731233 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:32.731085    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3d20bc8-5feb-4a2f-bfef-6f08c06d5114-model-cache\") pod \"a3d20bc8-5feb-4a2f-bfef-6f08c06d5114\" (UID: \"a3d20bc8-5feb-4a2f-bfef-6f08c06d5114\") "
Sep 10 15:50:32.731233 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:32.731190    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3d20bc8-5feb-4a2f-bfef-6f08c06d5114-home\") pod \"a3d20bc8-5feb-4a2f-bfef-6f08c06d5114\" (UID: \"a3d20bc8-5feb-4a2f-bfef-6f08c06d5114\") "
Sep 10 15:50:32.731406 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:32.731243    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3d20bc8-5feb-4a2f-bfef-6f08c06d5114-tmp-dir\") pod \"a3d20bc8-5feb-4a2f-bfef-6f08c06d5114\" (UID: \"a3d20bc8-5feb-4a2f-bfef-6f08c06d5114\") "
Sep 10 15:50:32.731406 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:32.731273    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3d20bc8-5feb-4a2f-bfef-6f08c06d5114-tls-certs\") pod \"a3d20bc8-5feb-4a2f-bfef-6f08c06d5114\" (UID: \"a3d20bc8-5feb-4a2f-bfef-6f08c06d5114\") "
Sep 10 15:50:32.731406 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:32.731353    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3d20bc8-5feb-4a2f-bfef-6f08c06d5114-model-cache" (OuterVolumeSpecName: "model-cache") pod "a3d20bc8-5feb-4a2f-bfef-6f08c06d5114" (UID: "a3d20bc8-5feb-4a2f-bfef-6f08c06d5114"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:50:32.731597 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:32.731419    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3d20bc8-5feb-4a2f-bfef-6f08c06d5114-home" (OuterVolumeSpecName: "home") pod "a3d20bc8-5feb-4a2f-bfef-6f08c06d5114" (UID: "a3d20bc8-5feb-4a2f-bfef-6f08c06d5114"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:50:32.731597 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:32.731510    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3d20bc8-5feb-4a2f-bfef-6f08c06d5114-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a3d20bc8-5feb-4a2f-bfef-6f08c06d5114" (UID: "a3d20bc8-5feb-4a2f-bfef-6f08c06d5114"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:50:32.731708 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:32.731669    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3d20bc8-5feb-4a2f-bfef-6f08c06d5114-home\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:50:32.731708 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:32.731683    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3d20bc8-5feb-4a2f-bfef-6f08c06d5114-tmp-dir\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:50:32.731708 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:32.731692    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3d20bc8-5feb-4a2f-bfef-6f08c06d5114-model-cache\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:50:32.733346 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:32.733326    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3d20bc8-5feb-4a2f-bfef-6f08c06d5114-dshm" (OuterVolumeSpecName: "dshm") pod "a3d20bc8-5feb-4a2f-bfef-6f08c06d5114" (UID: "a3d20bc8-5feb-4a2f-bfef-6f08c06d5114"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:50:32.733346 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:32.733341    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3d20bc8-5feb-4a2f-bfef-6f08c06d5114-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a3d20bc8-5feb-4a2f-bfef-6f08c06d5114" (UID: "a3d20bc8-5feb-4a2f-bfef-6f08c06d5114"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:50:32.733497 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:32.733438    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3d20bc8-5feb-4a2f-bfef-6f08c06d5114-kube-api-access-4t6wc" (OuterVolumeSpecName: "kube-api-access-4t6wc") pod "a3d20bc8-5feb-4a2f-bfef-6f08c06d5114" (UID: "a3d20bc8-5feb-4a2f-bfef-6f08c06d5114"). InnerVolumeSpecName "kube-api-access-4t6wc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:50:32.832793 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:32.832767    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3d20bc8-5feb-4a2f-bfef-6f08c06d5114-tls-certs\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:50:32.832793 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:32.832792    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3d20bc8-5feb-4a2f-bfef-6f08c06d5114-dshm\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:50:32.833035 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:32.832801    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4t6wc\" (UniqueName: \"kubernetes.io/projected/a3d20bc8-5feb-4a2f-bfef-6f08c06d5114-kube-api-access-4t6wc\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:50:33.215412 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:33.215377    2559 generic.go:356] "Generic (PLEG): container finished" podID="a3d20bc8-5feb-4a2f-bfef-6f08c06d5114" containerID="0992cf65eae36ec5a379899d2bfc5439fc5e2b6ffa9ff773095346fecfd5c9a3" exitCode=0
Sep 10 15:50:33.215652 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:33.215418    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-c7257b/auth-test-model-c7257b-kserve-7dcfccff79-zxdpt"
Sep 10 15:50:33.215652 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:33.215463    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-c7257b/auth-test-model-c7257b-kserve-7dcfccff79-zxdpt" event={"ID":"a3d20bc8-5feb-4a2f-bfef-6f08c06d5114","Type":"ContainerDied","Data":"0992cf65eae36ec5a379899d2bfc5439fc5e2b6ffa9ff773095346fecfd5c9a3"}
Sep 10 15:50:33.215652 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:33.215505    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-c7257b/auth-test-model-c7257b-kserve-7dcfccff79-zxdpt" event={"ID":"a3d20bc8-5feb-4a2f-bfef-6f08c06d5114","Type":"ContainerDied","Data":"1f8157a37194bdddaf5aea5f482c73aa25ffcdea4c749abc3935b8bf23289046"}
Sep 10 15:50:33.215652 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:33.215525    2559 scope.go:117] "RemoveContainer" containerID="0992cf65eae36ec5a379899d2bfc5439fc5e2b6ffa9ff773095346fecfd5c9a3"
Sep 10 15:50:33.224798 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:33.224778    2559 scope.go:117] "RemoveContainer" containerID="0992cf65eae36ec5a379899d2bfc5439fc5e2b6ffa9ff773095346fecfd5c9a3"
Sep 10 15:50:33.225096 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:50:33.225077    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0992cf65eae36ec5a379899d2bfc5439fc5e2b6ffa9ff773095346fecfd5c9a3\": container with ID starting with 0992cf65eae36ec5a379899d2bfc5439fc5e2b6ffa9ff773095346fecfd5c9a3 not found: ID does not exist" containerID="0992cf65eae36ec5a379899d2bfc5439fc5e2b6ffa9ff773095346fecfd5c9a3"
Sep 10 15:50:33.225141 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:33.225104    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0992cf65eae36ec5a379899d2bfc5439fc5e2b6ffa9ff773095346fecfd5c9a3"} err="failed to get container status \"0992cf65eae36ec5a379899d2bfc5439fc5e2b6ffa9ff773095346fecfd5c9a3\": rpc error: code = NotFound desc = could not find container \"0992cf65eae36ec5a379899d2bfc5439fc5e2b6ffa9ff773095346fecfd5c9a3\": container with ID starting with 0992cf65eae36ec5a379899d2bfc5439fc5e2b6ffa9ff773095346fecfd5c9a3 not found: ID does not exist"
Sep 10 15:50:33.252435 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:33.252403    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-c7257b/auth-test-model-c7257b-kserve-7dcfccff79-zxdpt"]
Sep 10 15:50:33.256173 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:33.256149    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-c7257b/auth-test-model-c7257b-kserve-7dcfccff79-zxdpt"]
Sep 10 15:50:33.320573 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:33.320538    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-3650b0/auth-test-model-3650b0-kserve-75fc488795-psgzt" podUID="0358a9e7-fc9e-47ab-bde3-932fb43124f5" containerName="main" containerID="cri-o://bbe0e779f521783e2e744dd85deb52975a530ccf9da55552e78aa13bdbb8384f" gracePeriod=45
Sep 10 15:50:33.573993 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:33.573970    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-3650b0/auth-test-model-3650b0-kserve-75fc488795-psgzt"
Sep 10 15:50:33.642421 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:33.642374    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0358a9e7-fc9e-47ab-bde3-932fb43124f5-model-cache\") pod \"0358a9e7-fc9e-47ab-bde3-932fb43124f5\" (UID: \"0358a9e7-fc9e-47ab-bde3-932fb43124f5\") "
Sep 10 15:50:33.642640 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:33.642434    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4fpmr\" (UniqueName: \"kubernetes.io/projected/0358a9e7-fc9e-47ab-bde3-932fb43124f5-kube-api-access-4fpmr\") pod \"0358a9e7-fc9e-47ab-bde3-932fb43124f5\" (UID: \"0358a9e7-fc9e-47ab-bde3-932fb43124f5\") "
Sep 10 15:50:33.642640 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:33.642468    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0358a9e7-fc9e-47ab-bde3-932fb43124f5-dshm\") pod \"0358a9e7-fc9e-47ab-bde3-932fb43124f5\" (UID: \"0358a9e7-fc9e-47ab-bde3-932fb43124f5\") "
Sep 10 15:50:33.642640 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:33.642503    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0358a9e7-fc9e-47ab-bde3-932fb43124f5-tls-certs\") pod \"0358a9e7-fc9e-47ab-bde3-932fb43124f5\" (UID: \"0358a9e7-fc9e-47ab-bde3-932fb43124f5\") "
Sep 10 15:50:33.642640 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:33.642592    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0358a9e7-fc9e-47ab-bde3-932fb43124f5-tmp-dir\") pod \"0358a9e7-fc9e-47ab-bde3-932fb43124f5\" (UID: \"0358a9e7-fc9e-47ab-bde3-932fb43124f5\") "
Sep 10 15:50:33.642640 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:33.642626    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0358a9e7-fc9e-47ab-bde3-932fb43124f5-home\") pod \"0358a9e7-fc9e-47ab-bde3-932fb43124f5\" (UID: \"0358a9e7-fc9e-47ab-bde3-932fb43124f5\") "
Sep 10 15:50:33.642927 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:33.642660    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0358a9e7-fc9e-47ab-bde3-932fb43124f5-model-cache" (OuterVolumeSpecName: "model-cache") pod "0358a9e7-fc9e-47ab-bde3-932fb43124f5" (UID: "0358a9e7-fc9e-47ab-bde3-932fb43124f5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:50:33.642927 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:33.642844    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0358a9e7-fc9e-47ab-bde3-932fb43124f5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0358a9e7-fc9e-47ab-bde3-932fb43124f5" (UID: "0358a9e7-fc9e-47ab-bde3-932fb43124f5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:50:33.643034 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:33.643004    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0358a9e7-fc9e-47ab-bde3-932fb43124f5-tmp-dir\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:50:33.643034 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:33.643024    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0358a9e7-fc9e-47ab-bde3-932fb43124f5-model-cache\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:50:33.643139 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:33.643021    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0358a9e7-fc9e-47ab-bde3-932fb43124f5-home" (OuterVolumeSpecName: "home") pod "0358a9e7-fc9e-47ab-bde3-932fb43124f5" (UID: "0358a9e7-fc9e-47ab-bde3-932fb43124f5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:50:33.645362 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:33.645336    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0358a9e7-fc9e-47ab-bde3-932fb43124f5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0358a9e7-fc9e-47ab-bde3-932fb43124f5" (UID: "0358a9e7-fc9e-47ab-bde3-932fb43124f5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:50:33.645362 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:33.645336    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0358a9e7-fc9e-47ab-bde3-932fb43124f5-dshm" (OuterVolumeSpecName: "dshm") pod "0358a9e7-fc9e-47ab-bde3-932fb43124f5" (UID: "0358a9e7-fc9e-47ab-bde3-932fb43124f5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:50:33.645565 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:33.645471    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0358a9e7-fc9e-47ab-bde3-932fb43124f5-kube-api-access-4fpmr" (OuterVolumeSpecName: "kube-api-access-4fpmr") pod "0358a9e7-fc9e-47ab-bde3-932fb43124f5" (UID: "0358a9e7-fc9e-47ab-bde3-932fb43124f5"). InnerVolumeSpecName "kube-api-access-4fpmr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:50:33.743893 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:33.743809    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0358a9e7-fc9e-47ab-bde3-932fb43124f5-tls-certs\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:50:33.743893 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:33.743835    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0358a9e7-fc9e-47ab-bde3-932fb43124f5-home\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:50:33.743893 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:33.743845    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fpmr\" (UniqueName: \"kubernetes.io/projected/0358a9e7-fc9e-47ab-bde3-932fb43124f5-kube-api-access-4fpmr\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:50:33.743893 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:33.743853    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0358a9e7-fc9e-47ab-bde3-932fb43124f5-dshm\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:50:33.831540 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:33.831507    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-c7257b/sub-test-model-c7257b-kserve-86db97955f-2x784"]
Sep 10 15:50:33.831927 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:33.831912    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2cacf38-f42a-4dcf-b350-2209b839209e" containerName="console"
Sep 10 15:50:33.831975 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:33.831929    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2cacf38-f42a-4dcf-b350-2209b839209e" containerName="console"
Sep 10 15:50:33.831975 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:33.831943    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3d20bc8-5feb-4a2f-bfef-6f08c06d5114" containerName="main"
Sep 10 15:50:33.831975 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:33.831949    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3d20bc8-5feb-4a2f-bfef-6f08c06d5114" containerName="main"
Sep 10 15:50:33.831975 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:33.831959    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0358a9e7-fc9e-47ab-bde3-932fb43124f5" containerName="main"
Sep 10 15:50:33.831975 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:33.831964    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="0358a9e7-fc9e-47ab-bde3-932fb43124f5" containerName="main"
Sep 10 15:50:33.832154 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:33.831977    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7df4d38-c459-48d6-a4d5-567ac624fd1a" containerName="maas-api"
Sep 10 15:50:33.832154 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:33.831982    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7df4d38-c459-48d6-a4d5-567ac624fd1a" containerName="maas-api"
Sep 10 15:50:33.832154 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:33.832059    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7df4d38-c459-48d6-a4d5-567ac624fd1a" containerName="maas-api"
Sep 10 15:50:33.832154 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:33.832067    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3d20bc8-5feb-4a2f-bfef-6f08c06d5114" containerName="main"
Sep 10 15:50:33.832154 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:33.832074    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="0358a9e7-fc9e-47ab-bde3-932fb43124f5" containerName="main"
Sep 10 15:50:33.832154 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:33.832081    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2cacf38-f42a-4dcf-b350-2209b839209e" containerName="console"
Sep 10 15:50:33.835257 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:33.835242    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-c7257b/sub-test-model-c7257b-kserve-86db97955f-2x784"
Sep 10 15:50:33.837946 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:33.837924    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-c7257b\"/\"kube-root-ca.crt\""
Sep 10 15:50:33.838065 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:33.838046    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-c7257b\"/\"openshift-service-ca.crt\""
Sep 10 15:50:33.838120 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:33.838046    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-c7257b\"/\"default-dockercfg-s5vj9\""
Sep 10 15:50:33.838120 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:33.838056    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-c7257b\"/\"sub-test-model-c7257b-kserve-self-signed-certs\""
Sep 10 15:50:33.846557 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:33.846530    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-c7257b/sub-test-model-c7257b-kserve-86db97955f-2x784"]
Sep 10 15:50:33.946306 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:33.946264    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68b779b4-60c6-468e-b6e4-de95dda02bf1-model-cache\") pod \"sub-test-model-c7257b-kserve-86db97955f-2x784\" (UID: \"68b779b4-60c6-468e-b6e4-de95dda02bf1\") " pod="ai-tenant-e2e-shared-a-w3-c7257b/sub-test-model-c7257b-kserve-86db97955f-2x784"
Sep 10 15:50:33.946508 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:33.946319    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68b779b4-60c6-468e-b6e4-de95dda02bf1-home\") pod \"sub-test-model-c7257b-kserve-86db97955f-2x784\" (UID: \"68b779b4-60c6-468e-b6e4-de95dda02bf1\") " pod="ai-tenant-e2e-shared-a-w3-c7257b/sub-test-model-c7257b-kserve-86db97955f-2x784"
Sep 10 15:50:33.946508 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:33.946409    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68b779b4-60c6-468e-b6e4-de95dda02bf1-tmp-dir\") pod \"sub-test-model-c7257b-kserve-86db97955f-2x784\" (UID: \"68b779b4-60c6-468e-b6e4-de95dda02bf1\") " pod="ai-tenant-e2e-shared-a-w3-c7257b/sub-test-model-c7257b-kserve-86db97955f-2x784"
Sep 10 15:50:33.946508 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:33.946430    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68b779b4-60c6-468e-b6e4-de95dda02bf1-tls-certs\") pod \"sub-test-model-c7257b-kserve-86db97955f-2x784\" (UID: \"68b779b4-60c6-468e-b6e4-de95dda02bf1\") " pod="ai-tenant-e2e-shared-a-w3-c7257b/sub-test-model-c7257b-kserve-86db97955f-2x784"
Sep 10 15:50:33.946508 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:33.946467    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ks854\" (UniqueName: \"kubernetes.io/projected/68b779b4-60c6-468e-b6e4-de95dda02bf1-kube-api-access-ks854\") pod \"sub-test-model-c7257b-kserve-86db97955f-2x784\" (UID: \"68b779b4-60c6-468e-b6e4-de95dda02bf1\") " pod="ai-tenant-e2e-shared-a-w3-c7257b/sub-test-model-c7257b-kserve-86db97955f-2x784"
Sep 10 15:50:33.946714 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:33.946551    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68b779b4-60c6-468e-b6e4-de95dda02bf1-dshm\") pod \"sub-test-model-c7257b-kserve-86db97955f-2x784\" (UID: \"68b779b4-60c6-468e-b6e4-de95dda02bf1\") " pod="ai-tenant-e2e-shared-a-w3-c7257b/sub-test-model-c7257b-kserve-86db97955f-2x784"
Sep 10 15:50:34.047511 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:34.047422    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68b779b4-60c6-468e-b6e4-de95dda02bf1-home\") pod \"sub-test-model-c7257b-kserve-86db97955f-2x784\" (UID: \"68b779b4-60c6-468e-b6e4-de95dda02bf1\") " pod="ai-tenant-e2e-shared-a-w3-c7257b/sub-test-model-c7257b-kserve-86db97955f-2x784"
Sep 10 15:50:34.047511 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:34.047474    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68b779b4-60c6-468e-b6e4-de95dda02bf1-tmp-dir\") pod \"sub-test-model-c7257b-kserve-86db97955f-2x784\" (UID: \"68b779b4-60c6-468e-b6e4-de95dda02bf1\") " pod="ai-tenant-e2e-shared-a-w3-c7257b/sub-test-model-c7257b-kserve-86db97955f-2x784"
Sep 10 15:50:34.047763 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:34.047607    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68b779b4-60c6-468e-b6e4-de95dda02bf1-tls-certs\") pod \"sub-test-model-c7257b-kserve-86db97955f-2x784\" (UID: \"68b779b4-60c6-468e-b6e4-de95dda02bf1\") " pod="ai-tenant-e2e-shared-a-w3-c7257b/sub-test-model-c7257b-kserve-86db97955f-2x784"
Sep 10 15:50:34.047763 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:34.047665    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ks854\" (UniqueName: \"kubernetes.io/projected/68b779b4-60c6-468e-b6e4-de95dda02bf1-kube-api-access-ks854\") pod \"sub-test-model-c7257b-kserve-86db97955f-2x784\" (UID: \"68b779b4-60c6-468e-b6e4-de95dda02bf1\") " pod="ai-tenant-e2e-shared-a-w3-c7257b/sub-test-model-c7257b-kserve-86db97955f-2x784"
Sep 10 15:50:34.047763 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:34.047695    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68b779b4-60c6-468e-b6e4-de95dda02bf1-dshm\") pod \"sub-test-model-c7257b-kserve-86db97955f-2x784\" (UID: \"68b779b4-60c6-468e-b6e4-de95dda02bf1\") " pod="ai-tenant-e2e-shared-a-w3-c7257b/sub-test-model-c7257b-kserve-86db97955f-2x784"
Sep 10 15:50:34.047924 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:34.047781    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68b779b4-60c6-468e-b6e4-de95dda02bf1-model-cache\") pod \"sub-test-model-c7257b-kserve-86db97955f-2x784\" (UID: \"68b779b4-60c6-468e-b6e4-de95dda02bf1\") " pod="ai-tenant-e2e-shared-a-w3-c7257b/sub-test-model-c7257b-kserve-86db97955f-2x784"
Sep 10 15:50:34.047924 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:34.047821    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68b779b4-60c6-468e-b6e4-de95dda02bf1-tmp-dir\") pod \"sub-test-model-c7257b-kserve-86db97955f-2x784\" (UID: \"68b779b4-60c6-468e-b6e4-de95dda02bf1\") " pod="ai-tenant-e2e-shared-a-w3-c7257b/sub-test-model-c7257b-kserve-86db97955f-2x784"
Sep 10 15:50:34.047924 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:34.047870    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68b779b4-60c6-468e-b6e4-de95dda02bf1-home\") pod \"sub-test-model-c7257b-kserve-86db97955f-2x784\" (UID: \"68b779b4-60c6-468e-b6e4-de95dda02bf1\") " pod="ai-tenant-e2e-shared-a-w3-c7257b/sub-test-model-c7257b-kserve-86db97955f-2x784"
Sep 10 15:50:34.048147 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:34.048127    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68b779b4-60c6-468e-b6e4-de95dda02bf1-model-cache\") pod \"sub-test-model-c7257b-kserve-86db97955f-2x784\" (UID: \"68b779b4-60c6-468e-b6e4-de95dda02bf1\") " pod="ai-tenant-e2e-shared-a-w3-c7257b/sub-test-model-c7257b-kserve-86db97955f-2x784"
Sep 10 15:50:34.049997 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:34.049973    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68b779b4-60c6-468e-b6e4-de95dda02bf1-dshm\") pod \"sub-test-model-c7257b-kserve-86db97955f-2x784\" (UID: \"68b779b4-60c6-468e-b6e4-de95dda02bf1\") " pod="ai-tenant-e2e-shared-a-w3-c7257b/sub-test-model-c7257b-kserve-86db97955f-2x784"
Sep 10 15:50:34.050174 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:34.050154    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68b779b4-60c6-468e-b6e4-de95dda02bf1-tls-certs\") pod \"sub-test-model-c7257b-kserve-86db97955f-2x784\" (UID: \"68b779b4-60c6-468e-b6e4-de95dda02bf1\") " pod="ai-tenant-e2e-shared-a-w3-c7257b/sub-test-model-c7257b-kserve-86db97955f-2x784"
Sep 10 15:50:34.061018 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:34.060998    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ks854\" (UniqueName: \"kubernetes.io/projected/68b779b4-60c6-468e-b6e4-de95dda02bf1-kube-api-access-ks854\") pod \"sub-test-model-c7257b-kserve-86db97955f-2x784\" (UID: \"68b779b4-60c6-468e-b6e4-de95dda02bf1\") " pod="ai-tenant-e2e-shared-a-w3-c7257b/sub-test-model-c7257b-kserve-86db97955f-2x784"
Sep 10 15:50:34.146236 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:34.146204    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-c7257b/sub-test-model-c7257b-kserve-86db97955f-2x784"
Sep 10 15:50:34.223353 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:34.223326    2559 generic.go:356] "Generic (PLEG): container finished" podID="0358a9e7-fc9e-47ab-bde3-932fb43124f5" containerID="bbe0e779f521783e2e744dd85deb52975a530ccf9da55552e78aa13bdbb8384f" exitCode=0
Sep 10 15:50:34.223529 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:34.223373    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-3650b0/auth-test-model-3650b0-kserve-75fc488795-psgzt"
Sep 10 15:50:34.223529 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:34.223410    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-3650b0/auth-test-model-3650b0-kserve-75fc488795-psgzt" event={"ID":"0358a9e7-fc9e-47ab-bde3-932fb43124f5","Type":"ContainerDied","Data":"bbe0e779f521783e2e744dd85deb52975a530ccf9da55552e78aa13bdbb8384f"}
Sep 10 15:50:34.223529 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:34.223456    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-3650b0/auth-test-model-3650b0-kserve-75fc488795-psgzt" event={"ID":"0358a9e7-fc9e-47ab-bde3-932fb43124f5","Type":"ContainerDied","Data":"ec01d3ed4dd66481780711e960d063e7ebdedb542c26e6f682c54ad041c82429"}
Sep 10 15:50:34.223529 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:34.223477    2559 scope.go:117] "RemoveContainer" containerID="bbe0e779f521783e2e744dd85deb52975a530ccf9da55552e78aa13bdbb8384f"
Sep 10 15:50:34.234110 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:34.234091    2559 scope.go:117] "RemoveContainer" containerID="bbe0e779f521783e2e744dd85deb52975a530ccf9da55552e78aa13bdbb8384f"
Sep 10 15:50:34.234389 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:50:34.234370    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbe0e779f521783e2e744dd85deb52975a530ccf9da55552e78aa13bdbb8384f\": container with ID starting with bbe0e779f521783e2e744dd85deb52975a530ccf9da55552e78aa13bdbb8384f not found: ID does not exist" containerID="bbe0e779f521783e2e744dd85deb52975a530ccf9da55552e78aa13bdbb8384f"
Sep 10 15:50:34.234443 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:34.234407    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbe0e779f521783e2e744dd85deb52975a530ccf9da55552e78aa13bdbb8384f"} err="failed to get container status \"bbe0e779f521783e2e744dd85deb52975a530ccf9da55552e78aa13bdbb8384f\": rpc error: code = NotFound desc = could not find container \"bbe0e779f521783e2e744dd85deb52975a530ccf9da55552e78aa13bdbb8384f\": container with ID starting with bbe0e779f521783e2e744dd85deb52975a530ccf9da55552e78aa13bdbb8384f not found: ID does not exist"
Sep 10 15:50:34.251747 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:34.251717    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3650b0/auth-test-model-3650b0-kserve-75fc488795-psgzt"]
Sep 10 15:50:34.257734 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:34.257713    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3650b0/auth-test-model-3650b0-kserve-75fc488795-psgzt"]
Sep 10 15:50:34.419203 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:34.419174    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-c7257b/sub-test-model-c7257b-kserve-86db97955f-2x784"]
Sep 10 15:50:34.420162 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:50:34.420137    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod68b779b4_60c6_468e_b6e4_de95dda02bf1.slice/crio-d65c46126d8641c472e8b43a45a6c8d14dd1680ef1f4ae264323bfdc69eadb3f WatchSource:0}: Error finding container d65c46126d8641c472e8b43a45a6c8d14dd1680ef1f4ae264323bfdc69eadb3f: Status 404 returned error can't find the container with id d65c46126d8641c472e8b43a45a6c8d14dd1680ef1f4ae264323bfdc69eadb3f
Sep 10 15:50:34.787567 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:34.787483    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0358a9e7-fc9e-47ab-bde3-932fb43124f5" path="/var/lib/kubelet/pods/0358a9e7-fc9e-47ab-bde3-932fb43124f5/volumes"
Sep 10 15:50:34.787956 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:34.787891    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3d20bc8-5feb-4a2f-bfef-6f08c06d5114" path="/var/lib/kubelet/pods/a3d20bc8-5feb-4a2f-bfef-6f08c06d5114/volumes"
Sep 10 15:50:35.232200 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:35.232157    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-c7257b/sub-test-model-c7257b-kserve-86db97955f-2x784" event={"ID":"68b779b4-60c6-468e-b6e4-de95dda02bf1","Type":"ContainerStarted","Data":"04dbeba9202f8b2324200c94f1ff28c0ded56891068d6c8cc1ccdc23f64d2c14"}
Sep 10 15:50:35.232200 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:35.232204    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-c7257b/sub-test-model-c7257b-kserve-86db97955f-2x784" event={"ID":"68b779b4-60c6-468e-b6e4-de95dda02bf1","Type":"ContainerStarted","Data":"d65c46126d8641c472e8b43a45a6c8d14dd1680ef1f4ae264323bfdc69eadb3f"}
Sep 10 15:50:35.251202 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:35.251148    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-c7257b/sub-test-model-c7257b-kserve-86db97955f-2x784" podStartSLOduration=2.251130966 podStartE2EDuration="2.251130966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:50:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:50:35.249482381 +0000 UTC m=+1410.991447488" watchObservedRunningTime="2026-09-10 15:50:35.251130966 +0000 UTC m=+1410.993096069"
Sep 10 15:50:38.320922 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:38.320884    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bs7hd"]
Sep 10 15:50:44.146660 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:44.146598    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-c7257b/sub-test-model-c7257b-kserve-86db97955f-2x784"
Sep 10 15:50:44.147093 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:44.146681    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-c7257b/sub-test-model-c7257b-kserve-86db97955f-2x784"
Sep 10 15:50:44.159614 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:44.159572    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-c7257b/sub-test-model-c7257b-kserve-86db97955f-2x784"
Sep 10 15:50:44.295205 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:50:44.295176    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-c7257b/sub-test-model-c7257b-kserve-86db97955f-2x784"
Sep 10 15:51:04.934091 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:51:04.934055    2559 scope.go:117] "RemoveContainer" containerID="e9a16da97b706ec2381a6082ed604e88674367dc2dff31ff6d8ee72c5edc69ea"
Sep 10 15:51:07.692691 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:51:07.692653    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-3650b0/sub-test-model-3650b0-kserve-8bc5c5779-sdxmr"]
Sep 10 15:51:07.697845 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:51:07.697822    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-3650b0/sub-test-model-3650b0-kserve-8bc5c5779-sdxmr"
Sep 10 15:51:07.701670 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:51:07.701643    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-3650b0\"/\"openshift-service-ca.crt\""
Sep 10 15:51:07.701820 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:51:07.701688    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-3650b0\"/\"kube-root-ca.crt\""
Sep 10 15:51:07.701820 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:51:07.701778    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-3650b0\"/\"sub-test-model-3650b0-kserve-self-signed-certs\""
Sep 10 15:51:07.701820 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:51:07.701651    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-3650b0\"/\"default-dockercfg-jk265\""
Sep 10 15:51:07.706811 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:51:07.706784    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3650b0/sub-test-model-3650b0-kserve-8bc5c5779-sdxmr"]
Sep 10 15:51:07.754562 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:51:07.754527    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wznsj\" (UniqueName: \"kubernetes.io/projected/dc6e0fa6-24eb-4547-b2e8-78f7c85f7ea5-kube-api-access-wznsj\") pod \"sub-test-model-3650b0-kserve-8bc5c5779-sdxmr\" (UID: \"dc6e0fa6-24eb-4547-b2e8-78f7c85f7ea5\") " pod="ai-tenant-e2e-shared-b-w3-3650b0/sub-test-model-3650b0-kserve-8bc5c5779-sdxmr"
Sep 10 15:51:07.754562 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:51:07.754570    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc6e0fa6-24eb-4547-b2e8-78f7c85f7ea5-home\") pod \"sub-test-model-3650b0-kserve-8bc5c5779-sdxmr\" (UID: \"dc6e0fa6-24eb-4547-b2e8-78f7c85f7ea5\") " pod="ai-tenant-e2e-shared-b-w3-3650b0/sub-test-model-3650b0-kserve-8bc5c5779-sdxmr"
Sep 10 15:51:07.754847 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:51:07.754667    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc6e0fa6-24eb-4547-b2e8-78f7c85f7ea5-dshm\") pod \"sub-test-model-3650b0-kserve-8bc5c5779-sdxmr\" (UID: \"dc6e0fa6-24eb-4547-b2e8-78f7c85f7ea5\") " pod="ai-tenant-e2e-shared-b-w3-3650b0/sub-test-model-3650b0-kserve-8bc5c5779-sdxmr"
Sep 10 15:51:07.754847 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:51:07.754696    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc6e0fa6-24eb-4547-b2e8-78f7c85f7ea5-tls-certs\") pod \"sub-test-model-3650b0-kserve-8bc5c5779-sdxmr\" (UID: \"dc6e0fa6-24eb-4547-b2e8-78f7c85f7ea5\") " pod="ai-tenant-e2e-shared-b-w3-3650b0/sub-test-model-3650b0-kserve-8bc5c5779-sdxmr"
Sep 10 15:51:07.754847 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:51:07.754719    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc6e0fa6-24eb-4547-b2e8-78f7c85f7ea5-model-cache\") pod \"sub-test-model-3650b0-kserve-8bc5c5779-sdxmr\" (UID: \"dc6e0fa6-24eb-4547-b2e8-78f7c85f7ea5\") " pod="ai-tenant-e2e-shared-b-w3-3650b0/sub-test-model-3650b0-kserve-8bc5c5779-sdxmr"
Sep 10 15:51:07.754847 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:51:07.754794    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc6e0fa6-24eb-4547-b2e8-78f7c85f7ea5-tmp-dir\") pod \"sub-test-model-3650b0-kserve-8bc5c5779-sdxmr\" (UID: \"dc6e0fa6-24eb-4547-b2e8-78f7c85f7ea5\") " pod="ai-tenant-e2e-shared-b-w3-3650b0/sub-test-model-3650b0-kserve-8bc5c5779-sdxmr"
Sep 10 15:51:07.855731 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:51:07.855693    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wznsj\" (UniqueName: \"kubernetes.io/projected/dc6e0fa6-24eb-4547-b2e8-78f7c85f7ea5-kube-api-access-wznsj\") pod \"sub-test-model-3650b0-kserve-8bc5c5779-sdxmr\" (UID: \"dc6e0fa6-24eb-4547-b2e8-78f7c85f7ea5\") " pod="ai-tenant-e2e-shared-b-w3-3650b0/sub-test-model-3650b0-kserve-8bc5c5779-sdxmr"
Sep 10 15:51:07.855731 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:51:07.855734    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc6e0fa6-24eb-4547-b2e8-78f7c85f7ea5-home\") pod \"sub-test-model-3650b0-kserve-8bc5c5779-sdxmr\" (UID: \"dc6e0fa6-24eb-4547-b2e8-78f7c85f7ea5\") " pod="ai-tenant-e2e-shared-b-w3-3650b0/sub-test-model-3650b0-kserve-8bc5c5779-sdxmr"
Sep 10 15:51:07.856007 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:51:07.855753    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc6e0fa6-24eb-4547-b2e8-78f7c85f7ea5-dshm\") pod \"sub-test-model-3650b0-kserve-8bc5c5779-sdxmr\" (UID: \"dc6e0fa6-24eb-4547-b2e8-78f7c85f7ea5\") " pod="ai-tenant-e2e-shared-b-w3-3650b0/sub-test-model-3650b0-kserve-8bc5c5779-sdxmr"
Sep 10 15:51:07.856007 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:51:07.855794    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc6e0fa6-24eb-4547-b2e8-78f7c85f7ea5-tls-certs\") pod \"sub-test-model-3650b0-kserve-8bc5c5779-sdxmr\" (UID: \"dc6e0fa6-24eb-4547-b2e8-78f7c85f7ea5\") " pod="ai-tenant-e2e-shared-b-w3-3650b0/sub-test-model-3650b0-kserve-8bc5c5779-sdxmr"
Sep 10 15:51:07.856007 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:51:07.855813    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc6e0fa6-24eb-4547-b2e8-78f7c85f7ea5-model-cache\") pod \"sub-test-model-3650b0-kserve-8bc5c5779-sdxmr\" (UID: \"dc6e0fa6-24eb-4547-b2e8-78f7c85f7ea5\") " pod="ai-tenant-e2e-shared-b-w3-3650b0/sub-test-model-3650b0-kserve-8bc5c5779-sdxmr"
Sep 10 15:51:07.856007 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:51:07.855873    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc6e0fa6-24eb-4547-b2e8-78f7c85f7ea5-tmp-dir\") pod \"sub-test-model-3650b0-kserve-8bc5c5779-sdxmr\" (UID: \"dc6e0fa6-24eb-4547-b2e8-78f7c85f7ea5\") " pod="ai-tenant-e2e-shared-b-w3-3650b0/sub-test-model-3650b0-kserve-8bc5c5779-sdxmr"
Sep 10 15:51:07.856232 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:51:07.856182    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc6e0fa6-24eb-4547-b2e8-78f7c85f7ea5-home\") pod \"sub-test-model-3650b0-kserve-8bc5c5779-sdxmr\" (UID: \"dc6e0fa6-24eb-4547-b2e8-78f7c85f7ea5\") " pod="ai-tenant-e2e-shared-b-w3-3650b0/sub-test-model-3650b0-kserve-8bc5c5779-sdxmr"
Sep 10 15:51:07.856317 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:51:07.856290    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc6e0fa6-24eb-4547-b2e8-78f7c85f7ea5-model-cache\") pod \"sub-test-model-3650b0-kserve-8bc5c5779-sdxmr\" (UID: \"dc6e0fa6-24eb-4547-b2e8-78f7c85f7ea5\") " pod="ai-tenant-e2e-shared-b-w3-3650b0/sub-test-model-3650b0-kserve-8bc5c5779-sdxmr"
Sep 10 15:51:07.856317 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:51:07.856313    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc6e0fa6-24eb-4547-b2e8-78f7c85f7ea5-tmp-dir\") pod \"sub-test-model-3650b0-kserve-8bc5c5779-sdxmr\" (UID: \"dc6e0fa6-24eb-4547-b2e8-78f7c85f7ea5\") " pod="ai-tenant-e2e-shared-b-w3-3650b0/sub-test-model-3650b0-kserve-8bc5c5779-sdxmr"
Sep 10 15:51:07.858167 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:51:07.858144    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc6e0fa6-24eb-4547-b2e8-78f7c85f7ea5-dshm\") pod \"sub-test-model-3650b0-kserve-8bc5c5779-sdxmr\" (UID: \"dc6e0fa6-24eb-4547-b2e8-78f7c85f7ea5\") " pod="ai-tenant-e2e-shared-b-w3-3650b0/sub-test-model-3650b0-kserve-8bc5c5779-sdxmr"
Sep 10 15:51:07.858513 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:51:07.858497    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc6e0fa6-24eb-4547-b2e8-78f7c85f7ea5-tls-certs\") pod \"sub-test-model-3650b0-kserve-8bc5c5779-sdxmr\" (UID: \"dc6e0fa6-24eb-4547-b2e8-78f7c85f7ea5\") " pod="ai-tenant-e2e-shared-b-w3-3650b0/sub-test-model-3650b0-kserve-8bc5c5779-sdxmr"
Sep 10 15:51:07.868377 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:51:07.868356    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wznsj\" (UniqueName: \"kubernetes.io/projected/dc6e0fa6-24eb-4547-b2e8-78f7c85f7ea5-kube-api-access-wznsj\") pod \"sub-test-model-3650b0-kserve-8bc5c5779-sdxmr\" (UID: \"dc6e0fa6-24eb-4547-b2e8-78f7c85f7ea5\") " pod="ai-tenant-e2e-shared-b-w3-3650b0/sub-test-model-3650b0-kserve-8bc5c5779-sdxmr"
Sep 10 15:51:08.023754 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:51:08.023662    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-3650b0/sub-test-model-3650b0-kserve-8bc5c5779-sdxmr"
Sep 10 15:51:08.308473 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:51:08.308447    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3650b0/sub-test-model-3650b0-kserve-8bc5c5779-sdxmr"]
Sep 10 15:51:08.310091 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:51:08.310068    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddc6e0fa6_24eb_4547_b2e8_78f7c85f7ea5.slice/crio-1f4faeee671c047b4066dcb6d5109953e44ef9cdaca99e1334bb16f9dd5b7383 WatchSource:0}: Error finding container 1f4faeee671c047b4066dcb6d5109953e44ef9cdaca99e1334bb16f9dd5b7383: Status 404 returned error can't find the container with id 1f4faeee671c047b4066dcb6d5109953e44ef9cdaca99e1334bb16f9dd5b7383
Sep 10 15:51:08.427060 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:51:08.427023    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-3650b0/sub-test-model-3650b0-kserve-8bc5c5779-sdxmr" event={"ID":"dc6e0fa6-24eb-4547-b2e8-78f7c85f7ea5","Type":"ContainerStarted","Data":"fc77f0a140a7c57d649562e254c0acf104a2d086923e1db009c9ebeaedaa1413"}
Sep 10 15:51:08.427249 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:51:08.427074    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-3650b0/sub-test-model-3650b0-kserve-8bc5c5779-sdxmr" event={"ID":"dc6e0fa6-24eb-4547-b2e8-78f7c85f7ea5","Type":"ContainerStarted","Data":"1f4faeee671c047b4066dcb6d5109953e44ef9cdaca99e1334bb16f9dd5b7383"}
Sep 10 15:51:08.447797 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:51:08.447739    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-3650b0/sub-test-model-3650b0-kserve-8bc5c5779-sdxmr" podStartSLOduration=1.44772394 podStartE2EDuration="1.44772394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:51:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:51:08.445299031 +0000 UTC m=+1444.187264135" watchObservedRunningTime="2026-09-10 15:51:08.44772394 +0000 UTC m=+1444.189689085"
Sep 10 15:51:13.707913 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:51:13.707878    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bs7hd"]
Sep 10 15:51:18.023362 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:51:18.023320    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-3650b0/sub-test-model-3650b0-kserve-8bc5c5779-sdxmr"
Sep 10 15:51:18.023933 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:51:18.023375    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-3650b0/sub-test-model-3650b0-kserve-8bc5c5779-sdxmr"
Sep 10 15:51:18.036139 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:51:18.036109    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-3650b0/sub-test-model-3650b0-kserve-8bc5c5779-sdxmr"
Sep 10 15:51:18.493788 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:51:18.493760    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-3650b0/sub-test-model-3650b0-kserve-8bc5c5779-sdxmr"
Sep 10 15:51:29.701115 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:51:29.701080    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bs7hd"]
Sep 10 15:51:43.472535 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:51:43.472495    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-59c481-openshift-default-779dc66c4b-d55nm"]
Sep 10 15:51:43.480744 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:51:43.480719    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-59c481-openshift-default-779dc66c4b-d55nm"
Sep 10 15:51:43.483295 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:51:43.483273    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-59c481-openshift-default-dockercfg-rrmvh\""
Sep 10 15:51:43.492914 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:51:43.492882    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-59c481-openshift-default-779dc66c4b-d55nm"]
Sep 10 15:51:43.585003 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:51:43.584961    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06dbe419-7af5-4fcf-84d2-a14d5a89b104-istio-token\") pod \"e2e-api-b-59c481-openshift-default-779dc66c4b-d55nm\" (UID: \"06dbe419-7af5-4fcf-84d2-a14d5a89b104\") " pod="openshift-ingress/e2e-api-b-59c481-openshift-default-779dc66c4b-d55nm"
Sep 10 15:51:43.585003 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:51:43.585015    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06dbe419-7af5-4fcf-84d2-a14d5a89b104-istio-envoy\") pod \"e2e-api-b-59c481-openshift-default-779dc66c4b-d55nm\" (UID: \"06dbe419-7af5-4fcf-84d2-a14d5a89b104\") " pod="openshift-ingress/e2e-api-b-59c481-openshift-default-779dc66c4b-d55nm"
Sep 10 15:51:43.585294 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:51:43.585047    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06dbe419-7af5-4fcf-84d2-a14d5a89b104-credential-socket\") pod \"e2e-api-b-59c481-openshift-default-779dc66c4b-d55nm\" (UID: \"06dbe419-7af5-4fcf-84d2-a14d5a89b104\") " pod="openshift-ingress/e2e-api-b-59c481-openshift-default-779dc66c4b-d55nm"
Sep 10 15:51:43.585294 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:51:43.585125    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52zvz\" (UniqueName: \"kubernetes.io/projected/06dbe419-7af5-4fcf-84d2-a14d5a89b104-kube-api-access-52zvz\") pod \"e2e-api-b-59c481-openshift-default-779dc66c4b-d55nm\" (UID: \"06dbe419-7af5-4fcf-84d2-a14d5a89b104\") " pod="openshift-ingress/e2e-api-b-59c481-openshift-default-779dc66c4b-d55nm"
Sep 10 15:51:43.585294 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:51:43.585223    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06dbe419-7af5-4fcf-84d2-a14d5a89b104-istio-data\") pod \"e2e-api-b-59c481-openshift-default-779dc66c4b-d55nm\" (UID: \"06dbe419-7af5-4fcf-84d2-a14d5a89b104\") " pod="openshift-ingress/e2e-api-b-59c481-openshift-default-779dc66c4b-d55nm"
Sep 10 15:51:43.585294 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:51:43.585253    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06dbe419-7af5-4fcf-84d2-a14d5a89b104-istiod-ca-cert\") pod \"e2e-api-b-59c481-openshift-default-779dc66c4b-d55nm\" (UID: \"06dbe419-7af5-4fcf-84d2-a14d5a89b104\") " pod="openshift-ingress/e2e-api-b-59c481-openshift-default-779dc66c4b-d55nm"
Sep 10 15:51:43.585294 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:51:43.585281    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06dbe419-7af5-4fcf-84d2-a14d5a89b104-workload-socket\") pod \"e2e-api-b-59c481-openshift-default-779dc66c4b-d55nm\" (UID: \"06dbe419-7af5-4fcf-84d2-a14d5a89b104\") " pod="openshift-ingress/e2e-api-b-59c481-openshift-default-779dc66c4b-d55nm"
Sep 10 15:51:43.585661 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:51:43.585306    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06dbe419-7af5-4fcf-84d2-a14d5a89b104-workload-certs\") pod \"e2e-api-b-59c481-openshift-default-779dc66c4b-d55nm\" (UID: \"06dbe419-7af5-4fcf-84d2-a14d5a89b104\") " pod="openshift-ingress/e2e-api-b-59c481-openshift-default-779dc66c4b-d55nm"
Sep 10 15:51:43.585661 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:51:43.585348    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06dbe419-7af5-4fcf-84d2-a14d5a89b104-istio-podinfo\") pod \"e2e-api-b-59c481-openshift-default-779dc66c4b-d55nm\" (UID: \"06dbe419-7af5-4fcf-84d2-a14d5a89b104\") " pod="openshift-ingress/e2e-api-b-59c481-openshift-default-779dc66c4b-d55nm"
Sep 10 15:51:43.686294 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:51:43.686258    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06dbe419-7af5-4fcf-84d2-a14d5a89b104-istio-token\") pod \"e2e-api-b-59c481-openshift-default-779dc66c4b-d55nm\" (UID: \"06dbe419-7af5-4fcf-84d2-a14d5a89b104\") " pod="openshift-ingress/e2e-api-b-59c481-openshift-default-779dc66c4b-d55nm"
Sep 10 15:51:43.686294 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:51:43.686303    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06dbe419-7af5-4fcf-84d2-a14d5a89b104-istio-envoy\") pod \"e2e-api-b-59c481-openshift-default-779dc66c4b-d55nm\" (UID: \"06dbe419-7af5-4fcf-84d2-a14d5a89b104\") " pod="openshift-ingress/e2e-api-b-59c481-openshift-default-779dc66c4b-d55nm"
Sep 10 15:51:43.686601 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:51:43.686483    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06dbe419-7af5-4fcf-84d2-a14d5a89b104-credential-socket\") pod \"e2e-api-b-59c481-openshift-default-779dc66c4b-d55nm\" (UID: \"06dbe419-7af5-4fcf-84d2-a14d5a89b104\") " pod="openshift-ingress/e2e-api-b-59c481-openshift-default-779dc66c4b-d55nm"
Sep 10 15:51:43.686601 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:51:43.686517    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52zvz\" (UniqueName: \"kubernetes.io/projected/06dbe419-7af5-4fcf-84d2-a14d5a89b104-kube-api-access-52zvz\") pod \"e2e-api-b-59c481-openshift-default-779dc66c4b-d55nm\" (UID: \"06dbe419-7af5-4fcf-84d2-a14d5a89b104\") " pod="openshift-ingress/e2e-api-b-59c481-openshift-default-779dc66c4b-d55nm"
Sep 10 15:51:43.686601 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:51:43.686555    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06dbe419-7af5-4fcf-84d2-a14d5a89b104-istio-data\") pod \"e2e-api-b-59c481-openshift-default-779dc66c4b-d55nm\" (UID: \"06dbe419-7af5-4fcf-84d2-a14d5a89b104\") " pod="openshift-ingress/e2e-api-b-59c481-openshift-default-779dc66c4b-d55nm"
Sep 10 15:51:43.686754 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:51:43.686689    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06dbe419-7af5-4fcf-84d2-a14d5a89b104-istiod-ca-cert\") pod \"e2e-api-b-59c481-openshift-default-779dc66c4b-d55nm\" (UID: \"06dbe419-7af5-4fcf-84d2-a14d5a89b104\") " pod="openshift-ingress/e2e-api-b-59c481-openshift-default-779dc66c4b-d55nm"
Sep 10 15:51:43.686754 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:51:43.686730    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06dbe419-7af5-4fcf-84d2-a14d5a89b104-workload-socket\") pod \"e2e-api-b-59c481-openshift-default-779dc66c4b-d55nm\" (UID: \"06dbe419-7af5-4fcf-84d2-a14d5a89b104\") " pod="openshift-ingress/e2e-api-b-59c481-openshift-default-779dc66c4b-d55nm"
Sep 10 15:51:43.686866 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:51:43.686849    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06dbe419-7af5-4fcf-84d2-a14d5a89b104-workload-certs\") pod \"e2e-api-b-59c481-openshift-default-779dc66c4b-d55nm\" (UID: \"06dbe419-7af5-4fcf-84d2-a14d5a89b104\") " pod="openshift-ingress/e2e-api-b-59c481-openshift-default-779dc66c4b-d55nm"
Sep 10 15:51:43.686922 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:51:43.686909    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06dbe419-7af5-4fcf-84d2-a14d5a89b104-istio-podinfo\") pod \"e2e-api-b-59c481-openshift-default-779dc66c4b-d55nm\" (UID: \"06dbe419-7af5-4fcf-84d2-a14d5a89b104\") " pod="openshift-ingress/e2e-api-b-59c481-openshift-default-779dc66c4b-d55nm"
Sep 10 15:51:43.687003 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:51:43.686976    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06dbe419-7af5-4fcf-84d2-a14d5a89b104-istio-data\") pod \"e2e-api-b-59c481-openshift-default-779dc66c4b-d55nm\" (UID: \"06dbe419-7af5-4fcf-84d2-a14d5a89b104\") " pod="openshift-ingress/e2e-api-b-59c481-openshift-default-779dc66c4b-d55nm"
Sep 10 15:51:43.687071 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:51:43.686998    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06dbe419-7af5-4fcf-84d2-a14d5a89b104-credential-socket\") pod \"e2e-api-b-59c481-openshift-default-779dc66c4b-d55nm\" (UID: \"06dbe419-7af5-4fcf-84d2-a14d5a89b104\") " pod="openshift-ingress/e2e-api-b-59c481-openshift-default-779dc66c4b-d55nm"
Sep 10 15:51:43.687131 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:51:43.687072    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06dbe419-7af5-4fcf-84d2-a14d5a89b104-workload-socket\") pod \"e2e-api-b-59c481-openshift-default-779dc66c4b-d55nm\" (UID: \"06dbe419-7af5-4fcf-84d2-a14d5a89b104\") " pod="openshift-ingress/e2e-api-b-59c481-openshift-default-779dc66c4b-d55nm"
Sep 10 15:51:43.687302 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:51:43.687285    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06dbe419-7af5-4fcf-84d2-a14d5a89b104-workload-certs\") pod \"e2e-api-b-59c481-openshift-default-779dc66c4b-d55nm\" (UID: \"06dbe419-7af5-4fcf-84d2-a14d5a89b104\") " pod="openshift-ingress/e2e-api-b-59c481-openshift-default-779dc66c4b-d55nm"
Sep 10 15:51:43.687604 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:51:43.687561    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06dbe419-7af5-4fcf-84d2-a14d5a89b104-istiod-ca-cert\") pod \"e2e-api-b-59c481-openshift-default-779dc66c4b-d55nm\" (UID: \"06dbe419-7af5-4fcf-84d2-a14d5a89b104\") " pod="openshift-ingress/e2e-api-b-59c481-openshift-default-779dc66c4b-d55nm"
Sep 10 15:51:43.688842 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:51:43.688824    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06dbe419-7af5-4fcf-84d2-a14d5a89b104-istio-envoy\") pod \"e2e-api-b-59c481-openshift-default-779dc66c4b-d55nm\" (UID: \"06dbe419-7af5-4fcf-84d2-a14d5a89b104\") " pod="openshift-ingress/e2e-api-b-59c481-openshift-default-779dc66c4b-d55nm"
Sep 10 15:51:43.689254 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:51:43.689238    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06dbe419-7af5-4fcf-84d2-a14d5a89b104-istio-podinfo\") pod \"e2e-api-b-59c481-openshift-default-779dc66c4b-d55nm\" (UID: \"06dbe419-7af5-4fcf-84d2-a14d5a89b104\") " pod="openshift-ingress/e2e-api-b-59c481-openshift-default-779dc66c4b-d55nm"
Sep 10 15:51:43.696002 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:51:43.695982    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06dbe419-7af5-4fcf-84d2-a14d5a89b104-istio-token\") pod \"e2e-api-b-59c481-openshift-default-779dc66c4b-d55nm\" (UID: \"06dbe419-7af5-4fcf-84d2-a14d5a89b104\") " pod="openshift-ingress/e2e-api-b-59c481-openshift-default-779dc66c4b-d55nm"
Sep 10 15:51:43.696149 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:51:43.696132    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-52zvz\" (UniqueName: \"kubernetes.io/projected/06dbe419-7af5-4fcf-84d2-a14d5a89b104-kube-api-access-52zvz\") pod \"e2e-api-b-59c481-openshift-default-779dc66c4b-d55nm\" (UID: \"06dbe419-7af5-4fcf-84d2-a14d5a89b104\") " pod="openshift-ingress/e2e-api-b-59c481-openshift-default-779dc66c4b-d55nm"
Sep 10 15:51:43.793643 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:51:43.793536    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-59c481-openshift-default-779dc66c4b-d55nm"
Sep 10 15:51:44.071698 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:51:44.071663    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-59c481-openshift-default-779dc66c4b-d55nm"]
Sep 10 15:51:44.072809 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:51:44.072785    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06dbe419_7af5_4fcf_84d2_a14d5a89b104.slice/crio-2c1d89d0b5b83b8efc27c34daf5b2d68cde2e31df23b651ced29427a206a3f46 WatchSource:0}: Error finding container 2c1d89d0b5b83b8efc27c34daf5b2d68cde2e31df23b651ced29427a206a3f46: Status 404 returned error can't find the container with id 2c1d89d0b5b83b8efc27c34daf5b2d68cde2e31df23b651ced29427a206a3f46
Sep 10 15:51:44.075171 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:51:44.075138    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 15:51:44.075251 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:51:44.075205    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 15:51:44.075251 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:51:44.075234    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 15:51:44.626744 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:51:44.626703    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-59c481-openshift-default-779dc66c4b-d55nm" event={"ID":"06dbe419-7af5-4fcf-84d2-a14d5a89b104","Type":"ContainerStarted","Data":"374faaf00fbf35fe4c0d6b1fcce6c5461341e66984544186311e85c8e70ba16f"}
Sep 10 15:51:44.627157 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:51:44.626767    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-59c481-openshift-default-779dc66c4b-d55nm" event={"ID":"06dbe419-7af5-4fcf-84d2-a14d5a89b104","Type":"ContainerStarted","Data":"2c1d89d0b5b83b8efc27c34daf5b2d68cde2e31df23b651ced29427a206a3f46"}
Sep 10 15:51:44.650916 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:51:44.650867    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-59c481-openshift-default-779dc66c4b-d55nm" podStartSLOduration=1.650848992 podStartE2EDuration="1.650848992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:51:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:51:44.648357818 +0000 UTC m=+1480.390322923" watchObservedRunningTime="2026-09-10 15:51:44.650848992 +0000 UTC m=+1480.392814095"
Sep 10 15:51:44.794075 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:51:44.794047    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-59c481-openshift-default-779dc66c4b-d55nm"
Sep 10 15:51:45.799647 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:51:45.799616    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-59c481-openshift-default-779dc66c4b-d55nm"
Sep 10 15:51:46.639833 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:51:46.639797    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-59c481-openshift-default-779dc66c4b-d55nm"
Sep 10 15:51:46.641059 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:51:46.641036    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-59c481-openshift-default-779dc66c4b-d55nm"
Sep 10 15:51:49.208984 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:51:49.208950    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bs7hd"]
Sep 10 15:51:51.222647 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:51:51.222609    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-59c481-59447d66b7-dwcq9"]
Sep 10 15:51:51.226318 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:51:51.226295    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-59c481-59447d66b7-dwcq9"
Sep 10 15:51:51.229158 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:51:51.229129    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-59c481-serving-cert\""
Sep 10 15:51:51.237210 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:51:51.237183    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-59c481-59447d66b7-dwcq9"]
Sep 10 15:51:51.264774 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:51:51.264744    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4t7pd\" (UniqueName: \"kubernetes.io/projected/2a130f21-4148-4f20-aa48-8447a2a34082-kube-api-access-4t7pd\") pod \"maas-api-e2e-api-b-59c481-59447d66b7-dwcq9\" (UID: \"2a130f21-4148-4f20-aa48-8447a2a34082\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-59c481-59447d66b7-dwcq9"
Sep 10 15:51:51.264955 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:51:51.264869    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a130f21-4148-4f20-aa48-8447a2a34082-maas-api-tls\") pod \"maas-api-e2e-api-b-59c481-59447d66b7-dwcq9\" (UID: \"2a130f21-4148-4f20-aa48-8447a2a34082\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-59c481-59447d66b7-dwcq9"
Sep 10 15:51:51.264955 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:51:51.264912    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a130f21-4148-4f20-aa48-8447a2a34082-metrics-certs\") pod \"maas-api-e2e-api-b-59c481-59447d66b7-dwcq9\" (UID: \"2a130f21-4148-4f20-aa48-8447a2a34082\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-59c481-59447d66b7-dwcq9"
Sep 10 15:51:51.264955 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:51:51.264952    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a130f21-4148-4f20-aa48-8447a2a34082-openshift-service-ca\") pod \"maas-api-e2e-api-b-59c481-59447d66b7-dwcq9\" (UID: \"2a130f21-4148-4f20-aa48-8447a2a34082\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-59c481-59447d66b7-dwcq9"
Sep 10 15:51:51.366094 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:51:51.366058    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4t7pd\" (UniqueName: \"kubernetes.io/projected/2a130f21-4148-4f20-aa48-8447a2a34082-kube-api-access-4t7pd\") pod \"maas-api-e2e-api-b-59c481-59447d66b7-dwcq9\" (UID: \"2a130f21-4148-4f20-aa48-8447a2a34082\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-59c481-59447d66b7-dwcq9"
Sep 10 15:51:51.366300 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:51:51.366148    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a130f21-4148-4f20-aa48-8447a2a34082-maas-api-tls\") pod \"maas-api-e2e-api-b-59c481-59447d66b7-dwcq9\" (UID: \"2a130f21-4148-4f20-aa48-8447a2a34082\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-59c481-59447d66b7-dwcq9"
Sep 10 15:51:51.366300 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:51:51.366178    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a130f21-4148-4f20-aa48-8447a2a34082-metrics-certs\") pod \"maas-api-e2e-api-b-59c481-59447d66b7-dwcq9\" (UID: \"2a130f21-4148-4f20-aa48-8447a2a34082\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-59c481-59447d66b7-dwcq9"
Sep 10 15:51:51.366300 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:51:51.366213    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a130f21-4148-4f20-aa48-8447a2a34082-openshift-service-ca\") pod \"maas-api-e2e-api-b-59c481-59447d66b7-dwcq9\" (UID: \"2a130f21-4148-4f20-aa48-8447a2a34082\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-59c481-59447d66b7-dwcq9"
Sep 10 15:51:51.366484 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:51:51.366312    2559 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-59c481-serving-cert: secret "maas-api-e2e-api-b-59c481-serving-cert" not found
Sep 10 15:51:51.366484 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:51:51.366376    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a130f21-4148-4f20-aa48-8447a2a34082-maas-api-tls podName:2a130f21-4148-4f20-aa48-8447a2a34082 nodeName:}" failed. No retries permitted until 2026-09-10 15:51:51.86635529 +0000 UTC m=+1487.608320390 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2a130f21-4148-4f20-aa48-8447a2a34082-maas-api-tls") pod "maas-api-e2e-api-b-59c481-59447d66b7-dwcq9" (UID: "2a130f21-4148-4f20-aa48-8447a2a34082") : secret "maas-api-e2e-api-b-59c481-serving-cert" not found
Sep 10 15:51:51.366999 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:51:51.366966    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a130f21-4148-4f20-aa48-8447a2a34082-openshift-service-ca\") pod \"maas-api-e2e-api-b-59c481-59447d66b7-dwcq9\" (UID: \"2a130f21-4148-4f20-aa48-8447a2a34082\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-59c481-59447d66b7-dwcq9"
Sep 10 15:51:51.369101 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:51:51.369077    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a130f21-4148-4f20-aa48-8447a2a34082-metrics-certs\") pod \"maas-api-e2e-api-b-59c481-59447d66b7-dwcq9\" (UID: \"2a130f21-4148-4f20-aa48-8447a2a34082\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-59c481-59447d66b7-dwcq9"
Sep 10 15:51:51.378210 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:51:51.378177    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4t7pd\" (UniqueName: \"kubernetes.io/projected/2a130f21-4148-4f20-aa48-8447a2a34082-kube-api-access-4t7pd\") pod \"maas-api-e2e-api-b-59c481-59447d66b7-dwcq9\" (UID: \"2a130f21-4148-4f20-aa48-8447a2a34082\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-59c481-59447d66b7-dwcq9"
Sep 10 15:51:51.871446 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:51:51.871410    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a130f21-4148-4f20-aa48-8447a2a34082-maas-api-tls\") pod \"maas-api-e2e-api-b-59c481-59447d66b7-dwcq9\" (UID: \"2a130f21-4148-4f20-aa48-8447a2a34082\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-59c481-59447d66b7-dwcq9"
Sep 10 15:51:51.873825 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:51:51.873794    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a130f21-4148-4f20-aa48-8447a2a34082-maas-api-tls\") pod \"maas-api-e2e-api-b-59c481-59447d66b7-dwcq9\" (UID: \"2a130f21-4148-4f20-aa48-8447a2a34082\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-59c481-59447d66b7-dwcq9"
Sep 10 15:51:52.138866 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:51:52.138757    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-59c481-59447d66b7-dwcq9"
Sep 10 15:51:52.413844 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:51:52.413821    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-59c481-59447d66b7-dwcq9"]
Sep 10 15:51:52.415511 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:51:52.415483    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a130f21_4148_4f20_aa48_8447a2a34082.slice/crio-5b46fcc568d59d7095ffbf28fc8b32601dca62ab09466c576f7eb62721053079 WatchSource:0}: Error finding container 5b46fcc568d59d7095ffbf28fc8b32601dca62ab09466c576f7eb62721053079: Status 404 returned error can't find the container with id 5b46fcc568d59d7095ffbf28fc8b32601dca62ab09466c576f7eb62721053079
Sep 10 15:51:52.416938 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:51:52.416922    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 15:51:52.678028 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:51:52.677933    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-59c481-59447d66b7-dwcq9" event={"ID":"2a130f21-4148-4f20-aa48-8447a2a34082","Type":"ContainerStarted","Data":"5b46fcc568d59d7095ffbf28fc8b32601dca62ab09466c576f7eb62721053079"}
Sep 10 15:51:53.685142 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:51:53.685106    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-59c481-59447d66b7-dwcq9" event={"ID":"2a130f21-4148-4f20-aa48-8447a2a34082","Type":"ContainerStarted","Data":"3a7d40a9ab5d203a6383205683677d507a1f75808987d956d76965ac5c7e2e6b"}
Sep 10 15:51:53.685536 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:51:53.685214    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-59c481-59447d66b7-dwcq9"
Sep 10 15:51:53.706637 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:51:53.706568    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-59c481-59447d66b7-dwcq9" podStartSLOduration=2.147755159 podStartE2EDuration="2.70655414s" totalImagesPullingTime="558.798981ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:51:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:51:53.705393398 +0000 UTC m=+1489.447358529" watchObservedRunningTime="2026-09-10 15:51:53.70655414 +0000 UTC m=+1489.448519242"
Sep 10 15:51:59.485245 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:51:59.485211    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-c7257b/sub-test-model-c7257b-kserve-86db97955f-2x784"]
Sep 10 15:51:59.696284 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:51:59.696253    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-59c481-59447d66b7-dwcq9"
Sep 10 15:52:00.392030 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:00.391993    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3650b0/sub-test-model-3650b0-kserve-8bc5c5779-sdxmr"]
Sep 10 15:52:04.523617 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:04.523569    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-c7257b/rate-test-model-c7257b-kserve-64f78d4b67-vmzm8"]
Sep 10 15:52:04.527481 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:04.527459    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-c7257b/rate-test-model-c7257b-kserve-64f78d4b67-vmzm8"
Sep 10 15:52:04.530106 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:04.530084    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-c7257b\"/\"rate-test-model-c7257b-kserve-self-signed-certs\""
Sep 10 15:52:04.537873 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:04.537850    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-c7257b/rate-test-model-c7257b-kserve-64f78d4b67-vmzm8"]
Sep 10 15:52:04.606720 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:04.606683    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36b69e2d-fc8c-459a-b70e-7828ab9edb18-model-cache\") pod \"rate-test-model-c7257b-kserve-64f78d4b67-vmzm8\" (UID: \"36b69e2d-fc8c-459a-b70e-7828ab9edb18\") " pod="ai-tenant-e2e-shared-a-w3-c7257b/rate-test-model-c7257b-kserve-64f78d4b67-vmzm8"
Sep 10 15:52:04.606917 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:04.606731    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36b69e2d-fc8c-459a-b70e-7828ab9edb18-tls-certs\") pod \"rate-test-model-c7257b-kserve-64f78d4b67-vmzm8\" (UID: \"36b69e2d-fc8c-459a-b70e-7828ab9edb18\") " pod="ai-tenant-e2e-shared-a-w3-c7257b/rate-test-model-c7257b-kserve-64f78d4b67-vmzm8"
Sep 10 15:52:04.606917 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:04.606760    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36b69e2d-fc8c-459a-b70e-7828ab9edb18-tmp-dir\") pod \"rate-test-model-c7257b-kserve-64f78d4b67-vmzm8\" (UID: \"36b69e2d-fc8c-459a-b70e-7828ab9edb18\") " pod="ai-tenant-e2e-shared-a-w3-c7257b/rate-test-model-c7257b-kserve-64f78d4b67-vmzm8"
Sep 10 15:52:04.606917 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:04.606847    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36b69e2d-fc8c-459a-b70e-7828ab9edb18-home\") pod \"rate-test-model-c7257b-kserve-64f78d4b67-vmzm8\" (UID: \"36b69e2d-fc8c-459a-b70e-7828ab9edb18\") " pod="ai-tenant-e2e-shared-a-w3-c7257b/rate-test-model-c7257b-kserve-64f78d4b67-vmzm8"
Sep 10 15:52:04.607089 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:04.606939    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36b69e2d-fc8c-459a-b70e-7828ab9edb18-dshm\") pod \"rate-test-model-c7257b-kserve-64f78d4b67-vmzm8\" (UID: \"36b69e2d-fc8c-459a-b70e-7828ab9edb18\") " pod="ai-tenant-e2e-shared-a-w3-c7257b/rate-test-model-c7257b-kserve-64f78d4b67-vmzm8"
Sep 10 15:52:04.607089 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:04.606986    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkrdc\" (UniqueName: \"kubernetes.io/projected/36b69e2d-fc8c-459a-b70e-7828ab9edb18-kube-api-access-xkrdc\") pod \"rate-test-model-c7257b-kserve-64f78d4b67-vmzm8\" (UID: \"36b69e2d-fc8c-459a-b70e-7828ab9edb18\") " pod="ai-tenant-e2e-shared-a-w3-c7257b/rate-test-model-c7257b-kserve-64f78d4b67-vmzm8"
Sep 10 15:52:04.708141 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:04.708108    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36b69e2d-fc8c-459a-b70e-7828ab9edb18-home\") pod \"rate-test-model-c7257b-kserve-64f78d4b67-vmzm8\" (UID: \"36b69e2d-fc8c-459a-b70e-7828ab9edb18\") " pod="ai-tenant-e2e-shared-a-w3-c7257b/rate-test-model-c7257b-kserve-64f78d4b67-vmzm8"
Sep 10 15:52:04.708337 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:04.708186    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36b69e2d-fc8c-459a-b70e-7828ab9edb18-dshm\") pod \"rate-test-model-c7257b-kserve-64f78d4b67-vmzm8\" (UID: \"36b69e2d-fc8c-459a-b70e-7828ab9edb18\") " pod="ai-tenant-e2e-shared-a-w3-c7257b/rate-test-model-c7257b-kserve-64f78d4b67-vmzm8"
Sep 10 15:52:04.708337 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:04.708219    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkrdc\" (UniqueName: \"kubernetes.io/projected/36b69e2d-fc8c-459a-b70e-7828ab9edb18-kube-api-access-xkrdc\") pod \"rate-test-model-c7257b-kserve-64f78d4b67-vmzm8\" (UID: \"36b69e2d-fc8c-459a-b70e-7828ab9edb18\") " pod="ai-tenant-e2e-shared-a-w3-c7257b/rate-test-model-c7257b-kserve-64f78d4b67-vmzm8"
Sep 10 15:52:04.708337 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:04.708254    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36b69e2d-fc8c-459a-b70e-7828ab9edb18-model-cache\") pod \"rate-test-model-c7257b-kserve-64f78d4b67-vmzm8\" (UID: \"36b69e2d-fc8c-459a-b70e-7828ab9edb18\") " pod="ai-tenant-e2e-shared-a-w3-c7257b/rate-test-model-c7257b-kserve-64f78d4b67-vmzm8"
Sep 10 15:52:04.708337 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:04.708292    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36b69e2d-fc8c-459a-b70e-7828ab9edb18-tls-certs\") pod \"rate-test-model-c7257b-kserve-64f78d4b67-vmzm8\" (UID: \"36b69e2d-fc8c-459a-b70e-7828ab9edb18\") " pod="ai-tenant-e2e-shared-a-w3-c7257b/rate-test-model-c7257b-kserve-64f78d4b67-vmzm8"
Sep 10 15:52:04.708337 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:04.708326    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36b69e2d-fc8c-459a-b70e-7828ab9edb18-tmp-dir\") pod \"rate-test-model-c7257b-kserve-64f78d4b67-vmzm8\" (UID: \"36b69e2d-fc8c-459a-b70e-7828ab9edb18\") " pod="ai-tenant-e2e-shared-a-w3-c7257b/rate-test-model-c7257b-kserve-64f78d4b67-vmzm8"
Sep 10 15:52:04.708754 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:04.708601    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36b69e2d-fc8c-459a-b70e-7828ab9edb18-home\") pod \"rate-test-model-c7257b-kserve-64f78d4b67-vmzm8\" (UID: \"36b69e2d-fc8c-459a-b70e-7828ab9edb18\") " pod="ai-tenant-e2e-shared-a-w3-c7257b/rate-test-model-c7257b-kserve-64f78d4b67-vmzm8"
Sep 10 15:52:04.708754 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:04.708734    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36b69e2d-fc8c-459a-b70e-7828ab9edb18-tmp-dir\") pod \"rate-test-model-c7257b-kserve-64f78d4b67-vmzm8\" (UID: \"36b69e2d-fc8c-459a-b70e-7828ab9edb18\") " pod="ai-tenant-e2e-shared-a-w3-c7257b/rate-test-model-c7257b-kserve-64f78d4b67-vmzm8"
Sep 10 15:52:04.708879 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:04.708812    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36b69e2d-fc8c-459a-b70e-7828ab9edb18-model-cache\") pod \"rate-test-model-c7257b-kserve-64f78d4b67-vmzm8\" (UID: \"36b69e2d-fc8c-459a-b70e-7828ab9edb18\") " pod="ai-tenant-e2e-shared-a-w3-c7257b/rate-test-model-c7257b-kserve-64f78d4b67-vmzm8"
Sep 10 15:52:04.711041 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:04.711002    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36b69e2d-fc8c-459a-b70e-7828ab9edb18-dshm\") pod \"rate-test-model-c7257b-kserve-64f78d4b67-vmzm8\" (UID: \"36b69e2d-fc8c-459a-b70e-7828ab9edb18\") " pod="ai-tenant-e2e-shared-a-w3-c7257b/rate-test-model-c7257b-kserve-64f78d4b67-vmzm8"
Sep 10 15:52:04.711173 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:04.711105    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-c7257b\"/\"rate-test-model-c7257b-kserve-self-signed-certs\""
Sep 10 15:52:04.718701 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:04.718673    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkrdc\" (UniqueName: \"kubernetes.io/projected/36b69e2d-fc8c-459a-b70e-7828ab9edb18-kube-api-access-xkrdc\") pod \"rate-test-model-c7257b-kserve-64f78d4b67-vmzm8\" (UID: \"36b69e2d-fc8c-459a-b70e-7828ab9edb18\") " pod="ai-tenant-e2e-shared-a-w3-c7257b/rate-test-model-c7257b-kserve-64f78d4b67-vmzm8"
Sep 10 15:52:04.721168 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:04.721142    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36b69e2d-fc8c-459a-b70e-7828ab9edb18-tls-certs\") pod \"rate-test-model-c7257b-kserve-64f78d4b67-vmzm8\" (UID: \"36b69e2d-fc8c-459a-b70e-7828ab9edb18\") " pod="ai-tenant-e2e-shared-a-w3-c7257b/rate-test-model-c7257b-kserve-64f78d4b67-vmzm8"
Sep 10 15:52:04.837611 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:04.837495    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-c7257b/rate-test-model-c7257b-kserve-64f78d4b67-vmzm8"
Sep 10 15:52:05.171395 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:05.171353    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-c7257b/rate-test-model-c7257b-kserve-64f78d4b67-vmzm8"]
Sep 10 15:52:05.172806 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:52:05.172780    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod36b69e2d_fc8c_459a_b70e_7828ab9edb18.slice/crio-2149eb06e9aa2afe24e53fa25ac636b5426d2c33316eebc98f92e3f3729005d0 WatchSource:0}: Error finding container 2149eb06e9aa2afe24e53fa25ac636b5426d2c33316eebc98f92e3f3729005d0: Status 404 returned error can't find the container with id 2149eb06e9aa2afe24e53fa25ac636b5426d2c33316eebc98f92e3f3729005d0
Sep 10 15:52:05.760994 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:05.760945    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-c7257b/rate-test-model-c7257b-kserve-64f78d4b67-vmzm8" event={"ID":"36b69e2d-fc8c-459a-b70e-7828ab9edb18","Type":"ContainerStarted","Data":"97707a6080f2d30b2a479fa85db7766f6806acba96cc1c604c2ca8e88c03745e"}
Sep 10 15:52:05.761513 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:05.761006    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-c7257b/rate-test-model-c7257b-kserve-64f78d4b67-vmzm8" event={"ID":"36b69e2d-fc8c-459a-b70e-7828ab9edb18","Type":"ContainerStarted","Data":"2149eb06e9aa2afe24e53fa25ac636b5426d2c33316eebc98f92e3f3729005d0"}
Sep 10 15:52:05.782826 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:05.782751    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-c7257b/rate-test-model-c7257b-kserve-64f78d4b67-vmzm8" podStartSLOduration=1.7827291029999999 podStartE2EDuration="1.782729103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:52:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:52:05.782541794 +0000 UTC m=+1501.524506898" watchObservedRunningTime="2026-09-10 15:52:05.782729103 +0000 UTC m=+1501.524694210"
Sep 10 15:52:10.208512 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:10.208477    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bs7hd"]
Sep 10 15:52:10.801296 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:10.801260    2559 generic.go:356] "Generic (PLEG): container finished" podID="06dbe419-7af5-4fcf-84d2-a14d5a89b104" containerID="374faaf00fbf35fe4c0d6b1fcce6c5461341e66984544186311e85c8e70ba16f" exitCode=0
Sep 10 15:52:10.801517 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:10.801332    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-59c481-openshift-default-779dc66c4b-d55nm" event={"ID":"06dbe419-7af5-4fcf-84d2-a14d5a89b104","Type":"ContainerDied","Data":"374faaf00fbf35fe4c0d6b1fcce6c5461341e66984544186311e85c8e70ba16f"}
Sep 10 15:52:10.801735 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:10.801720    2559 scope.go:117] "RemoveContainer" containerID="374faaf00fbf35fe4c0d6b1fcce6c5461341e66984544186311e85c8e70ba16f"
Sep 10 15:52:10.802861 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:10.802836    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 15:52:10.802934 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:10.802900    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 15:52:10.802934 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:10.802930    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 15:52:11.810607 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:11.810546    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-59c481-openshift-default-779dc66c4b-d55nm" event={"ID":"06dbe419-7af5-4fcf-84d2-a14d5a89b104","Type":"ContainerStarted","Data":"e13ea3d4d7e4252a7dc6cbbdac19197008ec6da590ee97fe0afc19d47793235a"}
Sep 10 15:52:12.794033 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:12.793972    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-59c481-openshift-default-779dc66c4b-d55nm"
Sep 10 15:52:12.795392 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:12.795366    2559 patch_prober.go:28] interesting pod/e2e-api-b-59c481-openshift-default-779dc66c4b-d55nm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Sep 10 15:52:12.795518 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:12.795416    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-59c481-openshift-default-779dc66c4b-d55nm" podUID="06dbe419-7af5-4fcf-84d2-a14d5a89b104" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Sep 10 15:52:13.793897 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:13.793863    2559 patch_prober.go:28] interesting pod/e2e-api-b-59c481-openshift-default-779dc66c4b-d55nm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Sep 10 15:52:13.794344 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:13.793917    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-59c481-openshift-default-779dc66c4b-d55nm" podUID="06dbe419-7af5-4fcf-84d2-a14d5a89b104" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Sep 10 15:52:14.504652 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:14.504613    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-c7257b/sub-test-model-c7257b-kserve-86db97955f-2x784" podUID="68b779b4-60c6-468e-b6e4-de95dda02bf1" containerName="main" containerID="cri-o://04dbeba9202f8b2324200c94f1ff28c0ded56891068d6c8cc1ccdc23f64d2c14" gracePeriod=45
Sep 10 15:52:14.794040 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:14.794010    2559 patch_prober.go:28] interesting pod/e2e-api-b-59c481-openshift-default-779dc66c4b-d55nm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Sep 10 15:52:14.794391 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:14.794060    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-59c481-openshift-default-779dc66c4b-d55nm" podUID="06dbe419-7af5-4fcf-84d2-a14d5a89b104" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Sep 10 15:52:14.799419 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:14.799399    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-c7257b/sub-test-model-c7257b-kserve-86db97955f-2x784"
Sep 10 15:52:14.830936 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:14.830908    2559 generic.go:356] "Generic (PLEG): container finished" podID="68b779b4-60c6-468e-b6e4-de95dda02bf1" containerID="04dbeba9202f8b2324200c94f1ff28c0ded56891068d6c8cc1ccdc23f64d2c14" exitCode=0
Sep 10 15:52:14.831107 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:14.830939    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-c7257b/sub-test-model-c7257b-kserve-86db97955f-2x784" event={"ID":"68b779b4-60c6-468e-b6e4-de95dda02bf1","Type":"ContainerDied","Data":"04dbeba9202f8b2324200c94f1ff28c0ded56891068d6c8cc1ccdc23f64d2c14"}
Sep 10 15:52:14.831107 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:14.830980    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-c7257b/sub-test-model-c7257b-kserve-86db97955f-2x784" event={"ID":"68b779b4-60c6-468e-b6e4-de95dda02bf1","Type":"ContainerDied","Data":"d65c46126d8641c472e8b43a45a6c8d14dd1680ef1f4ae264323bfdc69eadb3f"}
Sep 10 15:52:14.831107 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:14.831000    2559 scope.go:117] "RemoveContainer" containerID="04dbeba9202f8b2324200c94f1ff28c0ded56891068d6c8cc1ccdc23f64d2c14"
Sep 10 15:52:14.831107 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:14.831042    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-c7257b/sub-test-model-c7257b-kserve-86db97955f-2x784"
Sep 10 15:52:14.837624 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:14.837603    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-c7257b/rate-test-model-c7257b-kserve-64f78d4b67-vmzm8"
Sep 10 15:52:14.837750 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:14.837645    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-c7257b/rate-test-model-c7257b-kserve-64f78d4b67-vmzm8"
Sep 10 15:52:14.871458 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:14.870798    2559 scope.go:117] "RemoveContainer" containerID="04dbeba9202f8b2324200c94f1ff28c0ded56891068d6c8cc1ccdc23f64d2c14"
Sep 10 15:52:14.871458 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:52:14.871313    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04dbeba9202f8b2324200c94f1ff28c0ded56891068d6c8cc1ccdc23f64d2c14\": container with ID starting with 04dbeba9202f8b2324200c94f1ff28c0ded56891068d6c8cc1ccdc23f64d2c14 not found: ID does not exist" containerID="04dbeba9202f8b2324200c94f1ff28c0ded56891068d6c8cc1ccdc23f64d2c14"
Sep 10 15:52:14.871458 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:14.871352    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04dbeba9202f8b2324200c94f1ff28c0ded56891068d6c8cc1ccdc23f64d2c14"} err="failed to get container status \"04dbeba9202f8b2324200c94f1ff28c0ded56891068d6c8cc1ccdc23f64d2c14\": rpc error: code = NotFound desc = could not find container \"04dbeba9202f8b2324200c94f1ff28c0ded56891068d6c8cc1ccdc23f64d2c14\": container with ID starting with 04dbeba9202f8b2324200c94f1ff28c0ded56891068d6c8cc1ccdc23f64d2c14 not found: ID does not exist"
Sep 10 15:52:14.880363 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:14.880340    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-c7257b/rate-test-model-c7257b-kserve-64f78d4b67-vmzm8"
Sep 10 15:52:14.913857 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:14.913830    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ks854\" (UniqueName: \"kubernetes.io/projected/68b779b4-60c6-468e-b6e4-de95dda02bf1-kube-api-access-ks854\") pod \"68b779b4-60c6-468e-b6e4-de95dda02bf1\" (UID: \"68b779b4-60c6-468e-b6e4-de95dda02bf1\") "
Sep 10 15:52:14.914024 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:14.913931    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68b779b4-60c6-468e-b6e4-de95dda02bf1-model-cache\") pod \"68b779b4-60c6-468e-b6e4-de95dda02bf1\" (UID: \"68b779b4-60c6-468e-b6e4-de95dda02bf1\") "
Sep 10 15:52:14.914090 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:14.914025    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68b779b4-60c6-468e-b6e4-de95dda02bf1-tls-certs\") pod \"68b779b4-60c6-468e-b6e4-de95dda02bf1\" (UID: \"68b779b4-60c6-468e-b6e4-de95dda02bf1\") "
Sep 10 15:52:14.914090 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:14.914065    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68b779b4-60c6-468e-b6e4-de95dda02bf1-dshm\") pod \"68b779b4-60c6-468e-b6e4-de95dda02bf1\" (UID: \"68b779b4-60c6-468e-b6e4-de95dda02bf1\") "
Sep 10 15:52:14.914191 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:14.914121    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68b779b4-60c6-468e-b6e4-de95dda02bf1-tmp-dir\") pod \"68b779b4-60c6-468e-b6e4-de95dda02bf1\" (UID: \"68b779b4-60c6-468e-b6e4-de95dda02bf1\") "
Sep 10 15:52:14.914191 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:14.914148    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68b779b4-60c6-468e-b6e4-de95dda02bf1-home\") pod \"68b779b4-60c6-468e-b6e4-de95dda02bf1\" (UID: \"68b779b4-60c6-468e-b6e4-de95dda02bf1\") "
Sep 10 15:52:14.914294 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:14.914177    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68b779b4-60c6-468e-b6e4-de95dda02bf1-model-cache" (OuterVolumeSpecName: "model-cache") pod "68b779b4-60c6-468e-b6e4-de95dda02bf1" (UID: "68b779b4-60c6-468e-b6e4-de95dda02bf1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:52:14.914424 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:14.914375    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68b779b4-60c6-468e-b6e4-de95dda02bf1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "68b779b4-60c6-468e-b6e4-de95dda02bf1" (UID: "68b779b4-60c6-468e-b6e4-de95dda02bf1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:52:14.914495 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:14.914450    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68b779b4-60c6-468e-b6e4-de95dda02bf1-home" (OuterVolumeSpecName: "home") pod "68b779b4-60c6-468e-b6e4-de95dda02bf1" (UID: "68b779b4-60c6-468e-b6e4-de95dda02bf1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:52:14.914554 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:14.914541    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68b779b4-60c6-468e-b6e4-de95dda02bf1-tmp-dir\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:52:14.914642 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:14.914561    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68b779b4-60c6-468e-b6e4-de95dda02bf1-home\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:52:14.914642 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:14.914598    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68b779b4-60c6-468e-b6e4-de95dda02bf1-model-cache\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:52:14.916095 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:14.916070    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68b779b4-60c6-468e-b6e4-de95dda02bf1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "68b779b4-60c6-468e-b6e4-de95dda02bf1" (UID: "68b779b4-60c6-468e-b6e4-de95dda02bf1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:52:14.916197 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:14.916073    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68b779b4-60c6-468e-b6e4-de95dda02bf1-kube-api-access-ks854" (OuterVolumeSpecName: "kube-api-access-ks854") pod "68b779b4-60c6-468e-b6e4-de95dda02bf1" (UID: "68b779b4-60c6-468e-b6e4-de95dda02bf1"). InnerVolumeSpecName "kube-api-access-ks854". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:52:14.916491 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:14.916479    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68b779b4-60c6-468e-b6e4-de95dda02bf1-dshm" (OuterVolumeSpecName: "dshm") pod "68b779b4-60c6-468e-b6e4-de95dda02bf1" (UID: "68b779b4-60c6-468e-b6e4-de95dda02bf1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:52:15.015815 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:15.015774    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68b779b4-60c6-468e-b6e4-de95dda02bf1-tls-certs\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:52:15.015815 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:15.015807    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68b779b4-60c6-468e-b6e4-de95dda02bf1-dshm\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:52:15.015815 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:15.015820    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ks854\" (UniqueName: \"kubernetes.io/projected/68b779b4-60c6-468e-b6e4-de95dda02bf1-kube-api-access-ks854\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:52:15.158338 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:15.158294    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-c7257b/sub-test-model-c7257b-kserve-86db97955f-2x784"]
Sep 10 15:52:15.162273 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:15.162241    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-c7257b/sub-test-model-c7257b-kserve-86db97955f-2x784"]
Sep 10 15:52:15.413388 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:15.413351    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-3650b0/sub-test-model-3650b0-kserve-8bc5c5779-sdxmr" podUID="dc6e0fa6-24eb-4547-b2e8-78f7c85f7ea5" containerName="main" containerID="cri-o://fc77f0a140a7c57d649562e254c0acf104a2d086923e1db009c9ebeaedaa1413" gracePeriod=45
Sep 10 15:52:15.722875 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:15.722852    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-3650b0/sub-test-model-3650b0-kserve-8bc5c5779-sdxmr"
Sep 10 15:52:15.794812 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:15.794780    2559 patch_prober.go:28] interesting pod/e2e-api-b-59c481-openshift-default-779dc66c4b-d55nm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Sep 10 15:52:15.795196 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:15.794829    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-59c481-openshift-default-779dc66c4b-d55nm" podUID="06dbe419-7af5-4fcf-84d2-a14d5a89b104" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Sep 10 15:52:15.824420 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:15.824391    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc6e0fa6-24eb-4547-b2e8-78f7c85f7ea5-home\") pod \"dc6e0fa6-24eb-4547-b2e8-78f7c85f7ea5\" (UID: \"dc6e0fa6-24eb-4547-b2e8-78f7c85f7ea5\") "
Sep 10 15:52:15.824565 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:15.824439    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc6e0fa6-24eb-4547-b2e8-78f7c85f7ea5-model-cache\") pod \"dc6e0fa6-24eb-4547-b2e8-78f7c85f7ea5\" (UID: \"dc6e0fa6-24eb-4547-b2e8-78f7c85f7ea5\") "
Sep 10 15:52:15.824565 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:15.824512    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wznsj\" (UniqueName: \"kubernetes.io/projected/dc6e0fa6-24eb-4547-b2e8-78f7c85f7ea5-kube-api-access-wznsj\") pod \"dc6e0fa6-24eb-4547-b2e8-78f7c85f7ea5\" (UID: \"dc6e0fa6-24eb-4547-b2e8-78f7c85f7ea5\") "
Sep 10 15:52:15.824565 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:15.824532    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc6e0fa6-24eb-4547-b2e8-78f7c85f7ea5-dshm\") pod \"dc6e0fa6-24eb-4547-b2e8-78f7c85f7ea5\" (UID: \"dc6e0fa6-24eb-4547-b2e8-78f7c85f7ea5\") "
Sep 10 15:52:15.824565 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:15.824552    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc6e0fa6-24eb-4547-b2e8-78f7c85f7ea5-tls-certs\") pod \"dc6e0fa6-24eb-4547-b2e8-78f7c85f7ea5\" (UID: \"dc6e0fa6-24eb-4547-b2e8-78f7c85f7ea5\") "
Sep 10 15:52:15.824808 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:15.824679    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc6e0fa6-24eb-4547-b2e8-78f7c85f7ea5-tmp-dir\") pod \"dc6e0fa6-24eb-4547-b2e8-78f7c85f7ea5\" (UID: \"dc6e0fa6-24eb-4547-b2e8-78f7c85f7ea5\") "
Sep 10 15:52:15.824808 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:15.824701    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc6e0fa6-24eb-4547-b2e8-78f7c85f7ea5-home" (OuterVolumeSpecName: "home") pod "dc6e0fa6-24eb-4547-b2e8-78f7c85f7ea5" (UID: "dc6e0fa6-24eb-4547-b2e8-78f7c85f7ea5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:52:15.824808 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:15.824732    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc6e0fa6-24eb-4547-b2e8-78f7c85f7ea5-model-cache" (OuterVolumeSpecName: "model-cache") pod "dc6e0fa6-24eb-4547-b2e8-78f7c85f7ea5" (UID: "dc6e0fa6-24eb-4547-b2e8-78f7c85f7ea5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:52:15.825020 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:15.824997    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc6e0fa6-24eb-4547-b2e8-78f7c85f7ea5-home\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:52:15.825178 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:15.825027    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc6e0fa6-24eb-4547-b2e8-78f7c85f7ea5-model-cache\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:52:15.825178 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:15.824993    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc6e0fa6-24eb-4547-b2e8-78f7c85f7ea5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "dc6e0fa6-24eb-4547-b2e8-78f7c85f7ea5" (UID: "dc6e0fa6-24eb-4547-b2e8-78f7c85f7ea5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:52:15.826758 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:15.826732    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc6e0fa6-24eb-4547-b2e8-78f7c85f7ea5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dc6e0fa6-24eb-4547-b2e8-78f7c85f7ea5" (UID: "dc6e0fa6-24eb-4547-b2e8-78f7c85f7ea5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:52:15.826758 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:15.826741    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc6e0fa6-24eb-4547-b2e8-78f7c85f7ea5-dshm" (OuterVolumeSpecName: "dshm") pod "dc6e0fa6-24eb-4547-b2e8-78f7c85f7ea5" (UID: "dc6e0fa6-24eb-4547-b2e8-78f7c85f7ea5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:52:15.826879 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:15.826750    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc6e0fa6-24eb-4547-b2e8-78f7c85f7ea5-kube-api-access-wznsj" (OuterVolumeSpecName: "kube-api-access-wznsj") pod "dc6e0fa6-24eb-4547-b2e8-78f7c85f7ea5" (UID: "dc6e0fa6-24eb-4547-b2e8-78f7c85f7ea5"). InnerVolumeSpecName "kube-api-access-wznsj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:52:15.870621 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:15.839749    2559 generic.go:356] "Generic (PLEG): container finished" podID="dc6e0fa6-24eb-4547-b2e8-78f7c85f7ea5" containerID="fc77f0a140a7c57d649562e254c0acf104a2d086923e1db009c9ebeaedaa1413" exitCode=0
Sep 10 15:52:15.870621 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:15.840797    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-3650b0/sub-test-model-3650b0-kserve-8bc5c5779-sdxmr"
Sep 10 15:52:15.870621 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:15.848073    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-3650b0/sub-test-model-3650b0-kserve-8bc5c5779-sdxmr" event={"ID":"dc6e0fa6-24eb-4547-b2e8-78f7c85f7ea5","Type":"ContainerDied","Data":"fc77f0a140a7c57d649562e254c0acf104a2d086923e1db009c9ebeaedaa1413"}
Sep 10 15:52:15.870621 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:15.848117    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-3650b0/sub-test-model-3650b0-kserve-8bc5c5779-sdxmr" event={"ID":"dc6e0fa6-24eb-4547-b2e8-78f7c85f7ea5","Type":"ContainerDied","Data":"1f4faeee671c047b4066dcb6d5109953e44ef9cdaca99e1334bb16f9dd5b7383"}
Sep 10 15:52:15.870621 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:15.848140    2559 scope.go:117] "RemoveContainer" containerID="fc77f0a140a7c57d649562e254c0acf104a2d086923e1db009c9ebeaedaa1413"
Sep 10 15:52:15.875040 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:15.874924    2559 scope.go:117] "RemoveContainer" containerID="fc77f0a140a7c57d649562e254c0acf104a2d086923e1db009c9ebeaedaa1413"
Sep 10 15:52:15.875377 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:52:15.875304    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc77f0a140a7c57d649562e254c0acf104a2d086923e1db009c9ebeaedaa1413\": container with ID starting with fc77f0a140a7c57d649562e254c0acf104a2d086923e1db009c9ebeaedaa1413 not found: ID does not exist" containerID="fc77f0a140a7c57d649562e254c0acf104a2d086923e1db009c9ebeaedaa1413"
Sep 10 15:52:15.875377 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:15.875339    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc77f0a140a7c57d649562e254c0acf104a2d086923e1db009c9ebeaedaa1413"} err="failed to get container status \"fc77f0a140a7c57d649562e254c0acf104a2d086923e1db009c9ebeaedaa1413\": rpc error: code = NotFound desc = could not find container \"fc77f0a140a7c57d649562e254c0acf104a2d086923e1db009c9ebeaedaa1413\": container with ID starting with fc77f0a140a7c57d649562e254c0acf104a2d086923e1db009c9ebeaedaa1413 not found: ID does not exist"
Sep 10 15:52:15.884244 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:15.884218    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-c7257b/rate-test-model-c7257b-kserve-64f78d4b67-vmzm8"
Sep 10 15:52:15.896001 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:15.895975    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3650b0/sub-test-model-3650b0-kserve-8bc5c5779-sdxmr"]
Sep 10 15:52:15.910485 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:15.904541    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3650b0/sub-test-model-3650b0-kserve-8bc5c5779-sdxmr"]
Sep 10 15:52:15.925850 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:15.925829    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wznsj\" (UniqueName: \"kubernetes.io/projected/dc6e0fa6-24eb-4547-b2e8-78f7c85f7ea5-kube-api-access-wznsj\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:52:15.925850 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:15.925849    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc6e0fa6-24eb-4547-b2e8-78f7c85f7ea5-dshm\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:52:15.926047 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:15.925859    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc6e0fa6-24eb-4547-b2e8-78f7c85f7ea5-tls-certs\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:52:15.926047 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:15.925867    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc6e0fa6-24eb-4547-b2e8-78f7c85f7ea5-tmp-dir\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:52:16.640353 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:16.640317    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-59c481-openshift-default-779dc66c4b-d55nm"
Sep 10 15:52:16.785966 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:16.785929    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68b779b4-60c6-468e-b6e4-de95dda02bf1" path="/var/lib/kubelet/pods/68b779b4-60c6-468e-b6e4-de95dda02bf1/volumes"
Sep 10 15:52:16.786360 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:16.786347    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc6e0fa6-24eb-4547-b2e8-78f7c85f7ea5" path="/var/lib/kubelet/pods/dc6e0fa6-24eb-4547-b2e8-78f7c85f7ea5/volumes"
Sep 10 15:52:16.794076 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:16.794052    2559 patch_prober.go:28] interesting pod/e2e-api-b-59c481-openshift-default-779dc66c4b-d55nm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Sep 10 15:52:16.794147 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:16.794100    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-59c481-openshift-default-779dc66c4b-d55nm" podUID="06dbe419-7af5-4fcf-84d2-a14d5a89b104" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Sep 10 15:52:17.798441 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:17.798394    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-59c481-openshift-default-779dc66c4b-d55nm"
Sep 10 15:52:17.799312 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:17.799291    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-59c481-openshift-default-779dc66c4b-d55nm"
Sep 10 15:52:24.808098 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:24.808068    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-3650b0/rate-test-model-3650b0-kserve-56495c57bc-vbl7q"]
Sep 10 15:52:24.808693 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:24.808641    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68b779b4-60c6-468e-b6e4-de95dda02bf1" containerName="main"
Sep 10 15:52:24.808693 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:24.808661    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="68b779b4-60c6-468e-b6e4-de95dda02bf1" containerName="main"
Sep 10 15:52:24.808826 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:24.808698    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc6e0fa6-24eb-4547-b2e8-78f7c85f7ea5" containerName="main"
Sep 10 15:52:24.808826 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:24.808706    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc6e0fa6-24eb-4547-b2e8-78f7c85f7ea5" containerName="main"
Sep 10 15:52:24.808826 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:24.808810    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="68b779b4-60c6-468e-b6e4-de95dda02bf1" containerName="main"
Sep 10 15:52:24.808826 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:24.808823    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc6e0fa6-24eb-4547-b2e8-78f7c85f7ea5" containerName="main"
Sep 10 15:52:24.813395 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:24.813374    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-3650b0/rate-test-model-3650b0-kserve-56495c57bc-vbl7q"
Sep 10 15:52:24.816226 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:24.816203    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-3650b0\"/\"openshift-service-ca.crt\""
Sep 10 15:52:24.817173 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:24.817141    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-3650b0\"/\"default-dockercfg-jk265\""
Sep 10 15:52:24.817173 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:24.817156    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-3650b0\"/\"kube-root-ca.crt\""
Sep 10 15:52:24.817338 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:24.817143    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-3650b0\"/\"rate-test-model-3650b0-kserve-self-signed-certs\""
Sep 10 15:52:24.823208 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:24.823188    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3650b0/rate-test-model-3650b0-kserve-56495c57bc-vbl7q"]
Sep 10 15:52:24.919988 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:24.919949    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e64c2b7-5e44-439f-a859-66060de27075-tls-certs\") pod \"rate-test-model-3650b0-kserve-56495c57bc-vbl7q\" (UID: \"0e64c2b7-5e44-439f-a859-66060de27075\") " pod="ai-tenant-e2e-shared-b-w3-3650b0/rate-test-model-3650b0-kserve-56495c57bc-vbl7q"
Sep 10 15:52:24.920158 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:24.920089    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e64c2b7-5e44-439f-a859-66060de27075-home\") pod \"rate-test-model-3650b0-kserve-56495c57bc-vbl7q\" (UID: \"0e64c2b7-5e44-439f-a859-66060de27075\") " pod="ai-tenant-e2e-shared-b-w3-3650b0/rate-test-model-3650b0-kserve-56495c57bc-vbl7q"
Sep 10 15:52:24.920158 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:24.920106    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e64c2b7-5e44-439f-a859-66060de27075-model-cache\") pod \"rate-test-model-3650b0-kserve-56495c57bc-vbl7q\" (UID: \"0e64c2b7-5e44-439f-a859-66060de27075\") " pod="ai-tenant-e2e-shared-b-w3-3650b0/rate-test-model-3650b0-kserve-56495c57bc-vbl7q"
Sep 10 15:52:24.920158 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:24.920122    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e64c2b7-5e44-439f-a859-66060de27075-dshm\") pod \"rate-test-model-3650b0-kserve-56495c57bc-vbl7q\" (UID: \"0e64c2b7-5e44-439f-a859-66060de27075\") " pod="ai-tenant-e2e-shared-b-w3-3650b0/rate-test-model-3650b0-kserve-56495c57bc-vbl7q"
Sep 10 15:52:24.920158 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:24.920144    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e64c2b7-5e44-439f-a859-66060de27075-tmp-dir\") pod \"rate-test-model-3650b0-kserve-56495c57bc-vbl7q\" (UID: \"0e64c2b7-5e44-439f-a859-66060de27075\") " pod="ai-tenant-e2e-shared-b-w3-3650b0/rate-test-model-3650b0-kserve-56495c57bc-vbl7q"
Sep 10 15:52:24.920370 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:24.920353    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sh6gj\" (UniqueName: \"kubernetes.io/projected/0e64c2b7-5e44-439f-a859-66060de27075-kube-api-access-sh6gj\") pod \"rate-test-model-3650b0-kserve-56495c57bc-vbl7q\" (UID: \"0e64c2b7-5e44-439f-a859-66060de27075\") " pod="ai-tenant-e2e-shared-b-w3-3650b0/rate-test-model-3650b0-kserve-56495c57bc-vbl7q"
Sep 10 15:52:25.021359 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:25.021320    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sh6gj\" (UniqueName: \"kubernetes.io/projected/0e64c2b7-5e44-439f-a859-66060de27075-kube-api-access-sh6gj\") pod \"rate-test-model-3650b0-kserve-56495c57bc-vbl7q\" (UID: \"0e64c2b7-5e44-439f-a859-66060de27075\") " pod="ai-tenant-e2e-shared-b-w3-3650b0/rate-test-model-3650b0-kserve-56495c57bc-vbl7q"
Sep 10 15:52:25.021543 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:25.021374    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e64c2b7-5e44-439f-a859-66060de27075-tls-certs\") pod \"rate-test-model-3650b0-kserve-56495c57bc-vbl7q\" (UID: \"0e64c2b7-5e44-439f-a859-66060de27075\") " pod="ai-tenant-e2e-shared-b-w3-3650b0/rate-test-model-3650b0-kserve-56495c57bc-vbl7q"
Sep 10 15:52:25.021543 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:25.021500    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e64c2b7-5e44-439f-a859-66060de27075-home\") pod \"rate-test-model-3650b0-kserve-56495c57bc-vbl7q\" (UID: \"0e64c2b7-5e44-439f-a859-66060de27075\") " pod="ai-tenant-e2e-shared-b-w3-3650b0/rate-test-model-3650b0-kserve-56495c57bc-vbl7q"
Sep 10 15:52:25.021543 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:25.021523    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e64c2b7-5e44-439f-a859-66060de27075-model-cache\") pod \"rate-test-model-3650b0-kserve-56495c57bc-vbl7q\" (UID: \"0e64c2b7-5e44-439f-a859-66060de27075\") " pod="ai-tenant-e2e-shared-b-w3-3650b0/rate-test-model-3650b0-kserve-56495c57bc-vbl7q"
Sep 10 15:52:25.021748 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:25.021547    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e64c2b7-5e44-439f-a859-66060de27075-dshm\") pod \"rate-test-model-3650b0-kserve-56495c57bc-vbl7q\" (UID: \"0e64c2b7-5e44-439f-a859-66060de27075\") " pod="ai-tenant-e2e-shared-b-w3-3650b0/rate-test-model-3650b0-kserve-56495c57bc-vbl7q"
Sep 10 15:52:25.021748 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:25.021621    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e64c2b7-5e44-439f-a859-66060de27075-tmp-dir\") pod \"rate-test-model-3650b0-kserve-56495c57bc-vbl7q\" (UID: \"0e64c2b7-5e44-439f-a859-66060de27075\") " pod="ai-tenant-e2e-shared-b-w3-3650b0/rate-test-model-3650b0-kserve-56495c57bc-vbl7q"
Sep 10 15:52:25.022000 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:25.021975    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e64c2b7-5e44-439f-a859-66060de27075-home\") pod \"rate-test-model-3650b0-kserve-56495c57bc-vbl7q\" (UID: \"0e64c2b7-5e44-439f-a859-66060de27075\") " pod="ai-tenant-e2e-shared-b-w3-3650b0/rate-test-model-3650b0-kserve-56495c57bc-vbl7q"
Sep 10 15:52:25.022076 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:25.022011    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e64c2b7-5e44-439f-a859-66060de27075-tmp-dir\") pod \"rate-test-model-3650b0-kserve-56495c57bc-vbl7q\" (UID: \"0e64c2b7-5e44-439f-a859-66060de27075\") " pod="ai-tenant-e2e-shared-b-w3-3650b0/rate-test-model-3650b0-kserve-56495c57bc-vbl7q"
Sep 10 15:52:25.022076 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:25.022023    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e64c2b7-5e44-439f-a859-66060de27075-model-cache\") pod \"rate-test-model-3650b0-kserve-56495c57bc-vbl7q\" (UID: \"0e64c2b7-5e44-439f-a859-66060de27075\") " pod="ai-tenant-e2e-shared-b-w3-3650b0/rate-test-model-3650b0-kserve-56495c57bc-vbl7q"
Sep 10 15:52:25.023714 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:25.023694    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e64c2b7-5e44-439f-a859-66060de27075-dshm\") pod \"rate-test-model-3650b0-kserve-56495c57bc-vbl7q\" (UID: \"0e64c2b7-5e44-439f-a859-66060de27075\") " pod="ai-tenant-e2e-shared-b-w3-3650b0/rate-test-model-3650b0-kserve-56495c57bc-vbl7q"
Sep 10 15:52:25.023890 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:25.023875    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e64c2b7-5e44-439f-a859-66060de27075-tls-certs\") pod \"rate-test-model-3650b0-kserve-56495c57bc-vbl7q\" (UID: \"0e64c2b7-5e44-439f-a859-66060de27075\") " pod="ai-tenant-e2e-shared-b-w3-3650b0/rate-test-model-3650b0-kserve-56495c57bc-vbl7q"
Sep 10 15:52:25.030843 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:25.030823    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sh6gj\" (UniqueName: \"kubernetes.io/projected/0e64c2b7-5e44-439f-a859-66060de27075-kube-api-access-sh6gj\") pod \"rate-test-model-3650b0-kserve-56495c57bc-vbl7q\" (UID: \"0e64c2b7-5e44-439f-a859-66060de27075\") " pod="ai-tenant-e2e-shared-b-w3-3650b0/rate-test-model-3650b0-kserve-56495c57bc-vbl7q"
Sep 10 15:52:25.123229 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:25.123143    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-3650b0/rate-test-model-3650b0-kserve-56495c57bc-vbl7q"
Sep 10 15:52:25.399789 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:25.399758    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3650b0/rate-test-model-3650b0-kserve-56495c57bc-vbl7q"]
Sep 10 15:52:25.401419 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:52:25.401389    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0e64c2b7_5e44_439f_a859_66060de27075.slice/crio-c6d913e6f67f901c3dd29166f7dae9195f5557332513d61d6da8441032f92d09 WatchSource:0}: Error finding container c6d913e6f67f901c3dd29166f7dae9195f5557332513d61d6da8441032f92d09: Status 404 returned error can't find the container with id c6d913e6f67f901c3dd29166f7dae9195f5557332513d61d6da8441032f92d09
Sep 10 15:52:25.903104 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:25.903068    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-3650b0/rate-test-model-3650b0-kserve-56495c57bc-vbl7q" event={"ID":"0e64c2b7-5e44-439f-a859-66060de27075","Type":"ContainerStarted","Data":"2f688365f9628941816823971732f65004930bb42c76118e482f003e640b678b"}
Sep 10 15:52:25.903562 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:25.903112    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-3650b0/rate-test-model-3650b0-kserve-56495c57bc-vbl7q" event={"ID":"0e64c2b7-5e44-439f-a859-66060de27075","Type":"ContainerStarted","Data":"c6d913e6f67f901c3dd29166f7dae9195f5557332513d61d6da8441032f92d09"}
Sep 10 15:52:25.925744 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:25.925702    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-3650b0/rate-test-model-3650b0-kserve-56495c57bc-vbl7q" podStartSLOduration=1.925686987 podStartE2EDuration="1.925686987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:52:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:52:25.92339524 +0000 UTC m=+1521.665360344" watchObservedRunningTime="2026-09-10 15:52:25.925686987 +0000 UTC m=+1521.667652090"
Sep 10 15:52:33.810460 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:33.810229    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bs7hd"]
Sep 10 15:52:35.123444 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:35.123409    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-3650b0/rate-test-model-3650b0-kserve-56495c57bc-vbl7q"
Sep 10 15:52:35.123895 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:35.123460    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-3650b0/rate-test-model-3650b0-kserve-56495c57bc-vbl7q"
Sep 10 15:52:35.142268 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:35.142228    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-3650b0/rate-test-model-3650b0-kserve-56495c57bc-vbl7q"
Sep 10 15:52:35.971774 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:35.971745    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-3650b0/rate-test-model-3650b0-kserve-56495c57bc-vbl7q"
Sep 10 15:52:48.103900 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:48.103862    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bs7hd"]
Sep 10 15:52:58.363237 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:58.363203    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-59c481-59447d66b7-dwcq9"]
Sep 10 15:52:58.363893 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:58.363425    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-59c481-59447d66b7-dwcq9" podUID="2a130f21-4148-4f20-aa48-8447a2a34082" containerName="maas-api" containerID="cri-o://3a7d40a9ab5d203a6383205683677d507a1f75808987d956d76965ac5c7e2e6b" gracePeriod=30
Sep 10 15:52:58.609914 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:58.609884    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-59c481-59447d66b7-dwcq9"
Sep 10 15:52:58.747433 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:58.746544    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a130f21-4148-4f20-aa48-8447a2a34082-openshift-service-ca\") pod \"2a130f21-4148-4f20-aa48-8447a2a34082\" (UID: \"2a130f21-4148-4f20-aa48-8447a2a34082\") "
Sep 10 15:52:58.747433 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:58.746679    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a130f21-4148-4f20-aa48-8447a2a34082-metrics-certs\") pod \"2a130f21-4148-4f20-aa48-8447a2a34082\" (UID: \"2a130f21-4148-4f20-aa48-8447a2a34082\") "
Sep 10 15:52:58.747433 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:58.746711    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a130f21-4148-4f20-aa48-8447a2a34082-maas-api-tls\") pod \"2a130f21-4148-4f20-aa48-8447a2a34082\" (UID: \"2a130f21-4148-4f20-aa48-8447a2a34082\") "
Sep 10 15:52:58.747433 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:58.746749    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4t7pd\" (UniqueName: \"kubernetes.io/projected/2a130f21-4148-4f20-aa48-8447a2a34082-kube-api-access-4t7pd\") pod \"2a130f21-4148-4f20-aa48-8447a2a34082\" (UID: \"2a130f21-4148-4f20-aa48-8447a2a34082\") "
Sep 10 15:52:58.747433 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:58.746935    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a130f21-4148-4f20-aa48-8447a2a34082-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2a130f21-4148-4f20-aa48-8447a2a34082" (UID: "2a130f21-4148-4f20-aa48-8447a2a34082"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:52:58.747433 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:58.747145    2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a130f21-4148-4f20-aa48-8447a2a34082-openshift-service-ca\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:52:58.750154 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:58.750107    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a130f21-4148-4f20-aa48-8447a2a34082-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "2a130f21-4148-4f20-aa48-8447a2a34082" (UID: "2a130f21-4148-4f20-aa48-8447a2a34082"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:52:58.750281 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:58.750190    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a130f21-4148-4f20-aa48-8447a2a34082-kube-api-access-4t7pd" (OuterVolumeSpecName: "kube-api-access-4t7pd") pod "2a130f21-4148-4f20-aa48-8447a2a34082" (UID: "2a130f21-4148-4f20-aa48-8447a2a34082"). InnerVolumeSpecName "kube-api-access-4t7pd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:52:58.751416 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:58.751389    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a130f21-4148-4f20-aa48-8447a2a34082-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2a130f21-4148-4f20-aa48-8447a2a34082" (UID: "2a130f21-4148-4f20-aa48-8447a2a34082"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:52:58.848257 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:58.848225    2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a130f21-4148-4f20-aa48-8447a2a34082-metrics-certs\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:52:58.848257 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:58.848250    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a130f21-4148-4f20-aa48-8447a2a34082-maas-api-tls\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:52:58.848257 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:58.848261    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4t7pd\" (UniqueName: \"kubernetes.io/projected/2a130f21-4148-4f20-aa48-8447a2a34082-kube-api-access-4t7pd\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:52:59.089793 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:59.089701    2559 generic.go:356] "Generic (PLEG): container finished" podID="2a130f21-4148-4f20-aa48-8447a2a34082" containerID="3a7d40a9ab5d203a6383205683677d507a1f75808987d956d76965ac5c7e2e6b" exitCode=0
Sep 10 15:52:59.089793 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:59.089751    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-59c481-59447d66b7-dwcq9"
Sep 10 15:52:59.089980 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:59.089789    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-59c481-59447d66b7-dwcq9" event={"ID":"2a130f21-4148-4f20-aa48-8447a2a34082","Type":"ContainerDied","Data":"3a7d40a9ab5d203a6383205683677d507a1f75808987d956d76965ac5c7e2e6b"}
Sep 10 15:52:59.089980 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:59.089845    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-59c481-59447d66b7-dwcq9" event={"ID":"2a130f21-4148-4f20-aa48-8447a2a34082","Type":"ContainerDied","Data":"5b46fcc568d59d7095ffbf28fc8b32601dca62ab09466c576f7eb62721053079"}
Sep 10 15:52:59.089980 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:59.089869    2559 scope.go:117] "RemoveContainer" containerID="3a7d40a9ab5d203a6383205683677d507a1f75808987d956d76965ac5c7e2e6b"
Sep 10 15:52:59.099364 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:59.099339    2559 scope.go:117] "RemoveContainer" containerID="3a7d40a9ab5d203a6383205683677d507a1f75808987d956d76965ac5c7e2e6b"
Sep 10 15:52:59.099621 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:52:59.099599    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a7d40a9ab5d203a6383205683677d507a1f75808987d956d76965ac5c7e2e6b\": container with ID starting with 3a7d40a9ab5d203a6383205683677d507a1f75808987d956d76965ac5c7e2e6b not found: ID does not exist" containerID="3a7d40a9ab5d203a6383205683677d507a1f75808987d956d76965ac5c7e2e6b"
Sep 10 15:52:59.099706 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:59.099634    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a7d40a9ab5d203a6383205683677d507a1f75808987d956d76965ac5c7e2e6b"} err="failed to get container status \"3a7d40a9ab5d203a6383205683677d507a1f75808987d956d76965ac5c7e2e6b\": rpc error: code = NotFound desc = could not find container \"3a7d40a9ab5d203a6383205683677d507a1f75808987d956d76965ac5c7e2e6b\": container with ID starting with 3a7d40a9ab5d203a6383205683677d507a1f75808987d956d76965ac5c7e2e6b not found: ID does not exist"
Sep 10 15:52:59.112894 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:59.112855    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-59c481-59447d66b7-dwcq9"]
Sep 10 15:52:59.115335 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:52:59.115312    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-59c481-59447d66b7-dwcq9"]
Sep 10 15:53:00.785354 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:00.785320    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a130f21-4148-4f20-aa48-8447a2a34082" path="/var/lib/kubelet/pods/2a130f21-4148-4f20-aa48-8447a2a34082/volumes"
Sep 10 15:53:02.612723 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:02.612685    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bs7hd"]
Sep 10 15:53:05.999054 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:05.999017    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-59c481-openshift-default-779dc66c4b-d55nm"]
Sep 10 15:53:05.999546 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:05.999235    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-59c481-openshift-default-779dc66c4b-d55nm" podUID="06dbe419-7af5-4fcf-84d2-a14d5a89b104" containerName="istio-proxy" containerID="cri-o://e13ea3d4d7e4252a7dc6cbbdac19197008ec6da590ee97fe0afc19d47793235a" gracePeriod=30
Sep 10 15:53:11.161486 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:11.161450    2559 generic.go:356] "Generic (PLEG): container finished" podID="06dbe419-7af5-4fcf-84d2-a14d5a89b104" containerID="e13ea3d4d7e4252a7dc6cbbdac19197008ec6da590ee97fe0afc19d47793235a" exitCode=0
Sep 10 15:53:11.161914 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:11.161521    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-59c481-openshift-default-779dc66c4b-d55nm" event={"ID":"06dbe419-7af5-4fcf-84d2-a14d5a89b104","Type":"ContainerDied","Data":"e13ea3d4d7e4252a7dc6cbbdac19197008ec6da590ee97fe0afc19d47793235a"}
Sep 10 15:53:11.161914 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:11.161609    2559 scope.go:117] "RemoveContainer" containerID="374faaf00fbf35fe4c0d6b1fcce6c5461341e66984544186311e85c8e70ba16f"
Sep 10 15:53:11.257777 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:11.257752    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-59c481-openshift-default-779dc66c4b-d55nm"
Sep 10 15:53:11.372443 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:11.372350    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06dbe419-7af5-4fcf-84d2-a14d5a89b104-credential-socket\") pod \"06dbe419-7af5-4fcf-84d2-a14d5a89b104\" (UID: \"06dbe419-7af5-4fcf-84d2-a14d5a89b104\") "
Sep 10 15:53:11.372443 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:11.372391    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-52zvz\" (UniqueName: \"kubernetes.io/projected/06dbe419-7af5-4fcf-84d2-a14d5a89b104-kube-api-access-52zvz\") pod \"06dbe419-7af5-4fcf-84d2-a14d5a89b104\" (UID: \"06dbe419-7af5-4fcf-84d2-a14d5a89b104\") "
Sep 10 15:53:11.372443 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:11.372426    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06dbe419-7af5-4fcf-84d2-a14d5a89b104-istio-token\") pod \"06dbe419-7af5-4fcf-84d2-a14d5a89b104\" (UID: \"06dbe419-7af5-4fcf-84d2-a14d5a89b104\") "
Sep 10 15:53:11.372770 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:11.372458    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06dbe419-7af5-4fcf-84d2-a14d5a89b104-workload-certs\") pod \"06dbe419-7af5-4fcf-84d2-a14d5a89b104\" (UID: \"06dbe419-7af5-4fcf-84d2-a14d5a89b104\") "
Sep 10 15:53:11.372770 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:11.372486    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06dbe419-7af5-4fcf-84d2-a14d5a89b104-istiod-ca-cert\") pod \"06dbe419-7af5-4fcf-84d2-a14d5a89b104\" (UID: \"06dbe419-7af5-4fcf-84d2-a14d5a89b104\") "
Sep 10 15:53:11.372770 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:11.372607    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06dbe419-7af5-4fcf-84d2-a14d5a89b104-istio-data\") pod \"06dbe419-7af5-4fcf-84d2-a14d5a89b104\" (UID: \"06dbe419-7af5-4fcf-84d2-a14d5a89b104\") "
Sep 10 15:53:11.372770 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:11.372644    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06dbe419-7af5-4fcf-84d2-a14d5a89b104-istio-envoy\") pod \"06dbe419-7af5-4fcf-84d2-a14d5a89b104\" (UID: \"06dbe419-7af5-4fcf-84d2-a14d5a89b104\") "
Sep 10 15:53:11.372770 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:11.372674    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06dbe419-7af5-4fcf-84d2-a14d5a89b104-istio-podinfo\") pod \"06dbe419-7af5-4fcf-84d2-a14d5a89b104\" (UID: \"06dbe419-7af5-4fcf-84d2-a14d5a89b104\") "
Sep 10 15:53:11.372770 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:11.372740    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06dbe419-7af5-4fcf-84d2-a14d5a89b104-workload-socket\") pod \"06dbe419-7af5-4fcf-84d2-a14d5a89b104\" (UID: \"06dbe419-7af5-4fcf-84d2-a14d5a89b104\") "
Sep 10 15:53:11.373078 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:11.372763    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06dbe419-7af5-4fcf-84d2-a14d5a89b104-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "06dbe419-7af5-4fcf-84d2-a14d5a89b104" (UID: "06dbe419-7af5-4fcf-84d2-a14d5a89b104"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:53:11.373078 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:11.372802    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06dbe419-7af5-4fcf-84d2-a14d5a89b104-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "06dbe419-7af5-4fcf-84d2-a14d5a89b104" (UID: "06dbe419-7af5-4fcf-84d2-a14d5a89b104"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:53:11.373305 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:11.373264    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06dbe419-7af5-4fcf-84d2-a14d5a89b104-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "06dbe419-7af5-4fcf-84d2-a14d5a89b104" (UID: "06dbe419-7af5-4fcf-84d2-a14d5a89b104"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:53:11.373305 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:11.373278    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06dbe419-7af5-4fcf-84d2-a14d5a89b104-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "06dbe419-7af5-4fcf-84d2-a14d5a89b104" (UID: "06dbe419-7af5-4fcf-84d2-a14d5a89b104"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:53:11.373305 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:11.373088    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06dbe419-7af5-4fcf-84d2-a14d5a89b104-istio-data" (OuterVolumeSpecName: "istio-data") pod "06dbe419-7af5-4fcf-84d2-a14d5a89b104" (UID: "06dbe419-7af5-4fcf-84d2-a14d5a89b104"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:53:11.373562 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:11.373456    2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06dbe419-7af5-4fcf-84d2-a14d5a89b104-workload-certs\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:53:11.373562 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:11.373474    2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06dbe419-7af5-4fcf-84d2-a14d5a89b104-istiod-ca-cert\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:53:11.373562 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:11.373484    2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06dbe419-7af5-4fcf-84d2-a14d5a89b104-istio-data\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:53:11.373562 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:11.373492    2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06dbe419-7af5-4fcf-84d2-a14d5a89b104-workload-socket\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:53:11.373562 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:11.373499    2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06dbe419-7af5-4fcf-84d2-a14d5a89b104-credential-socket\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:53:11.374865 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:11.374831    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06dbe419-7af5-4fcf-84d2-a14d5a89b104-istio-token" (OuterVolumeSpecName: "istio-token") pod "06dbe419-7af5-4fcf-84d2-a14d5a89b104" (UID: "06dbe419-7af5-4fcf-84d2-a14d5a89b104"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:53:11.374992 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:11.374969    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06dbe419-7af5-4fcf-84d2-a14d5a89b104-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "06dbe419-7af5-4fcf-84d2-a14d5a89b104" (UID: "06dbe419-7af5-4fcf-84d2-a14d5a89b104"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:53:11.375137 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:11.375115    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06dbe419-7af5-4fcf-84d2-a14d5a89b104-kube-api-access-52zvz" (OuterVolumeSpecName: "kube-api-access-52zvz") pod "06dbe419-7af5-4fcf-84d2-a14d5a89b104" (UID: "06dbe419-7af5-4fcf-84d2-a14d5a89b104"). InnerVolumeSpecName "kube-api-access-52zvz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:53:11.375255 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:11.375238    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/06dbe419-7af5-4fcf-84d2-a14d5a89b104-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "06dbe419-7af5-4fcf-84d2-a14d5a89b104" (UID: "06dbe419-7af5-4fcf-84d2-a14d5a89b104"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 15:53:11.474511 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:11.474477    2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06dbe419-7af5-4fcf-84d2-a14d5a89b104-istio-envoy\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:53:11.474511 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:11.474503    2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06dbe419-7af5-4fcf-84d2-a14d5a89b104-istio-podinfo\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:53:11.474511 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:11.474513    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52zvz\" (UniqueName: \"kubernetes.io/projected/06dbe419-7af5-4fcf-84d2-a14d5a89b104-kube-api-access-52zvz\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:53:11.474761 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:11.474522    2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06dbe419-7af5-4fcf-84d2-a14d5a89b104-istio-token\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:53:12.168507 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:12.168480    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-59c481-openshift-default-779dc66c4b-d55nm"
Sep 10 15:53:12.168963 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:12.168475    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-59c481-openshift-default-779dc66c4b-d55nm" event={"ID":"06dbe419-7af5-4fcf-84d2-a14d5a89b104","Type":"ContainerDied","Data":"2c1d89d0b5b83b8efc27c34daf5b2d68cde2e31df23b651ced29427a206a3f46"}
Sep 10 15:53:12.168963 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:12.168631    2559 scope.go:117] "RemoveContainer" containerID="e13ea3d4d7e4252a7dc6cbbdac19197008ec6da590ee97fe0afc19d47793235a"
Sep 10 15:53:12.194271 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:12.194246    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-59c481-openshift-default-779dc66c4b-d55nm"]
Sep 10 15:53:12.199325 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:12.199304    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-59c481-openshift-default-779dc66c4b-d55nm"]
Sep 10 15:53:12.791967 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:12.791931    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06dbe419-7af5-4fcf-84d2-a14d5a89b104" path="/var/lib/kubelet/pods/06dbe419-7af5-4fcf-84d2-a14d5a89b104/volumes"
Sep 10 15:53:21.402102 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:21.402072    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bs7hd"]
Sep 10 15:53:23.985821 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:23.985789    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-30d70c-openshift-default-786f7fff56-8sqxw"]
Sep 10 15:53:23.986249 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:23.985999    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-30d70c-openshift-default-786f7fff56-8sqxw" podUID="0b514c76-21e4-43ac-a01f-9f5a44f3e773" containerName="istio-proxy" containerID="cri-o://1329e99e26268019e553eabd003d841bad0da5a2dff6c15e0c8f4ff52392aead" gracePeriod=30
Sep 10 15:53:27.675722 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:27.675685    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-c7257b/rate-test-model-c7257b-kserve-64f78d4b67-vmzm8"]
Sep 10 15:53:28.043152 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:28.043057    2559 patch_prober.go:28] interesting pod/e2e-shared-a-w2-30d70c-openshift-default-786f7fff56-8sqxw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 10 15:53:28.043152 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:28.043138    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-30d70c-openshift-default-786f7fff56-8sqxw" podUID="0b514c76-21e4-43ac-a01f-9f5a44f3e773" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 10 15:53:28.479675 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:28.479638    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3650b0/rate-test-model-3650b0-kserve-56495c57bc-vbl7q"]
Sep 10 15:53:29.247316 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:29.247248    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-30d70c-openshift-default-786f7fff56-8sqxw"
Sep 10 15:53:29.279042 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:29.279010    2559 generic.go:356] "Generic (PLEG): container finished" podID="0b514c76-21e4-43ac-a01f-9f5a44f3e773" containerID="1329e99e26268019e553eabd003d841bad0da5a2dff6c15e0c8f4ff52392aead" exitCode=0
Sep 10 15:53:29.279214 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:29.279063    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-30d70c-openshift-default-786f7fff56-8sqxw"
Sep 10 15:53:29.279214 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:29.279162    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-30d70c-openshift-default-786f7fff56-8sqxw" event={"ID":"0b514c76-21e4-43ac-a01f-9f5a44f3e773","Type":"ContainerDied","Data":"1329e99e26268019e553eabd003d841bad0da5a2dff6c15e0c8f4ff52392aead"}
Sep 10 15:53:29.279214 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:29.279202    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-30d70c-openshift-default-786f7fff56-8sqxw" event={"ID":"0b514c76-21e4-43ac-a01f-9f5a44f3e773","Type":"ContainerDied","Data":"942f4883f80c1503675f6fcd7e7866e9c6f8fe15e785e38e1ac60602d42aaab3"}
Sep 10 15:53:29.279456 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:29.279228    2559 scope.go:117] "RemoveContainer" containerID="1329e99e26268019e553eabd003d841bad0da5a2dff6c15e0c8f4ff52392aead"
Sep 10 15:53:29.288298 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:29.288276    2559 scope.go:117] "RemoveContainer" containerID="1227fdcb190e4d6a9429b01f358d7474501e53fdcf1f85e2e87dcabc326add10"
Sep 10 15:53:29.296549 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:29.296531    2559 scope.go:117] "RemoveContainer" containerID="1329e99e26268019e553eabd003d841bad0da5a2dff6c15e0c8f4ff52392aead"
Sep 10 15:53:29.296841 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:53:29.296825    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1329e99e26268019e553eabd003d841bad0da5a2dff6c15e0c8f4ff52392aead\": container with ID starting with 1329e99e26268019e553eabd003d841bad0da5a2dff6c15e0c8f4ff52392aead not found: ID does not exist" containerID="1329e99e26268019e553eabd003d841bad0da5a2dff6c15e0c8f4ff52392aead"
Sep 10 15:53:29.296896 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:29.296850    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1329e99e26268019e553eabd003d841bad0da5a2dff6c15e0c8f4ff52392aead"} err="failed to get container status \"1329e99e26268019e553eabd003d841bad0da5a2dff6c15e0c8f4ff52392aead\": rpc error: code = NotFound desc = could not find container \"1329e99e26268019e553eabd003d841bad0da5a2dff6c15e0c8f4ff52392aead\": container with ID starting with 1329e99e26268019e553eabd003d841bad0da5a2dff6c15e0c8f4ff52392aead not found: ID does not exist"
Sep 10 15:53:29.296896 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:29.296865    2559 scope.go:117] "RemoveContainer" containerID="1227fdcb190e4d6a9429b01f358d7474501e53fdcf1f85e2e87dcabc326add10"
Sep 10 15:53:29.297158 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:53:29.297137    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1227fdcb190e4d6a9429b01f358d7474501e53fdcf1f85e2e87dcabc326add10\": container with ID starting with 1227fdcb190e4d6a9429b01f358d7474501e53fdcf1f85e2e87dcabc326add10 not found: ID does not exist" containerID="1227fdcb190e4d6a9429b01f358d7474501e53fdcf1f85e2e87dcabc326add10"
Sep 10 15:53:29.297217 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:29.297168    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1227fdcb190e4d6a9429b01f358d7474501e53fdcf1f85e2e87dcabc326add10"} err="failed to get container status \"1227fdcb190e4d6a9429b01f358d7474501e53fdcf1f85e2e87dcabc326add10\": rpc error: code = NotFound desc = could not find container \"1227fdcb190e4d6a9429b01f358d7474501e53fdcf1f85e2e87dcabc326add10\": container with ID starting with 1227fdcb190e4d6a9429b01f358d7474501e53fdcf1f85e2e87dcabc326add10 not found: ID does not exist"
Sep 10 15:53:29.334343 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:29.334314    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b514c76-21e4-43ac-a01f-9f5a44f3e773-workload-socket\") pod \"0b514c76-21e4-43ac-a01f-9f5a44f3e773\" (UID: \"0b514c76-21e4-43ac-a01f-9f5a44f3e773\") "
Sep 10 15:53:29.334517 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:29.334361    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b514c76-21e4-43ac-a01f-9f5a44f3e773-istio-data\") pod \"0b514c76-21e4-43ac-a01f-9f5a44f3e773\" (UID: \"0b514c76-21e4-43ac-a01f-9f5a44f3e773\") "
Sep 10 15:53:29.334517 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:29.334458    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b514c76-21e4-43ac-a01f-9f5a44f3e773-istiod-ca-cert\") pod \"0b514c76-21e4-43ac-a01f-9f5a44f3e773\" (UID: \"0b514c76-21e4-43ac-a01f-9f5a44f3e773\") "
Sep 10 15:53:29.334517 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:29.334487    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b514c76-21e4-43ac-a01f-9f5a44f3e773-istio-podinfo\") pod \"0b514c76-21e4-43ac-a01f-9f5a44f3e773\" (UID: \"0b514c76-21e4-43ac-a01f-9f5a44f3e773\") "
Sep 10 15:53:29.334517 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:29.334516    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8d8zn\" (UniqueName: \"kubernetes.io/projected/0b514c76-21e4-43ac-a01f-9f5a44f3e773-kube-api-access-8d8zn\") pod \"0b514c76-21e4-43ac-a01f-9f5a44f3e773\" (UID: \"0b514c76-21e4-43ac-a01f-9f5a44f3e773\") "
Sep 10 15:53:29.334792 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:29.334536    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b514c76-21e4-43ac-a01f-9f5a44f3e773-istio-envoy\") pod \"0b514c76-21e4-43ac-a01f-9f5a44f3e773\" (UID: \"0b514c76-21e4-43ac-a01f-9f5a44f3e773\") "
Sep 10 15:53:29.334792 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:29.334558    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b514c76-21e4-43ac-a01f-9f5a44f3e773-workload-certs\") pod \"0b514c76-21e4-43ac-a01f-9f5a44f3e773\" (UID: \"0b514c76-21e4-43ac-a01f-9f5a44f3e773\") "
Sep 10 15:53:29.334792 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:29.334627    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b514c76-21e4-43ac-a01f-9f5a44f3e773-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0b514c76-21e4-43ac-a01f-9f5a44f3e773" (UID: "0b514c76-21e4-43ac-a01f-9f5a44f3e773"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:53:29.334792 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:29.334634    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b514c76-21e4-43ac-a01f-9f5a44f3e773-credential-socket\") pod \"0b514c76-21e4-43ac-a01f-9f5a44f3e773\" (UID: \"0b514c76-21e4-43ac-a01f-9f5a44f3e773\") "
Sep 10 15:53:29.334792 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:29.334678    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b514c76-21e4-43ac-a01f-9f5a44f3e773-istio-data" (OuterVolumeSpecName: "istio-data") pod "0b514c76-21e4-43ac-a01f-9f5a44f3e773" (UID: "0b514c76-21e4-43ac-a01f-9f5a44f3e773"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:53:29.334792 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:29.334713    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b514c76-21e4-43ac-a01f-9f5a44f3e773-istio-token\") pod \"0b514c76-21e4-43ac-a01f-9f5a44f3e773\" (UID: \"0b514c76-21e4-43ac-a01f-9f5a44f3e773\") "
Sep 10 15:53:29.335067 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:29.334850    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b514c76-21e4-43ac-a01f-9f5a44f3e773-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0b514c76-21e4-43ac-a01f-9f5a44f3e773" (UID: "0b514c76-21e4-43ac-a01f-9f5a44f3e773"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:53:29.335067 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:29.334897    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b514c76-21e4-43ac-a01f-9f5a44f3e773-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0b514c76-21e4-43ac-a01f-9f5a44f3e773" (UID: "0b514c76-21e4-43ac-a01f-9f5a44f3e773"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:53:29.335067 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:29.334919    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b514c76-21e4-43ac-a01f-9f5a44f3e773-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0b514c76-21e4-43ac-a01f-9f5a44f3e773" (UID: "0b514c76-21e4-43ac-a01f-9f5a44f3e773"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:53:29.335180 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:29.335102    2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b514c76-21e4-43ac-a01f-9f5a44f3e773-workload-certs\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:53:29.335180 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:29.335120    2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b514c76-21e4-43ac-a01f-9f5a44f3e773-credential-socket\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:53:29.335180 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:29.335135    2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b514c76-21e4-43ac-a01f-9f5a44f3e773-workload-socket\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:53:29.335180 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:29.335148    2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b514c76-21e4-43ac-a01f-9f5a44f3e773-istio-data\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:53:29.335180 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:29.335160    2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b514c76-21e4-43ac-a01f-9f5a44f3e773-istiod-ca-cert\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:53:29.336823 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:29.336781    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0b514c76-21e4-43ac-a01f-9f5a44f3e773-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0b514c76-21e4-43ac-a01f-9f5a44f3e773" (UID: "0b514c76-21e4-43ac-a01f-9f5a44f3e773"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 15:53:29.336823 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:29.336800    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b514c76-21e4-43ac-a01f-9f5a44f3e773-istio-token" (OuterVolumeSpecName: "istio-token") pod "0b514c76-21e4-43ac-a01f-9f5a44f3e773" (UID: "0b514c76-21e4-43ac-a01f-9f5a44f3e773"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:53:29.337095 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:29.337073    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b514c76-21e4-43ac-a01f-9f5a44f3e773-kube-api-access-8d8zn" (OuterVolumeSpecName: "kube-api-access-8d8zn") pod "0b514c76-21e4-43ac-a01f-9f5a44f3e773" (UID: "0b514c76-21e4-43ac-a01f-9f5a44f3e773"). InnerVolumeSpecName "kube-api-access-8d8zn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:53:29.337187 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:29.337075    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b514c76-21e4-43ac-a01f-9f5a44f3e773-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0b514c76-21e4-43ac-a01f-9f5a44f3e773" (UID: "0b514c76-21e4-43ac-a01f-9f5a44f3e773"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:53:29.436426 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:29.436400    2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b514c76-21e4-43ac-a01f-9f5a44f3e773-istio-podinfo\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:53:29.436426 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:29.436422    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8d8zn\" (UniqueName: \"kubernetes.io/projected/0b514c76-21e4-43ac-a01f-9f5a44f3e773-kube-api-access-8d8zn\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:53:29.436426 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:29.436431    2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b514c76-21e4-43ac-a01f-9f5a44f3e773-istio-envoy\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:53:29.436645 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:29.436440    2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b514c76-21e4-43ac-a01f-9f5a44f3e773-istio-token\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:53:29.607939 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:29.607902    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-30d70c-openshift-default-786f7fff56-8sqxw"]
Sep 10 15:53:29.613389 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:29.613362    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-30d70c-openshift-default-786f7fff56-8sqxw"]
Sep 10 15:53:30.784946 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:30.784909    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b514c76-21e4-43ac-a01f-9f5a44f3e773" path="/var/lib/kubelet/pods/0b514c76-21e4-43ac-a01f-9f5a44f3e773/volumes"
Sep 10 15:53:34.400729 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:34.400700    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ec08ac-5c45cc5d-zr65m"]
Sep 10 15:53:34.401187 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:34.400933    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ec08ac-5c45cc5d-zr65m" podUID="5cc8c332-932a-4bc8-9bc4-5483c84f4a3d" containerName="maas-api" containerID="cri-o://ddd9cff474768d9e2beb515aae0cf70d38ad8215b555c783fab1a3c6924ee969" gracePeriod=30
Sep 10 15:53:34.656303 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:34.656245    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ec08ac-5c45cc5d-zr65m"
Sep 10 15:53:34.733543 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:34.733507    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8cb756-7bcbfc58d8-67pz2"]
Sep 10 15:53:34.734125 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:34.734102    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b514c76-21e4-43ac-a01f-9f5a44f3e773" containerName="istio-proxy"
Sep 10 15:53:34.734187 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:34.734130    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b514c76-21e4-43ac-a01f-9f5a44f3e773" containerName="istio-proxy"
Sep 10 15:53:34.734187 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:34.734157    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b514c76-21e4-43ac-a01f-9f5a44f3e773" containerName="istio-proxy"
Sep 10 15:53:34.734187 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:34.734166    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b514c76-21e4-43ac-a01f-9f5a44f3e773" containerName="istio-proxy"
Sep 10 15:53:34.734187 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:34.734182    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5cc8c332-932a-4bc8-9bc4-5483c84f4a3d" containerName="maas-api"
Sep 10 15:53:34.734463 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:34.734192    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="5cc8c332-932a-4bc8-9bc4-5483c84f4a3d" containerName="maas-api"
Sep 10 15:53:34.734463 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:34.734219    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a130f21-4148-4f20-aa48-8447a2a34082" containerName="maas-api"
Sep 10 15:53:34.734463 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:34.734228    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a130f21-4148-4f20-aa48-8447a2a34082" containerName="maas-api"
Sep 10 15:53:34.734463 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:34.734238    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06dbe419-7af5-4fcf-84d2-a14d5a89b104" containerName="istio-proxy"
Sep 10 15:53:34.734463 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:34.734246    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="06dbe419-7af5-4fcf-84d2-a14d5a89b104" containerName="istio-proxy"
Sep 10 15:53:34.734463 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:34.734260    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06dbe419-7af5-4fcf-84d2-a14d5a89b104" containerName="istio-proxy"
Sep 10 15:53:34.734463 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:34.734269    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="06dbe419-7af5-4fcf-84d2-a14d5a89b104" containerName="istio-proxy"
Sep 10 15:53:34.734463 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:34.734365    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a130f21-4148-4f20-aa48-8447a2a34082" containerName="maas-api"
Sep 10 15:53:34.734463 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:34.734382    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="06dbe419-7af5-4fcf-84d2-a14d5a89b104" containerName="istio-proxy"
Sep 10 15:53:34.734463 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:34.734393    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="06dbe419-7af5-4fcf-84d2-a14d5a89b104" containerName="istio-proxy"
Sep 10 15:53:34.734463 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:34.734402    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b514c76-21e4-43ac-a01f-9f5a44f3e773" containerName="istio-proxy"
Sep 10 15:53:34.734463 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:34.734420    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b514c76-21e4-43ac-a01f-9f5a44f3e773" containerName="istio-proxy"
Sep 10 15:53:34.734463 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:34.734431    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="5cc8c332-932a-4bc8-9bc4-5483c84f4a3d" containerName="maas-api"
Sep 10 15:53:34.737821 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:34.737803    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8cb756-7bcbfc58d8-67pz2"
Sep 10 15:53:34.741793 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:34.741766    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-8cb756-serving-cert\""
Sep 10 15:53:34.751331 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:34.751304    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8cb756-7bcbfc58d8-67pz2"]
Sep 10 15:53:34.787559 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:34.787529    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tj65q\" (UniqueName: \"kubernetes.io/projected/5cc8c332-932a-4bc8-9bc4-5483c84f4a3d-kube-api-access-tj65q\") pod \"5cc8c332-932a-4bc8-9bc4-5483c84f4a3d\" (UID: \"5cc8c332-932a-4bc8-9bc4-5483c84f4a3d\") "
Sep 10 15:53:34.787750 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:34.787572    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cc8c332-932a-4bc8-9bc4-5483c84f4a3d-openshift-service-ca\") pod \"5cc8c332-932a-4bc8-9bc4-5483c84f4a3d\" (UID: \"5cc8c332-932a-4bc8-9bc4-5483c84f4a3d\") "
Sep 10 15:53:34.787750 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:34.787636    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5cc8c332-932a-4bc8-9bc4-5483c84f4a3d-maas-api-tls\") pod \"5cc8c332-932a-4bc8-9bc4-5483c84f4a3d\" (UID: \"5cc8c332-932a-4bc8-9bc4-5483c84f4a3d\") "
Sep 10 15:53:34.787750 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:34.787718    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cc8c332-932a-4bc8-9bc4-5483c84f4a3d-metrics-certs\") pod \"5cc8c332-932a-4bc8-9bc4-5483c84f4a3d\" (UID: \"5cc8c332-932a-4bc8-9bc4-5483c84f4a3d\") "
Sep 10 15:53:34.787937 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:34.787831    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdblp\" (UniqueName: \"kubernetes.io/projected/d252c941-0566-44bd-a79b-f433db7fbd51-kube-api-access-tdblp\") pod \"maas-api-e2e-ipp-a-8cb756-7bcbfc58d8-67pz2\" (UID: \"d252c941-0566-44bd-a79b-f433db7fbd51\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8cb756-7bcbfc58d8-67pz2"
Sep 10 15:53:34.788027 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:34.787998    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cc8c332-932a-4bc8-9bc4-5483c84f4a3d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5cc8c332-932a-4bc8-9bc4-5483c84f4a3d" (UID: "5cc8c332-932a-4bc8-9bc4-5483c84f4a3d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:53:34.788226 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:34.788198    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d252c941-0566-44bd-a79b-f433db7fbd51-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-8cb756-7bcbfc58d8-67pz2\" (UID: \"d252c941-0566-44bd-a79b-f433db7fbd51\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8cb756-7bcbfc58d8-67pz2"
Sep 10 15:53:34.788308 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:34.788260    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d252c941-0566-44bd-a79b-f433db7fbd51-metrics-certs\") pod \"maas-api-e2e-ipp-a-8cb756-7bcbfc58d8-67pz2\" (UID: \"d252c941-0566-44bd-a79b-f433db7fbd51\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8cb756-7bcbfc58d8-67pz2"
Sep 10 15:53:34.788308 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:34.788290    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d252c941-0566-44bd-a79b-f433db7fbd51-maas-api-tls\") pod \"maas-api-e2e-ipp-a-8cb756-7bcbfc58d8-67pz2\" (UID: \"d252c941-0566-44bd-a79b-f433db7fbd51\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8cb756-7bcbfc58d8-67pz2"
Sep 10 15:53:34.788437 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:34.788391    2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cc8c332-932a-4bc8-9bc4-5483c84f4a3d-openshift-service-ca\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:53:34.790041 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:34.790006    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5cc8c332-932a-4bc8-9bc4-5483c84f4a3d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "5cc8c332-932a-4bc8-9bc4-5483c84f4a3d" (UID: "5cc8c332-932a-4bc8-9bc4-5483c84f4a3d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:53:34.790214 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:34.790193    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5cc8c332-932a-4bc8-9bc4-5483c84f4a3d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5cc8c332-932a-4bc8-9bc4-5483c84f4a3d" (UID: "5cc8c332-932a-4bc8-9bc4-5483c84f4a3d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:53:34.790293 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:34.790209    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cc8c332-932a-4bc8-9bc4-5483c84f4a3d-kube-api-access-tj65q" (OuterVolumeSpecName: "kube-api-access-tj65q") pod "5cc8c332-932a-4bc8-9bc4-5483c84f4a3d" (UID: "5cc8c332-932a-4bc8-9bc4-5483c84f4a3d"). InnerVolumeSpecName "kube-api-access-tj65q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:53:34.888881 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:34.888845    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdblp\" (UniqueName: \"kubernetes.io/projected/d252c941-0566-44bd-a79b-f433db7fbd51-kube-api-access-tdblp\") pod \"maas-api-e2e-ipp-a-8cb756-7bcbfc58d8-67pz2\" (UID: \"d252c941-0566-44bd-a79b-f433db7fbd51\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8cb756-7bcbfc58d8-67pz2"
Sep 10 15:53:34.889092 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:34.888902    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d252c941-0566-44bd-a79b-f433db7fbd51-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-8cb756-7bcbfc58d8-67pz2\" (UID: \"d252c941-0566-44bd-a79b-f433db7fbd51\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8cb756-7bcbfc58d8-67pz2"
Sep 10 15:53:34.889092 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:34.888938    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d252c941-0566-44bd-a79b-f433db7fbd51-metrics-certs\") pod \"maas-api-e2e-ipp-a-8cb756-7bcbfc58d8-67pz2\" (UID: \"d252c941-0566-44bd-a79b-f433db7fbd51\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8cb756-7bcbfc58d8-67pz2"
Sep 10 15:53:34.889092 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:34.888958    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d252c941-0566-44bd-a79b-f433db7fbd51-maas-api-tls\") pod \"maas-api-e2e-ipp-a-8cb756-7bcbfc58d8-67pz2\" (UID: \"d252c941-0566-44bd-a79b-f433db7fbd51\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8cb756-7bcbfc58d8-67pz2"
Sep 10 15:53:34.889092 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:34.889057    2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cc8c332-932a-4bc8-9bc4-5483c84f4a3d-metrics-certs\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:53:34.889092 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:34.889072    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tj65q\" (UniqueName: \"kubernetes.io/projected/5cc8c332-932a-4bc8-9bc4-5483c84f4a3d-kube-api-access-tj65q\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:53:34.889092 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:34.889087    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5cc8c332-932a-4bc8-9bc4-5483c84f4a3d-maas-api-tls\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:53:34.889688 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:34.889654    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d252c941-0566-44bd-a79b-f433db7fbd51-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-8cb756-7bcbfc58d8-67pz2\" (UID: \"d252c941-0566-44bd-a79b-f433db7fbd51\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8cb756-7bcbfc58d8-67pz2"
Sep 10 15:53:34.891560 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:34.891538    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d252c941-0566-44bd-a79b-f433db7fbd51-metrics-certs\") pod \"maas-api-e2e-ipp-a-8cb756-7bcbfc58d8-67pz2\" (UID: \"d252c941-0566-44bd-a79b-f433db7fbd51\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8cb756-7bcbfc58d8-67pz2"
Sep 10 15:53:34.891660 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:34.891627    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d252c941-0566-44bd-a79b-f433db7fbd51-maas-api-tls\") pod \"maas-api-e2e-ipp-a-8cb756-7bcbfc58d8-67pz2\" (UID: \"d252c941-0566-44bd-a79b-f433db7fbd51\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8cb756-7bcbfc58d8-67pz2"
Sep 10 15:53:34.897397 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:34.897373    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdblp\" (UniqueName: \"kubernetes.io/projected/d252c941-0566-44bd-a79b-f433db7fbd51-kube-api-access-tdblp\") pod \"maas-api-e2e-ipp-a-8cb756-7bcbfc58d8-67pz2\" (UID: \"d252c941-0566-44bd-a79b-f433db7fbd51\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8cb756-7bcbfc58d8-67pz2"
Sep 10 15:53:35.055446 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:35.055411    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8cb756-7bcbfc58d8-67pz2"
Sep 10 15:53:35.314903 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:35.314818    2559 generic.go:356] "Generic (PLEG): container finished" podID="5cc8c332-932a-4bc8-9bc4-5483c84f4a3d" containerID="ddd9cff474768d9e2beb515aae0cf70d38ad8215b555c783fab1a3c6924ee969" exitCode=0
Sep 10 15:53:35.314903 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:35.314878    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ec08ac-5c45cc5d-zr65m" event={"ID":"5cc8c332-932a-4bc8-9bc4-5483c84f4a3d","Type":"ContainerDied","Data":"ddd9cff474768d9e2beb515aae0cf70d38ad8215b555c783fab1a3c6924ee969"}
Sep 10 15:53:35.314903 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:35.314893    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ec08ac-5c45cc5d-zr65m"
Sep 10 15:53:35.315197 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:35.314960    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ec08ac-5c45cc5d-zr65m" event={"ID":"5cc8c332-932a-4bc8-9bc4-5483c84f4a3d","Type":"ContainerDied","Data":"581a0d2f3cc88e2087be10660b6d04856a0d70999e77ecc83d80db4b97b1ad28"}
Sep 10 15:53:35.315197 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:35.314991    2559 scope.go:117] "RemoveContainer" containerID="ddd9cff474768d9e2beb515aae0cf70d38ad8215b555c783fab1a3c6924ee969"
Sep 10 15:53:35.325316 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:35.325299    2559 scope.go:117] "RemoveContainer" containerID="ddd9cff474768d9e2beb515aae0cf70d38ad8215b555c783fab1a3c6924ee969"
Sep 10 15:53:35.325591 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:53:35.325561    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddd9cff474768d9e2beb515aae0cf70d38ad8215b555c783fab1a3c6924ee969\": container with ID starting with ddd9cff474768d9e2beb515aae0cf70d38ad8215b555c783fab1a3c6924ee969 not found: ID does not exist" containerID="ddd9cff474768d9e2beb515aae0cf70d38ad8215b555c783fab1a3c6924ee969"
Sep 10 15:53:35.325642 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:35.325601    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddd9cff474768d9e2beb515aae0cf70d38ad8215b555c783fab1a3c6924ee969"} err="failed to get container status \"ddd9cff474768d9e2beb515aae0cf70d38ad8215b555c783fab1a3c6924ee969\": rpc error: code = NotFound desc = could not find container \"ddd9cff474768d9e2beb515aae0cf70d38ad8215b555c783fab1a3c6924ee969\": container with ID starting with ddd9cff474768d9e2beb515aae0cf70d38ad8215b555c783fab1a3c6924ee969 not found: ID does not exist"
Sep 10 15:53:35.331211 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:35.331191    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8cb756-7bcbfc58d8-67pz2"]
Sep 10 15:53:35.332650 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:53:35.332628    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd252c941_0566_44bd_a79b_f433db7fbd51.slice/crio-6e8478df7e95ed471484dd177f6a637dfe9b53c74db73cd32953130c213296c9 WatchSource:0}: Error finding container 6e8478df7e95ed471484dd177f6a637dfe9b53c74db73cd32953130c213296c9: Status 404 returned error can't find the container with id 6e8478df7e95ed471484dd177f6a637dfe9b53c74db73cd32953130c213296c9
Sep 10 15:53:35.339201 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:35.339180    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ec08ac-5c45cc5d-zr65m"]
Sep 10 15:53:35.347777 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:35.347755    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ec08ac-5c45cc5d-zr65m"]
Sep 10 15:53:36.325629 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:36.325592    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8cb756-7bcbfc58d8-67pz2" event={"ID":"d252c941-0566-44bd-a79b-f433db7fbd51","Type":"ContainerStarted","Data":"550d7b331d1f7e068741830a52e36fcbba90ad849abcd43dea73c0c9d1a1ad87"}
Sep 10 15:53:36.326070 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:36.325653    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8cb756-7bcbfc58d8-67pz2" event={"ID":"d252c941-0566-44bd-a79b-f433db7fbd51","Type":"ContainerStarted","Data":"6e8478df7e95ed471484dd177f6a637dfe9b53c74db73cd32953130c213296c9"}
Sep 10 15:53:36.326070 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:36.325707    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8cb756-7bcbfc58d8-67pz2"
Sep 10 15:53:36.344778 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:36.344731    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8cb756-7bcbfc58d8-67pz2" podStartSLOduration=1.905075071 podStartE2EDuration="2.344716333s" totalImagesPullingTime="439.641262ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:53:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:53:36.343235324 +0000 UTC m=+1592.085200428" watchObservedRunningTime="2026-09-10 15:53:36.344716333 +0000 UTC m=+1592.086681435"
Sep 10 15:53:36.785522 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:36.785493    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5cc8c332-932a-4bc8-9bc4-5483c84f4a3d" path="/var/lib/kubelet/pods/5cc8c332-932a-4bc8-9bc4-5483c84f4a3d/volumes"
Sep 10 15:53:39.118795 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:39.118764    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bs7hd"]
Sep 10 15:53:41.263841 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:41.263803    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ec08ac-openshift-default-55ff657f88-tddkk"]
Sep 10 15:53:41.264323 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:41.264091    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-ec08ac-openshift-default-55ff657f88-tddkk" podUID="23f9cdab-2ba9-4e1b-a322-7b6d5b09409c" containerName="istio-proxy" containerID="cri-o://9711ff3626e63d4681c61fe7b21d19cf1f693d84e4c5305517924ce8b7175d8f" gracePeriod=30
Sep 10 15:53:42.336441 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:42.336404    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8cb756-7bcbfc58d8-67pz2"
Sep 10 15:53:42.695874 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:42.695834    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-c7257b/rate-test-model-c7257b-kserve-64f78d4b67-vmzm8" podUID="36b69e2d-fc8c-459a-b70e-7828ab9edb18" containerName="main" containerID="cri-o://97707a6080f2d30b2a479fa85db7766f6806acba96cc1c604c2ca8e88c03745e" gracePeriod=45
Sep 10 15:53:42.946296 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:42.946233    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-c7257b/rate-test-model-c7257b-kserve-64f78d4b67-vmzm8"
Sep 10 15:53:43.061898 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:43.061865    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36b69e2d-fc8c-459a-b70e-7828ab9edb18-tls-certs\") pod \"36b69e2d-fc8c-459a-b70e-7828ab9edb18\" (UID: \"36b69e2d-fc8c-459a-b70e-7828ab9edb18\") "
Sep 10 15:53:43.062071 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:43.061933    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36b69e2d-fc8c-459a-b70e-7828ab9edb18-model-cache\") pod \"36b69e2d-fc8c-459a-b70e-7828ab9edb18\" (UID: \"36b69e2d-fc8c-459a-b70e-7828ab9edb18\") "
Sep 10 15:53:43.062071 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:43.061956    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36b69e2d-fc8c-459a-b70e-7828ab9edb18-home\") pod \"36b69e2d-fc8c-459a-b70e-7828ab9edb18\" (UID: \"36b69e2d-fc8c-459a-b70e-7828ab9edb18\") "
Sep 10 15:53:43.062071 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:43.062000    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36b69e2d-fc8c-459a-b70e-7828ab9edb18-tmp-dir\") pod \"36b69e2d-fc8c-459a-b70e-7828ab9edb18\" (UID: \"36b69e2d-fc8c-459a-b70e-7828ab9edb18\") "
Sep 10 15:53:43.062071 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:43.062027    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xkrdc\" (UniqueName: \"kubernetes.io/projected/36b69e2d-fc8c-459a-b70e-7828ab9edb18-kube-api-access-xkrdc\") pod \"36b69e2d-fc8c-459a-b70e-7828ab9edb18\" (UID: \"36b69e2d-fc8c-459a-b70e-7828ab9edb18\") "
Sep 10 15:53:43.062240 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:43.062178    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36b69e2d-fc8c-459a-b70e-7828ab9edb18-dshm\") pod \"36b69e2d-fc8c-459a-b70e-7828ab9edb18\" (UID: \"36b69e2d-fc8c-459a-b70e-7828ab9edb18\") "
Sep 10 15:53:43.062295 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:43.062239    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36b69e2d-fc8c-459a-b70e-7828ab9edb18-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "36b69e2d-fc8c-459a-b70e-7828ab9edb18" (UID: "36b69e2d-fc8c-459a-b70e-7828ab9edb18"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:53:43.062295 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:43.062257    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36b69e2d-fc8c-459a-b70e-7828ab9edb18-model-cache" (OuterVolumeSpecName: "model-cache") pod "36b69e2d-fc8c-459a-b70e-7828ab9edb18" (UID: "36b69e2d-fc8c-459a-b70e-7828ab9edb18"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:53:43.062295 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:43.062281    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36b69e2d-fc8c-459a-b70e-7828ab9edb18-home" (OuterVolumeSpecName: "home") pod "36b69e2d-fc8c-459a-b70e-7828ab9edb18" (UID: "36b69e2d-fc8c-459a-b70e-7828ab9edb18"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:53:43.062530 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:43.062514    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36b69e2d-fc8c-459a-b70e-7828ab9edb18-model-cache\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:53:43.062594 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:43.062531    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36b69e2d-fc8c-459a-b70e-7828ab9edb18-home\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:53:43.062594 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:43.062544    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36b69e2d-fc8c-459a-b70e-7828ab9edb18-tmp-dir\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:53:43.064107 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:43.064083    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36b69e2d-fc8c-459a-b70e-7828ab9edb18-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "36b69e2d-fc8c-459a-b70e-7828ab9edb18" (UID: "36b69e2d-fc8c-459a-b70e-7828ab9edb18"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:53:43.064205 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:43.064124    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36b69e2d-fc8c-459a-b70e-7828ab9edb18-kube-api-access-xkrdc" (OuterVolumeSpecName: "kube-api-access-xkrdc") pod "36b69e2d-fc8c-459a-b70e-7828ab9edb18" (UID: "36b69e2d-fc8c-459a-b70e-7828ab9edb18"). InnerVolumeSpecName "kube-api-access-xkrdc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:53:43.064205 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:43.064152    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36b69e2d-fc8c-459a-b70e-7828ab9edb18-dshm" (OuterVolumeSpecName: "dshm") pod "36b69e2d-fc8c-459a-b70e-7828ab9edb18" (UID: "36b69e2d-fc8c-459a-b70e-7828ab9edb18"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:53:43.163282 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:43.163246    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36b69e2d-fc8c-459a-b70e-7828ab9edb18-dshm\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:53:43.163282 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:43.163276    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36b69e2d-fc8c-459a-b70e-7828ab9edb18-tls-certs\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:53:43.163282 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:43.163287    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkrdc\" (UniqueName: \"kubernetes.io/projected/36b69e2d-fc8c-459a-b70e-7828ab9edb18-kube-api-access-xkrdc\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:53:43.367234 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:43.367142    2559 generic.go:356] "Generic (PLEG): container finished" podID="36b69e2d-fc8c-459a-b70e-7828ab9edb18" containerID="97707a6080f2d30b2a479fa85db7766f6806acba96cc1c604c2ca8e88c03745e" exitCode=0
Sep 10 15:53:43.367234 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:43.367192    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-c7257b/rate-test-model-c7257b-kserve-64f78d4b67-vmzm8"
Sep 10 15:53:43.367674 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:43.367231    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-c7257b/rate-test-model-c7257b-kserve-64f78d4b67-vmzm8" event={"ID":"36b69e2d-fc8c-459a-b70e-7828ab9edb18","Type":"ContainerDied","Data":"97707a6080f2d30b2a479fa85db7766f6806acba96cc1c604c2ca8e88c03745e"}
Sep 10 15:53:43.367674 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:43.367277    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-c7257b/rate-test-model-c7257b-kserve-64f78d4b67-vmzm8" event={"ID":"36b69e2d-fc8c-459a-b70e-7828ab9edb18","Type":"ContainerDied","Data":"2149eb06e9aa2afe24e53fa25ac636b5426d2c33316eebc98f92e3f3729005d0"}
Sep 10 15:53:43.367674 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:43.367296    2559 scope.go:117] "RemoveContainer" containerID="97707a6080f2d30b2a479fa85db7766f6806acba96cc1c604c2ca8e88c03745e"
Sep 10 15:53:43.376504 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:43.376462    2559 scope.go:117] "RemoveContainer" containerID="97707a6080f2d30b2a479fa85db7766f6806acba96cc1c604c2ca8e88c03745e"
Sep 10 15:53:43.376773 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:53:43.376755    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97707a6080f2d30b2a479fa85db7766f6806acba96cc1c604c2ca8e88c03745e\": container with ID starting with 97707a6080f2d30b2a479fa85db7766f6806acba96cc1c604c2ca8e88c03745e not found: ID does not exist" containerID="97707a6080f2d30b2a479fa85db7766f6806acba96cc1c604c2ca8e88c03745e"
Sep 10 15:53:43.376820 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:43.376782    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97707a6080f2d30b2a479fa85db7766f6806acba96cc1c604c2ca8e88c03745e"} err="failed to get container status \"97707a6080f2d30b2a479fa85db7766f6806acba96cc1c604c2ca8e88c03745e\": rpc error: code = NotFound desc = could not find container \"97707a6080f2d30b2a479fa85db7766f6806acba96cc1c604c2ca8e88c03745e\": container with ID starting with 97707a6080f2d30b2a479fa85db7766f6806acba96cc1c604c2ca8e88c03745e not found: ID does not exist"
Sep 10 15:53:43.389910 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:43.389867    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-c7257b/rate-test-model-c7257b-kserve-64f78d4b67-vmzm8"]
Sep 10 15:53:43.394252 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:43.394229    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-c7257b/rate-test-model-c7257b-kserve-64f78d4b67-vmzm8"]
Sep 10 15:53:43.498974 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:43.498929    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-3650b0/rate-test-model-3650b0-kserve-56495c57bc-vbl7q" podUID="0e64c2b7-5e44-439f-a859-66060de27075" containerName="main" containerID="cri-o://2f688365f9628941816823971732f65004930bb42c76118e482f003e640b678b" gracePeriod=45
Sep 10 15:53:43.750257 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:43.750231    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-3650b0/rate-test-model-3650b0-kserve-56495c57bc-vbl7q"
Sep 10 15:53:43.869593 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:43.869545    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sh6gj\" (UniqueName: \"kubernetes.io/projected/0e64c2b7-5e44-439f-a859-66060de27075-kube-api-access-sh6gj\") pod \"0e64c2b7-5e44-439f-a859-66060de27075\" (UID: \"0e64c2b7-5e44-439f-a859-66060de27075\") "
Sep 10 15:53:43.869795 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:43.869625    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e64c2b7-5e44-439f-a859-66060de27075-tls-certs\") pod \"0e64c2b7-5e44-439f-a859-66060de27075\" (UID: \"0e64c2b7-5e44-439f-a859-66060de27075\") "
Sep 10 15:53:43.869795 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:43.869710    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e64c2b7-5e44-439f-a859-66060de27075-tmp-dir\") pod \"0e64c2b7-5e44-439f-a859-66060de27075\" (UID: \"0e64c2b7-5e44-439f-a859-66060de27075\") "
Sep 10 15:53:43.869795 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:43.869771    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e64c2b7-5e44-439f-a859-66060de27075-model-cache\") pod \"0e64c2b7-5e44-439f-a859-66060de27075\" (UID: \"0e64c2b7-5e44-439f-a859-66060de27075\") "
Sep 10 15:53:43.869976 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:43.869796    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e64c2b7-5e44-439f-a859-66060de27075-dshm\") pod \"0e64c2b7-5e44-439f-a859-66060de27075\" (UID: \"0e64c2b7-5e44-439f-a859-66060de27075\") "
Sep 10 15:53:43.869976 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:43.869822    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e64c2b7-5e44-439f-a859-66060de27075-home\") pod \"0e64c2b7-5e44-439f-a859-66060de27075\" (UID: \"0e64c2b7-5e44-439f-a859-66060de27075\") "
Sep 10 15:53:43.870096 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:43.869974    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e64c2b7-5e44-439f-a859-66060de27075-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0e64c2b7-5e44-439f-a859-66060de27075" (UID: "0e64c2b7-5e44-439f-a859-66060de27075"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:53:43.870096 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:43.870002    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e64c2b7-5e44-439f-a859-66060de27075-model-cache" (OuterVolumeSpecName: "model-cache") pod "0e64c2b7-5e44-439f-a859-66060de27075" (UID: "0e64c2b7-5e44-439f-a859-66060de27075"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:53:43.870201 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:43.870125    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e64c2b7-5e44-439f-a859-66060de27075-home" (OuterVolumeSpecName: "home") pod "0e64c2b7-5e44-439f-a859-66060de27075" (UID: "0e64c2b7-5e44-439f-a859-66060de27075"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:53:43.870272 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:43.870253    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e64c2b7-5e44-439f-a859-66060de27075-model-cache\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:53:43.870328 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:43.870275    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e64c2b7-5e44-439f-a859-66060de27075-home\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:53:43.870328 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:43.870288    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e64c2b7-5e44-439f-a859-66060de27075-tmp-dir\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:53:43.871908 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:43.871870    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e64c2b7-5e44-439f-a859-66060de27075-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0e64c2b7-5e44-439f-a859-66060de27075" (UID: "0e64c2b7-5e44-439f-a859-66060de27075"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:53:43.871908 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:43.871885    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e64c2b7-5e44-439f-a859-66060de27075-dshm" (OuterVolumeSpecName: "dshm") pod "0e64c2b7-5e44-439f-a859-66060de27075" (UID: "0e64c2b7-5e44-439f-a859-66060de27075"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:53:43.871908 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:43.871890    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e64c2b7-5e44-439f-a859-66060de27075-kube-api-access-sh6gj" (OuterVolumeSpecName: "kube-api-access-sh6gj") pod "0e64c2b7-5e44-439f-a859-66060de27075" (UID: "0e64c2b7-5e44-439f-a859-66060de27075"). InnerVolumeSpecName "kube-api-access-sh6gj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:53:43.971619 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:43.971505    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e64c2b7-5e44-439f-a859-66060de27075-dshm\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:53:43.971619 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:43.971536    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sh6gj\" (UniqueName: \"kubernetes.io/projected/0e64c2b7-5e44-439f-a859-66060de27075-kube-api-access-sh6gj\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:53:43.971619 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:43.971549    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e64c2b7-5e44-439f-a859-66060de27075-tls-certs\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:53:44.374087 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:44.373987    2559 generic.go:356] "Generic (PLEG): container finished" podID="0e64c2b7-5e44-439f-a859-66060de27075" containerID="2f688365f9628941816823971732f65004930bb42c76118e482f003e640b678b" exitCode=0
Sep 10 15:53:44.374087 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:44.374035    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-3650b0/rate-test-model-3650b0-kserve-56495c57bc-vbl7q"
Sep 10 15:53:44.374711 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:44.374080    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-3650b0/rate-test-model-3650b0-kserve-56495c57bc-vbl7q" event={"ID":"0e64c2b7-5e44-439f-a859-66060de27075","Type":"ContainerDied","Data":"2f688365f9628941816823971732f65004930bb42c76118e482f003e640b678b"}
Sep 10 15:53:44.374711 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:44.374134    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-3650b0/rate-test-model-3650b0-kserve-56495c57bc-vbl7q" event={"ID":"0e64c2b7-5e44-439f-a859-66060de27075","Type":"ContainerDied","Data":"c6d913e6f67f901c3dd29166f7dae9195f5557332513d61d6da8441032f92d09"}
Sep 10 15:53:44.374711 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:44.374158    2559 scope.go:117] "RemoveContainer" containerID="2f688365f9628941816823971732f65004930bb42c76118e482f003e640b678b"
Sep 10 15:53:44.383375 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:44.383359    2559 scope.go:117] "RemoveContainer" containerID="2f688365f9628941816823971732f65004930bb42c76118e482f003e640b678b"
Sep 10 15:53:44.383639 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:53:44.383621    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f688365f9628941816823971732f65004930bb42c76118e482f003e640b678b\": container with ID starting with 2f688365f9628941816823971732f65004930bb42c76118e482f003e640b678b not found: ID does not exist" containerID="2f688365f9628941816823971732f65004930bb42c76118e482f003e640b678b"
Sep 10 15:53:44.383718 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:44.383650    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f688365f9628941816823971732f65004930bb42c76118e482f003e640b678b"} err="failed to get container status \"2f688365f9628941816823971732f65004930bb42c76118e482f003e640b678b\": rpc error: code = NotFound desc = could not find container \"2f688365f9628941816823971732f65004930bb42c76118e482f003e640b678b\": container with ID starting with 2f688365f9628941816823971732f65004930bb42c76118e482f003e640b678b not found: ID does not exist"
Sep 10 15:53:44.401872 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:44.401806    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3650b0/rate-test-model-3650b0-kserve-56495c57bc-vbl7q"]
Sep 10 15:53:44.404042 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:44.404014    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3650b0/rate-test-model-3650b0-kserve-56495c57bc-vbl7q"]
Sep 10 15:53:44.786195 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:44.786161    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e64c2b7-5e44-439f-a859-66060de27075" path="/var/lib/kubelet/pods/0e64c2b7-5e44-439f-a859-66060de27075/volumes"
Sep 10 15:53:44.786594 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:44.786565    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="36b69e2d-fc8c-459a-b70e-7828ab9edb18" path="/var/lib/kubelet/pods/36b69e2d-fc8c-459a-b70e-7828ab9edb18/volumes"
Sep 10 15:53:46.388235 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:46.388206    2559 generic.go:356] "Generic (PLEG): container finished" podID="23f9cdab-2ba9-4e1b-a322-7b6d5b09409c" containerID="9711ff3626e63d4681c61fe7b21d19cf1f693d84e4c5305517924ce8b7175d8f" exitCode=0
Sep 10 15:53:46.388664 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:46.388270    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ec08ac-openshift-default-55ff657f88-tddkk" event={"ID":"23f9cdab-2ba9-4e1b-a322-7b6d5b09409c","Type":"ContainerDied","Data":"9711ff3626e63d4681c61fe7b21d19cf1f693d84e4c5305517924ce8b7175d8f"}
Sep 10 15:53:46.388664 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:46.388320    2559 scope.go:117] "RemoveContainer" containerID="ae069168d1e1d52d67ec180abbfc4217bf193db1f60ce8f9987c4867558b1f11"
Sep 10 15:53:46.522246 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:46.522217    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ec08ac-openshift-default-55ff657f88-tddkk"
Sep 10 15:53:46.594349 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:46.594258    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23f9cdab-2ba9-4e1b-a322-7b6d5b09409c-istio-podinfo\") pod \"23f9cdab-2ba9-4e1b-a322-7b6d5b09409c\" (UID: \"23f9cdab-2ba9-4e1b-a322-7b6d5b09409c\") "
Sep 10 15:53:46.594349 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:46.594305    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23f9cdab-2ba9-4e1b-a322-7b6d5b09409c-credential-socket\") pod \"23f9cdab-2ba9-4e1b-a322-7b6d5b09409c\" (UID: \"23f9cdab-2ba9-4e1b-a322-7b6d5b09409c\") "
Sep 10 15:53:46.594349 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:46.594331    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z9j75\" (UniqueName: \"kubernetes.io/projected/23f9cdab-2ba9-4e1b-a322-7b6d5b09409c-kube-api-access-z9j75\") pod \"23f9cdab-2ba9-4e1b-a322-7b6d5b09409c\" (UID: \"23f9cdab-2ba9-4e1b-a322-7b6d5b09409c\") "
Sep 10 15:53:46.594349 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:46.594349    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23f9cdab-2ba9-4e1b-a322-7b6d5b09409c-istio-envoy\") pod \"23f9cdab-2ba9-4e1b-a322-7b6d5b09409c\" (UID: \"23f9cdab-2ba9-4e1b-a322-7b6d5b09409c\") "
Sep 10 15:53:46.594723 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:46.594377    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23f9cdab-2ba9-4e1b-a322-7b6d5b09409c-istio-token\") pod \"23f9cdab-2ba9-4e1b-a322-7b6d5b09409c\" (UID: \"23f9cdab-2ba9-4e1b-a322-7b6d5b09409c\") "
Sep 10 15:53:46.594723 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:46.594423    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23f9cdab-2ba9-4e1b-a322-7b6d5b09409c-istiod-ca-cert\") pod \"23f9cdab-2ba9-4e1b-a322-7b6d5b09409c\" (UID: \"23f9cdab-2ba9-4e1b-a322-7b6d5b09409c\") "
Sep 10 15:53:46.594723 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:46.594450    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23f9cdab-2ba9-4e1b-a322-7b6d5b09409c-istio-data\") pod \"23f9cdab-2ba9-4e1b-a322-7b6d5b09409c\" (UID: \"23f9cdab-2ba9-4e1b-a322-7b6d5b09409c\") "
Sep 10 15:53:46.594723 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:46.594514    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23f9cdab-2ba9-4e1b-a322-7b6d5b09409c-workload-certs\") pod \"23f9cdab-2ba9-4e1b-a322-7b6d5b09409c\" (UID: \"23f9cdab-2ba9-4e1b-a322-7b6d5b09409c\") "
Sep 10 15:53:46.594723 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:46.594544    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23f9cdab-2ba9-4e1b-a322-7b6d5b09409c-workload-socket\") pod \"23f9cdab-2ba9-4e1b-a322-7b6d5b09409c\" (UID: \"23f9cdab-2ba9-4e1b-a322-7b6d5b09409c\") "
Sep 10 15:53:46.594723 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:46.594603    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23f9cdab-2ba9-4e1b-a322-7b6d5b09409c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "23f9cdab-2ba9-4e1b-a322-7b6d5b09409c" (UID: "23f9cdab-2ba9-4e1b-a322-7b6d5b09409c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:53:46.595036 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:46.594789    2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23f9cdab-2ba9-4e1b-a322-7b6d5b09409c-credential-socket\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:53:46.595036 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:46.594851    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23f9cdab-2ba9-4e1b-a322-7b6d5b09409c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "23f9cdab-2ba9-4e1b-a322-7b6d5b09409c" (UID: "23f9cdab-2ba9-4e1b-a322-7b6d5b09409c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:53:46.595036 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:46.595013    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23f9cdab-2ba9-4e1b-a322-7b6d5b09409c-istio-data" (OuterVolumeSpecName: "istio-data") pod "23f9cdab-2ba9-4e1b-a322-7b6d5b09409c" (UID: "23f9cdab-2ba9-4e1b-a322-7b6d5b09409c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:53:46.595154 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:46.595056    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23f9cdab-2ba9-4e1b-a322-7b6d5b09409c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "23f9cdab-2ba9-4e1b-a322-7b6d5b09409c" (UID: "23f9cdab-2ba9-4e1b-a322-7b6d5b09409c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:53:46.595310 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:46.595291    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23f9cdab-2ba9-4e1b-a322-7b6d5b09409c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "23f9cdab-2ba9-4e1b-a322-7b6d5b09409c" (UID: "23f9cdab-2ba9-4e1b-a322-7b6d5b09409c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:53:46.596689 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:46.596660    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/23f9cdab-2ba9-4e1b-a322-7b6d5b09409c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "23f9cdab-2ba9-4e1b-a322-7b6d5b09409c" (UID: "23f9cdab-2ba9-4e1b-a322-7b6d5b09409c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 15:53:46.596780 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:46.596742    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23f9cdab-2ba9-4e1b-a322-7b6d5b09409c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "23f9cdab-2ba9-4e1b-a322-7b6d5b09409c" (UID: "23f9cdab-2ba9-4e1b-a322-7b6d5b09409c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:53:46.597030 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:46.597008    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23f9cdab-2ba9-4e1b-a322-7b6d5b09409c-istio-token" (OuterVolumeSpecName: "istio-token") pod "23f9cdab-2ba9-4e1b-a322-7b6d5b09409c" (UID: "23f9cdab-2ba9-4e1b-a322-7b6d5b09409c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:53:46.597103 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:46.597042    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23f9cdab-2ba9-4e1b-a322-7b6d5b09409c-kube-api-access-z9j75" (OuterVolumeSpecName: "kube-api-access-z9j75") pod "23f9cdab-2ba9-4e1b-a322-7b6d5b09409c" (UID: "23f9cdab-2ba9-4e1b-a322-7b6d5b09409c"). InnerVolumeSpecName "kube-api-access-z9j75". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:53:46.695771 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:46.695733    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9j75\" (UniqueName: \"kubernetes.io/projected/23f9cdab-2ba9-4e1b-a322-7b6d5b09409c-kube-api-access-z9j75\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:53:46.695771 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:46.695764    2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23f9cdab-2ba9-4e1b-a322-7b6d5b09409c-istio-envoy\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:53:46.695771 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:46.695775    2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23f9cdab-2ba9-4e1b-a322-7b6d5b09409c-istio-token\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:53:46.696013 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:46.695784    2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23f9cdab-2ba9-4e1b-a322-7b6d5b09409c-istiod-ca-cert\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:53:46.696013 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:46.695794    2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23f9cdab-2ba9-4e1b-a322-7b6d5b09409c-istio-data\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:53:46.696013 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:46.695802    2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23f9cdab-2ba9-4e1b-a322-7b6d5b09409c-workload-certs\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:53:46.696013 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:46.695810    2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23f9cdab-2ba9-4e1b-a322-7b6d5b09409c-workload-socket\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:53:46.696013 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:46.695817    2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23f9cdab-2ba9-4e1b-a322-7b6d5b09409c-istio-podinfo\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:53:47.394665 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:47.394634    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ec08ac-openshift-default-55ff657f88-tddkk"
Sep 10 15:53:47.394665 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:47.394654    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ec08ac-openshift-default-55ff657f88-tddkk" event={"ID":"23f9cdab-2ba9-4e1b-a322-7b6d5b09409c","Type":"ContainerDied","Data":"d0621d61fb82187292328f8abb55843050c862cb4e99acb88bedcbf01cfa934c"}
Sep 10 15:53:47.395176 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:47.394714    2559 scope.go:117] "RemoveContainer" containerID="9711ff3626e63d4681c61fe7b21d19cf1f693d84e4c5305517924ce8b7175d8f"
Sep 10 15:53:47.415472 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:47.415441    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ec08ac-openshift-default-55ff657f88-tddkk"]
Sep 10 15:53:47.419470 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:47.419442    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ec08ac-openshift-default-55ff657f88-tddkk"]
Sep 10 15:53:48.785194 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:48.785158    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23f9cdab-2ba9-4e1b-a322-7b6d5b09409c" path="/var/lib/kubelet/pods/23f9cdab-2ba9-4e1b-a322-7b6d5b09409c/volumes"
Sep 10 15:53:58.470948 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:58.470864    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-6d929c-openshift-default-5cd456f5b6-4j7rh"]
Sep 10 15:53:58.471418 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:58.471402    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36b69e2d-fc8c-459a-b70e-7828ab9edb18" containerName="main"
Sep 10 15:53:58.471495 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:58.471421    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="36b69e2d-fc8c-459a-b70e-7828ab9edb18" containerName="main"
Sep 10 15:53:58.471495 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:58.471434    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e64c2b7-5e44-439f-a859-66060de27075" containerName="main"
Sep 10 15:53:58.471495 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:58.471443    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e64c2b7-5e44-439f-a859-66060de27075" containerName="main"
Sep 10 15:53:58.471495 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:58.471456    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23f9cdab-2ba9-4e1b-a322-7b6d5b09409c" containerName="istio-proxy"
Sep 10 15:53:58.471495 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:58.471466    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="23f9cdab-2ba9-4e1b-a322-7b6d5b09409c" containerName="istio-proxy"
Sep 10 15:53:58.471495 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:58.471491    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23f9cdab-2ba9-4e1b-a322-7b6d5b09409c" containerName="istio-proxy"
Sep 10 15:53:58.471807 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:58.471499    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="23f9cdab-2ba9-4e1b-a322-7b6d5b09409c" containerName="istio-proxy"
Sep 10 15:53:58.471807 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:58.471629    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e64c2b7-5e44-439f-a859-66060de27075" containerName="main"
Sep 10 15:53:58.471807 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:58.471641    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="36b69e2d-fc8c-459a-b70e-7828ab9edb18" containerName="main"
Sep 10 15:53:58.471807 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:58.471652    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="23f9cdab-2ba9-4e1b-a322-7b6d5b09409c" containerName="istio-proxy"
Sep 10 15:53:58.471807 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:58.471663    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="23f9cdab-2ba9-4e1b-a322-7b6d5b09409c" containerName="istio-proxy"
Sep 10 15:53:58.476226 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:58.476202    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-6d929c-openshift-default-5cd456f5b6-4j7rh"
Sep 10 15:53:58.478811 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:58.478789    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-6d929c-openshift-default-dockercfg-l9nlq\""
Sep 10 15:53:58.485966 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:58.485940    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-6d929c-openshift-default-5cd456f5b6-4j7rh"]
Sep 10 15:53:58.501399 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:58.501369    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fx4dw\" (UniqueName: \"kubernetes.io/projected/3f28530b-4e55-487a-8ed4-540d104609bb-kube-api-access-fx4dw\") pod \"e2e-ipp-b-6d929c-openshift-default-5cd456f5b6-4j7rh\" (UID: \"3f28530b-4e55-487a-8ed4-540d104609bb\") " pod="openshift-ingress/e2e-ipp-b-6d929c-openshift-default-5cd456f5b6-4j7rh"
Sep 10 15:53:58.501561 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:58.501451    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f28530b-4e55-487a-8ed4-540d104609bb-workload-certs\") pod \"e2e-ipp-b-6d929c-openshift-default-5cd456f5b6-4j7rh\" (UID: \"3f28530b-4e55-487a-8ed4-540d104609bb\") " pod="openshift-ingress/e2e-ipp-b-6d929c-openshift-default-5cd456f5b6-4j7rh"
Sep 10 15:53:58.501561 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:58.501495    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f28530b-4e55-487a-8ed4-540d104609bb-istio-podinfo\") pod \"e2e-ipp-b-6d929c-openshift-default-5cd456f5b6-4j7rh\" (UID: \"3f28530b-4e55-487a-8ed4-540d104609bb\") " pod="openshift-ingress/e2e-ipp-b-6d929c-openshift-default-5cd456f5b6-4j7rh"
Sep 10 15:53:58.501561 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:58.501517    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f28530b-4e55-487a-8ed4-540d104609bb-istio-envoy\") pod \"e2e-ipp-b-6d929c-openshift-default-5cd456f5b6-4j7rh\" (UID: \"3f28530b-4e55-487a-8ed4-540d104609bb\") " pod="openshift-ingress/e2e-ipp-b-6d929c-openshift-default-5cd456f5b6-4j7rh"
Sep 10 15:53:58.501561 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:58.501536    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f28530b-4e55-487a-8ed4-540d104609bb-istio-token\") pod \"e2e-ipp-b-6d929c-openshift-default-5cd456f5b6-4j7rh\" (UID: \"3f28530b-4e55-487a-8ed4-540d104609bb\") " pod="openshift-ingress/e2e-ipp-b-6d929c-openshift-default-5cd456f5b6-4j7rh"
Sep 10 15:53:58.501561 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:58.501553    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f28530b-4e55-487a-8ed4-540d104609bb-credential-socket\") pod \"e2e-ipp-b-6d929c-openshift-default-5cd456f5b6-4j7rh\" (UID: \"3f28530b-4e55-487a-8ed4-540d104609bb\") " pod="openshift-ingress/e2e-ipp-b-6d929c-openshift-default-5cd456f5b6-4j7rh"
Sep 10 15:53:58.501907 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:58.501593    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f28530b-4e55-487a-8ed4-540d104609bb-istio-data\") pod \"e2e-ipp-b-6d929c-openshift-default-5cd456f5b6-4j7rh\" (UID: \"3f28530b-4e55-487a-8ed4-540d104609bb\") " pod="openshift-ingress/e2e-ipp-b-6d929c-openshift-default-5cd456f5b6-4j7rh"
Sep 10 15:53:58.501907 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:58.501620    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f28530b-4e55-487a-8ed4-540d104609bb-istiod-ca-cert\") pod \"e2e-ipp-b-6d929c-openshift-default-5cd456f5b6-4j7rh\" (UID: \"3f28530b-4e55-487a-8ed4-540d104609bb\") " pod="openshift-ingress/e2e-ipp-b-6d929c-openshift-default-5cd456f5b6-4j7rh"
Sep 10 15:53:58.501907 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:58.501670    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f28530b-4e55-487a-8ed4-540d104609bb-workload-socket\") pod \"e2e-ipp-b-6d929c-openshift-default-5cd456f5b6-4j7rh\" (UID: \"3f28530b-4e55-487a-8ed4-540d104609bb\") " pod="openshift-ingress/e2e-ipp-b-6d929c-openshift-default-5cd456f5b6-4j7rh"
Sep 10 15:53:58.602988 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:58.602953    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f28530b-4e55-487a-8ed4-540d104609bb-istio-data\") pod \"e2e-ipp-b-6d929c-openshift-default-5cd456f5b6-4j7rh\" (UID: \"3f28530b-4e55-487a-8ed4-540d104609bb\") " pod="openshift-ingress/e2e-ipp-b-6d929c-openshift-default-5cd456f5b6-4j7rh"
Sep 10 15:53:58.603182 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:58.602994    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f28530b-4e55-487a-8ed4-540d104609bb-istiod-ca-cert\") pod \"e2e-ipp-b-6d929c-openshift-default-5cd456f5b6-4j7rh\" (UID: \"3f28530b-4e55-487a-8ed4-540d104609bb\") " pod="openshift-ingress/e2e-ipp-b-6d929c-openshift-default-5cd456f5b6-4j7rh"
Sep 10 15:53:58.603182 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:58.603048    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f28530b-4e55-487a-8ed4-540d104609bb-workload-socket\") pod \"e2e-ipp-b-6d929c-openshift-default-5cd456f5b6-4j7rh\" (UID: \"3f28530b-4e55-487a-8ed4-540d104609bb\") " pod="openshift-ingress/e2e-ipp-b-6d929c-openshift-default-5cd456f5b6-4j7rh"
Sep 10 15:53:58.603182 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:58.603087    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fx4dw\" (UniqueName: \"kubernetes.io/projected/3f28530b-4e55-487a-8ed4-540d104609bb-kube-api-access-fx4dw\") pod \"e2e-ipp-b-6d929c-openshift-default-5cd456f5b6-4j7rh\" (UID: \"3f28530b-4e55-487a-8ed4-540d104609bb\") " pod="openshift-ingress/e2e-ipp-b-6d929c-openshift-default-5cd456f5b6-4j7rh"
Sep 10 15:53:58.603182 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:58.603149    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f28530b-4e55-487a-8ed4-540d104609bb-workload-certs\") pod \"e2e-ipp-b-6d929c-openshift-default-5cd456f5b6-4j7rh\" (UID: \"3f28530b-4e55-487a-8ed4-540d104609bb\") " pod="openshift-ingress/e2e-ipp-b-6d929c-openshift-default-5cd456f5b6-4j7rh"
Sep 10 15:53:58.603182 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:58.603179    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f28530b-4e55-487a-8ed4-540d104609bb-istio-podinfo\") pod \"e2e-ipp-b-6d929c-openshift-default-5cd456f5b6-4j7rh\" (UID: \"3f28530b-4e55-487a-8ed4-540d104609bb\") " pod="openshift-ingress/e2e-ipp-b-6d929c-openshift-default-5cd456f5b6-4j7rh"
Sep 10 15:53:58.603467 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:58.603208    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f28530b-4e55-487a-8ed4-540d104609bb-istio-envoy\") pod \"e2e-ipp-b-6d929c-openshift-default-5cd456f5b6-4j7rh\" (UID: \"3f28530b-4e55-487a-8ed4-540d104609bb\") " pod="openshift-ingress/e2e-ipp-b-6d929c-openshift-default-5cd456f5b6-4j7rh"
Sep 10 15:53:58.603467 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:58.603240    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f28530b-4e55-487a-8ed4-540d104609bb-istio-token\") pod \"e2e-ipp-b-6d929c-openshift-default-5cd456f5b6-4j7rh\" (UID: \"3f28530b-4e55-487a-8ed4-540d104609bb\") " pod="openshift-ingress/e2e-ipp-b-6d929c-openshift-default-5cd456f5b6-4j7rh"
Sep 10 15:53:58.603467 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:58.603270    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f28530b-4e55-487a-8ed4-540d104609bb-credential-socket\") pod \"e2e-ipp-b-6d929c-openshift-default-5cd456f5b6-4j7rh\" (UID: \"3f28530b-4e55-487a-8ed4-540d104609bb\") " pod="openshift-ingress/e2e-ipp-b-6d929c-openshift-default-5cd456f5b6-4j7rh"
Sep 10 15:53:58.603640 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:58.603492    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f28530b-4e55-487a-8ed4-540d104609bb-workload-socket\") pod \"e2e-ipp-b-6d929c-openshift-default-5cd456f5b6-4j7rh\" (UID: \"3f28530b-4e55-487a-8ed4-540d104609bb\") " pod="openshift-ingress/e2e-ipp-b-6d929c-openshift-default-5cd456f5b6-4j7rh"
Sep 10 15:53:58.603640 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:58.603555    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f28530b-4e55-487a-8ed4-540d104609bb-istio-data\") pod \"e2e-ipp-b-6d929c-openshift-default-5cd456f5b6-4j7rh\" (UID: \"3f28530b-4e55-487a-8ed4-540d104609bb\") " pod="openshift-ingress/e2e-ipp-b-6d929c-openshift-default-5cd456f5b6-4j7rh"
Sep 10 15:53:58.603712 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:58.603676    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f28530b-4e55-487a-8ed4-540d104609bb-workload-certs\") pod \"e2e-ipp-b-6d929c-openshift-default-5cd456f5b6-4j7rh\" (UID: \"3f28530b-4e55-487a-8ed4-540d104609bb\") " pod="openshift-ingress/e2e-ipp-b-6d929c-openshift-default-5cd456f5b6-4j7rh"
Sep 10 15:53:58.603917 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:58.603897    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f28530b-4e55-487a-8ed4-540d104609bb-credential-socket\") pod \"e2e-ipp-b-6d929c-openshift-default-5cd456f5b6-4j7rh\" (UID: \"3f28530b-4e55-487a-8ed4-540d104609bb\") " pod="openshift-ingress/e2e-ipp-b-6d929c-openshift-default-5cd456f5b6-4j7rh"
Sep 10 15:53:58.604130 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:58.604090    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f28530b-4e55-487a-8ed4-540d104609bb-istiod-ca-cert\") pod \"e2e-ipp-b-6d929c-openshift-default-5cd456f5b6-4j7rh\" (UID: \"3f28530b-4e55-487a-8ed4-540d104609bb\") " pod="openshift-ingress/e2e-ipp-b-6d929c-openshift-default-5cd456f5b6-4j7rh"
Sep 10 15:53:58.606121 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:58.606101    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f28530b-4e55-487a-8ed4-540d104609bb-istio-envoy\") pod \"e2e-ipp-b-6d929c-openshift-default-5cd456f5b6-4j7rh\" (UID: \"3f28530b-4e55-487a-8ed4-540d104609bb\") " pod="openshift-ingress/e2e-ipp-b-6d929c-openshift-default-5cd456f5b6-4j7rh"
Sep 10 15:53:58.606468 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:58.606448    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f28530b-4e55-487a-8ed4-540d104609bb-istio-podinfo\") pod \"e2e-ipp-b-6d929c-openshift-default-5cd456f5b6-4j7rh\" (UID: \"3f28530b-4e55-487a-8ed4-540d104609bb\") " pod="openshift-ingress/e2e-ipp-b-6d929c-openshift-default-5cd456f5b6-4j7rh"
Sep 10 15:53:58.612831 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:58.612808    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f28530b-4e55-487a-8ed4-540d104609bb-istio-token\") pod \"e2e-ipp-b-6d929c-openshift-default-5cd456f5b6-4j7rh\" (UID: \"3f28530b-4e55-487a-8ed4-540d104609bb\") " pod="openshift-ingress/e2e-ipp-b-6d929c-openshift-default-5cd456f5b6-4j7rh"
Sep 10 15:53:58.613117 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:58.613077    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fx4dw\" (UniqueName: \"kubernetes.io/projected/3f28530b-4e55-487a-8ed4-540d104609bb-kube-api-access-fx4dw\") pod \"e2e-ipp-b-6d929c-openshift-default-5cd456f5b6-4j7rh\" (UID: \"3f28530b-4e55-487a-8ed4-540d104609bb\") " pod="openshift-ingress/e2e-ipp-b-6d929c-openshift-default-5cd456f5b6-4j7rh"
Sep 10 15:53:58.788008 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:58.787930    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-6d929c-openshift-default-5cd456f5b6-4j7rh"
Sep 10 15:53:59.067923 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:59.067897    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-6d929c-openshift-default-5cd456f5b6-4j7rh"]
Sep 10 15:53:59.069485 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:53:59.069453    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f28530b_4e55_487a_8ed4_540d104609bb.slice/crio-dfada672aa91f494454ebe61bdcbef673a4d8bbe234eb58f9c5f376b0cb92d42 WatchSource:0}: Error finding container dfada672aa91f494454ebe61bdcbef673a4d8bbe234eb58f9c5f376b0cb92d42: Status 404 returned error can't find the container with id dfada672aa91f494454ebe61bdcbef673a4d8bbe234eb58f9c5f376b0cb92d42
Sep 10 15:53:59.072150 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:59.072115    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 15:53:59.072235 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:59.072182    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 15:53:59.072235 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:59.072210    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 15:53:59.464042 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:59.464007    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6d929c-openshift-default-5cd456f5b6-4j7rh" event={"ID":"3f28530b-4e55-487a-8ed4-540d104609bb","Type":"ContainerStarted","Data":"5599d6092e025ae2b3be10148088fdfe69ade76230dcf5d243a07881401a5736"}
Sep 10 15:53:59.464235 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:59.464069    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6d929c-openshift-default-5cd456f5b6-4j7rh" event={"ID":"3f28530b-4e55-487a-8ed4-540d104609bb","Type":"ContainerStarted","Data":"dfada672aa91f494454ebe61bdcbef673a4d8bbe234eb58f9c5f376b0cb92d42"}
Sep 10 15:53:59.492371 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:59.492303    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-6d929c-openshift-default-5cd456f5b6-4j7rh" podStartSLOduration=1.4922800330000001 podStartE2EDuration="1.492280033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:53:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:53:59.490814205 +0000 UTC m=+1615.232779301" watchObservedRunningTime="2026-09-10 15:53:59.492280033 +0000 UTC m=+1615.234245137"
Sep 10 15:53:59.788365 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:53:59.788282    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-6d929c-openshift-default-5cd456f5b6-4j7rh"
Sep 10 15:54:00.793476 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:54:00.793446    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-6d929c-openshift-default-5cd456f5b6-4j7rh"
Sep 10 15:54:01.475260 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:54:01.475228    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-6d929c-openshift-default-5cd456f5b6-4j7rh"
Sep 10 15:54:01.476334 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:54:01.476310    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-6d929c-openshift-default-5cd456f5b6-4j7rh"
Sep 10 15:54:04.343758 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:54:04.343717    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-6d929c-6bf78f68b5-9b7nn"]
Sep 10 15:54:04.347449 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:54:04.347424    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6d929c-6bf78f68b5-9b7nn"
Sep 10 15:54:04.349981 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:54:04.349954    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-6d929c-serving-cert\""
Sep 10 15:54:04.356447 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:54:04.356381    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-6d929c-6bf78f68b5-9b7nn"]
Sep 10 15:54:04.459331 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:54:04.459297    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b878b42b-4947-4d62-a1c1-829389306f92-metrics-certs\") pod \"maas-api-e2e-ipp-b-6d929c-6bf78f68b5-9b7nn\" (UID: \"b878b42b-4947-4d62-a1c1-829389306f92\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6d929c-6bf78f68b5-9b7nn"
Sep 10 15:54:04.459526 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:54:04.459417    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b878b42b-4947-4d62-a1c1-829389306f92-maas-api-tls\") pod \"maas-api-e2e-ipp-b-6d929c-6bf78f68b5-9b7nn\" (UID: \"b878b42b-4947-4d62-a1c1-829389306f92\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6d929c-6bf78f68b5-9b7nn"
Sep 10 15:54:04.459526 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:54:04.459441    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbqqz\" (UniqueName: \"kubernetes.io/projected/b878b42b-4947-4d62-a1c1-829389306f92-kube-api-access-lbqqz\") pod \"maas-api-e2e-ipp-b-6d929c-6bf78f68b5-9b7nn\" (UID: \"b878b42b-4947-4d62-a1c1-829389306f92\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6d929c-6bf78f68b5-9b7nn"
Sep 10 15:54:04.459526 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:54:04.459508    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b878b42b-4947-4d62-a1c1-829389306f92-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-6d929c-6bf78f68b5-9b7nn\" (UID: \"b878b42b-4947-4d62-a1c1-829389306f92\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6d929c-6bf78f68b5-9b7nn"
Sep 10 15:54:04.560940 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:54:04.560899    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b878b42b-4947-4d62-a1c1-829389306f92-maas-api-tls\") pod \"maas-api-e2e-ipp-b-6d929c-6bf78f68b5-9b7nn\" (UID: \"b878b42b-4947-4d62-a1c1-829389306f92\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6d929c-6bf78f68b5-9b7nn"
Sep 10 15:54:04.560940 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:54:04.560941    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbqqz\" (UniqueName: \"kubernetes.io/projected/b878b42b-4947-4d62-a1c1-829389306f92-kube-api-access-lbqqz\") pod \"maas-api-e2e-ipp-b-6d929c-6bf78f68b5-9b7nn\" (UID: \"b878b42b-4947-4d62-a1c1-829389306f92\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6d929c-6bf78f68b5-9b7nn"
Sep 10 15:54:04.561174 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:54:04.560961    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b878b42b-4947-4d62-a1c1-829389306f92-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-6d929c-6bf78f68b5-9b7nn\" (UID: \"b878b42b-4947-4d62-a1c1-829389306f92\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6d929c-6bf78f68b5-9b7nn"
Sep 10 15:54:04.561174 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:54:04.561001    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b878b42b-4947-4d62-a1c1-829389306f92-metrics-certs\") pod \"maas-api-e2e-ipp-b-6d929c-6bf78f68b5-9b7nn\" (UID: \"b878b42b-4947-4d62-a1c1-829389306f92\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6d929c-6bf78f68b5-9b7nn"
Sep 10 15:54:04.561174 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:54:04.561072    2559 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-6d929c-serving-cert: secret "maas-api-e2e-ipp-b-6d929c-serving-cert" not found
Sep 10 15:54:04.561174 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:54:04.561152    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b878b42b-4947-4d62-a1c1-829389306f92-maas-api-tls podName:b878b42b-4947-4d62-a1c1-829389306f92 nodeName:}" failed. No retries permitted until 2026-09-10 15:54:05.061130523 +0000 UTC m=+1620.803095608 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b878b42b-4947-4d62-a1c1-829389306f92-maas-api-tls") pod "maas-api-e2e-ipp-b-6d929c-6bf78f68b5-9b7nn" (UID: "b878b42b-4947-4d62-a1c1-829389306f92") : secret "maas-api-e2e-ipp-b-6d929c-serving-cert" not found
Sep 10 15:54:04.561703 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:54:04.561673    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b878b42b-4947-4d62-a1c1-829389306f92-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-6d929c-6bf78f68b5-9b7nn\" (UID: \"b878b42b-4947-4d62-a1c1-829389306f92\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6d929c-6bf78f68b5-9b7nn"
Sep 10 15:54:04.563436 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:54:04.563413    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b878b42b-4947-4d62-a1c1-829389306f92-metrics-certs\") pod \"maas-api-e2e-ipp-b-6d929c-6bf78f68b5-9b7nn\" (UID: \"b878b42b-4947-4d62-a1c1-829389306f92\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6d929c-6bf78f68b5-9b7nn"
Sep 10 15:54:04.570760 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:54:04.570738    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbqqz\" (UniqueName: \"kubernetes.io/projected/b878b42b-4947-4d62-a1c1-829389306f92-kube-api-access-lbqqz\") pod \"maas-api-e2e-ipp-b-6d929c-6bf78f68b5-9b7nn\" (UID: \"b878b42b-4947-4d62-a1c1-829389306f92\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6d929c-6bf78f68b5-9b7nn"
Sep 10 15:54:05.066010 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:54:05.065979    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b878b42b-4947-4d62-a1c1-829389306f92-maas-api-tls\") pod \"maas-api-e2e-ipp-b-6d929c-6bf78f68b5-9b7nn\" (UID: \"b878b42b-4947-4d62-a1c1-829389306f92\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6d929c-6bf78f68b5-9b7nn"
Sep 10 15:54:05.068206 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:54:05.068186    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b878b42b-4947-4d62-a1c1-829389306f92-maas-api-tls\") pod \"maas-api-e2e-ipp-b-6d929c-6bf78f68b5-9b7nn\" (UID: \"b878b42b-4947-4d62-a1c1-829389306f92\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6d929c-6bf78f68b5-9b7nn"
Sep 10 15:54:05.261230 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:54:05.261141    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6d929c-6bf78f68b5-9b7nn"
Sep 10 15:54:05.537956 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:54:05.537925    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-6d929c-6bf78f68b5-9b7nn"]
Sep 10 15:54:05.539551 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:54:05.539525    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb878b42b_4947_4d62_a1c1_829389306f92.slice/crio-92d2450700c4c39feac77e086b8347faed1c91af860f619a308a7f97f8b64338 WatchSource:0}: Error finding container 92d2450700c4c39feac77e086b8347faed1c91af860f619a308a7f97f8b64338: Status 404 returned error can't find the container with id 92d2450700c4c39feac77e086b8347faed1c91af860f619a308a7f97f8b64338
Sep 10 15:54:06.505397 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:54:06.505356    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6d929c-6bf78f68b5-9b7nn" event={"ID":"b878b42b-4947-4d62-a1c1-829389306f92","Type":"ContainerStarted","Data":"f84a59377514d9ef408120016e463361253558796ea8f0ce1ed7a011b84be89e"}
Sep 10 15:54:06.505567 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:54:06.505437    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6d929c-6bf78f68b5-9b7nn" event={"ID":"b878b42b-4947-4d62-a1c1-829389306f92","Type":"ContainerStarted","Data":"92d2450700c4c39feac77e086b8347faed1c91af860f619a308a7f97f8b64338"}
Sep 10 15:54:06.505567 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:54:06.505458    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6d929c-6bf78f68b5-9b7nn"
Sep 10 15:54:06.528388 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:54:06.528335    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6d929c-6bf78f68b5-9b7nn" podStartSLOduration=1.995208679 podStartE2EDuration="2.528321105s" totalImagesPullingTime="533.112426ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:54:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:54:06.528101021 +0000 UTC m=+1622.270066131" watchObservedRunningTime="2026-09-10 15:54:06.528321105 +0000 UTC m=+1622.270286200"
Sep 10 15:54:10.301359 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:54:10.301321    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bs7hd"]
Sep 10 15:54:12.517305 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:54:12.517273    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6d929c-6bf78f68b5-9b7nn"
Sep 10 15:54:20.593426 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:54:20.593393    2559 generic.go:356] "Generic (PLEG): container finished" podID="3f28530b-4e55-487a-8ed4-540d104609bb" containerID="5599d6092e025ae2b3be10148088fdfe69ade76230dcf5d243a07881401a5736" exitCode=0
Sep 10 15:54:20.593973 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:54:20.593471    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6d929c-openshift-default-5cd456f5b6-4j7rh" event={"ID":"3f28530b-4e55-487a-8ed4-540d104609bb","Type":"ContainerDied","Data":"5599d6092e025ae2b3be10148088fdfe69ade76230dcf5d243a07881401a5736"}
Sep 10 15:54:20.594041 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:54:20.594024    2559 scope.go:117] "RemoveContainer" containerID="5599d6092e025ae2b3be10148088fdfe69ade76230dcf5d243a07881401a5736"
Sep 10 15:54:20.595164 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:54:20.595140    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 15:54:20.595236 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:54:20.595204    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 15:54:20.595273 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:54:20.595234    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 15:54:21.601516 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:54:21.601483    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6d929c-openshift-default-5cd456f5b6-4j7rh" event={"ID":"3f28530b-4e55-487a-8ed4-540d104609bb","Type":"ContainerStarted","Data":"efa31517c47cdd32fe277ca2a393555360d7595301b5792b4655835135a79dbe"}
Sep 10 15:54:21.788500 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:54:21.788460    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-6d929c-openshift-default-5cd456f5b6-4j7rh"
Sep 10 15:54:21.788757 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:54:21.788734    2559 patch_prober.go:28] interesting pod/e2e-ipp-b-6d929c-openshift-default-5cd456f5b6-4j7rh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Sep 10 15:54:21.788815 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:54:21.788787    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6d929c-openshift-default-5cd456f5b6-4j7rh" podUID="3f28530b-4e55-487a-8ed4-540d104609bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Sep 10 15:54:22.789243 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:54:22.789205    2559 patch_prober.go:28] interesting pod/e2e-ipp-b-6d929c-openshift-default-5cd456f5b6-4j7rh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Sep 10 15:54:22.789764 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:54:22.789267    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6d929c-openshift-default-5cd456f5b6-4j7rh" podUID="3f28530b-4e55-487a-8ed4-540d104609bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Sep 10 15:54:23.788757 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:54:23.788725    2559 patch_prober.go:28] interesting pod/e2e-ipp-b-6d929c-openshift-default-5cd456f5b6-4j7rh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Sep 10 15:54:23.788946 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:54:23.788777    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6d929c-openshift-default-5cd456f5b6-4j7rh" podUID="3f28530b-4e55-487a-8ed4-540d104609bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Sep 10 15:54:24.788625 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:54:24.788573    2559 patch_prober.go:28] interesting pod/e2e-ipp-b-6d929c-openshift-default-5cd456f5b6-4j7rh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Sep 10 15:54:24.789008 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:54:24.788646    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6d929c-openshift-default-5cd456f5b6-4j7rh" podUID="3f28530b-4e55-487a-8ed4-540d104609bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Sep 10 15:54:25.788342 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:54:25.788305    2559 patch_prober.go:28] interesting pod/e2e-ipp-b-6d929c-openshift-default-5cd456f5b6-4j7rh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Sep 10 15:54:25.788517 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:54:25.788361    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6d929c-openshift-default-5cd456f5b6-4j7rh" podUID="3f28530b-4e55-487a-8ed4-540d104609bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Sep 10 15:54:26.848762 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:54:26.848726    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-6d929c-openshift-default-5cd456f5b6-4j7rh"
Sep 10 15:54:26.849314 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:54:26.849294    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-6d929c-openshift-default-5cd456f5b6-4j7rh"
Sep 10 15:54:26.850667 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:54:26.850646    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-6d929c-openshift-default-5cd456f5b6-4j7rh"
Sep 10 15:54:33.004889 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:54:33.004852    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-8cb756/ipp-route-8cb756-kserve-596bdc755-27sqr"]
Sep 10 15:54:33.008761 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:54:33.008736    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-8cb756/ipp-route-8cb756-kserve-596bdc755-27sqr"
Sep 10 15:54:33.012847 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:54:33.012818    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-8cb756\"/\"openshift-service-ca.crt\""
Sep 10 15:54:33.012847 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:54:33.012840    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-8cb756\"/\"kube-root-ca.crt\""
Sep 10 15:54:33.013053 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:54:33.012826    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-8cb756\"/\"default-dockercfg-prc8d\""
Sep 10 15:54:33.013053 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:54:33.012826    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-8cb756\"/\"ipp-route-8cb756-kserve-self-signed-certs\""
Sep 10 15:54:33.017603 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:54:33.017555    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-8cb756/ipp-route-8cb756-kserve-596bdc755-27sqr"]
Sep 10 15:54:33.033793 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:54:33.033768    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a9082bf-20cc-48b1-92fa-c0bef5f5ab46-model-cache\") pod \"ipp-route-8cb756-kserve-596bdc755-27sqr\" (UID: \"1a9082bf-20cc-48b1-92fa-c0bef5f5ab46\") " pod="ai-tenant-e2e-ipp-a-8cb756/ipp-route-8cb756-kserve-596bdc755-27sqr"
Sep 10 15:54:33.033951 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:54:33.033813    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a9082bf-20cc-48b1-92fa-c0bef5f5ab46-tmp-dir\") pod \"ipp-route-8cb756-kserve-596bdc755-27sqr\" (UID: \"1a9082bf-20cc-48b1-92fa-c0bef5f5ab46\") " pod="ai-tenant-e2e-ipp-a-8cb756/ipp-route-8cb756-kserve-596bdc755-27sqr"
Sep 10 15:54:33.033951 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:54:33.033866    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a9082bf-20cc-48b1-92fa-c0bef5f5ab46-home\") pod \"ipp-route-8cb756-kserve-596bdc755-27sqr\" (UID: \"1a9082bf-20cc-48b1-92fa-c0bef5f5ab46\") " pod="ai-tenant-e2e-ipp-a-8cb756/ipp-route-8cb756-kserve-596bdc755-27sqr"
Sep 10 15:54:33.033951 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:54:33.033905    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9h86b\" (UniqueName: \"kubernetes.io/projected/1a9082bf-20cc-48b1-92fa-c0bef5f5ab46-kube-api-access-9h86b\") pod \"ipp-route-8cb756-kserve-596bdc755-27sqr\" (UID: \"1a9082bf-20cc-48b1-92fa-c0bef5f5ab46\") " pod="ai-tenant-e2e-ipp-a-8cb756/ipp-route-8cb756-kserve-596bdc755-27sqr"
Sep 10 15:54:33.034078 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:54:33.033953    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a9082bf-20cc-48b1-92fa-c0bef5f5ab46-tls-certs\") pod \"ipp-route-8cb756-kserve-596bdc755-27sqr\" (UID: \"1a9082bf-20cc-48b1-92fa-c0bef5f5ab46\") " pod="ai-tenant-e2e-ipp-a-8cb756/ipp-route-8cb756-kserve-596bdc755-27sqr"
Sep 10 15:54:33.034078 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:54:33.034015    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a9082bf-20cc-48b1-92fa-c0bef5f5ab46-dshm\") pod \"ipp-route-8cb756-kserve-596bdc755-27sqr\" (UID: \"1a9082bf-20cc-48b1-92fa-c0bef5f5ab46\") " pod="ai-tenant-e2e-ipp-a-8cb756/ipp-route-8cb756-kserve-596bdc755-27sqr"
Sep 10 15:54:33.134489 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:54:33.134447    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a9082bf-20cc-48b1-92fa-c0bef5f5ab46-dshm\") pod \"ipp-route-8cb756-kserve-596bdc755-27sqr\" (UID: \"1a9082bf-20cc-48b1-92fa-c0bef5f5ab46\") " pod="ai-tenant-e2e-ipp-a-8cb756/ipp-route-8cb756-kserve-596bdc755-27sqr"
Sep 10 15:54:33.134704 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:54:33.134535    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a9082bf-20cc-48b1-92fa-c0bef5f5ab46-model-cache\") pod \"ipp-route-8cb756-kserve-596bdc755-27sqr\" (UID: \"1a9082bf-20cc-48b1-92fa-c0bef5f5ab46\") " pod="ai-tenant-e2e-ipp-a-8cb756/ipp-route-8cb756-kserve-596bdc755-27sqr"
Sep 10 15:54:33.134704 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:54:33.134571    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a9082bf-20cc-48b1-92fa-c0bef5f5ab46-tmp-dir\") pod \"ipp-route-8cb756-kserve-596bdc755-27sqr\" (UID: \"1a9082bf-20cc-48b1-92fa-c0bef5f5ab46\") " pod="ai-tenant-e2e-ipp-a-8cb756/ipp-route-8cb756-kserve-596bdc755-27sqr"
Sep 10 15:54:33.134704 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:54:33.134630    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a9082bf-20cc-48b1-92fa-c0bef5f5ab46-home\") pod \"ipp-route-8cb756-kserve-596bdc755-27sqr\" (UID: \"1a9082bf-20cc-48b1-92fa-c0bef5f5ab46\") " pod="ai-tenant-e2e-ipp-a-8cb756/ipp-route-8cb756-kserve-596bdc755-27sqr"
Sep 10 15:54:33.134704 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:54:33.134655    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9h86b\" (UniqueName: \"kubernetes.io/projected/1a9082bf-20cc-48b1-92fa-c0bef5f5ab46-kube-api-access-9h86b\") pod \"ipp-route-8cb756-kserve-596bdc755-27sqr\" (UID: \"1a9082bf-20cc-48b1-92fa-c0bef5f5ab46\") " pod="ai-tenant-e2e-ipp-a-8cb756/ipp-route-8cb756-kserve-596bdc755-27sqr"
Sep 10 15:54:33.134704 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:54:33.134680    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a9082bf-20cc-48b1-92fa-c0bef5f5ab46-tls-certs\") pod \"ipp-route-8cb756-kserve-596bdc755-27sqr\" (UID: \"1a9082bf-20cc-48b1-92fa-c0bef5f5ab46\") " pod="ai-tenant-e2e-ipp-a-8cb756/ipp-route-8cb756-kserve-596bdc755-27sqr"
Sep 10 15:54:33.135043 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:54:33.135019    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a9082bf-20cc-48b1-92fa-c0bef5f5ab46-model-cache\") pod \"ipp-route-8cb756-kserve-596bdc755-27sqr\" (UID: \"1a9082bf-20cc-48b1-92fa-c0bef5f5ab46\") " pod="ai-tenant-e2e-ipp-a-8cb756/ipp-route-8cb756-kserve-596bdc755-27sqr"
Sep 10 15:54:33.135115 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:54:33.135065    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a9082bf-20cc-48b1-92fa-c0bef5f5ab46-home\") pod \"ipp-route-8cb756-kserve-596bdc755-27sqr\" (UID: \"1a9082bf-20cc-48b1-92fa-c0bef5f5ab46\") " pod="ai-tenant-e2e-ipp-a-8cb756/ipp-route-8cb756-kserve-596bdc755-27sqr"
Sep 10 15:54:33.135115 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:54:33.135091    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a9082bf-20cc-48b1-92fa-c0bef5f5ab46-tmp-dir\") pod \"ipp-route-8cb756-kserve-596bdc755-27sqr\" (UID: \"1a9082bf-20cc-48b1-92fa-c0bef5f5ab46\") " pod="ai-tenant-e2e-ipp-a-8cb756/ipp-route-8cb756-kserve-596bdc755-27sqr"
Sep 10 15:54:33.136767 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:54:33.136749    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a9082bf-20cc-48b1-92fa-c0bef5f5ab46-dshm\") pod \"ipp-route-8cb756-kserve-596bdc755-27sqr\" (UID: \"1a9082bf-20cc-48b1-92fa-c0bef5f5ab46\") " pod="ai-tenant-e2e-ipp-a-8cb756/ipp-route-8cb756-kserve-596bdc755-27sqr"
Sep 10 15:54:33.137214 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:54:33.137197    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a9082bf-20cc-48b1-92fa-c0bef5f5ab46-tls-certs\") pod \"ipp-route-8cb756-kserve-596bdc755-27sqr\" (UID: \"1a9082bf-20cc-48b1-92fa-c0bef5f5ab46\") " pod="ai-tenant-e2e-ipp-a-8cb756/ipp-route-8cb756-kserve-596bdc755-27sqr"
Sep 10 15:54:33.143729 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:54:33.143709    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9h86b\" (UniqueName: \"kubernetes.io/projected/1a9082bf-20cc-48b1-92fa-c0bef5f5ab46-kube-api-access-9h86b\") pod \"ipp-route-8cb756-kserve-596bdc755-27sqr\" (UID: \"1a9082bf-20cc-48b1-92fa-c0bef5f5ab46\") " pod="ai-tenant-e2e-ipp-a-8cb756/ipp-route-8cb756-kserve-596bdc755-27sqr"
Sep 10 15:54:33.319091 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:54:33.319003    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-8cb756/ipp-route-8cb756-kserve-596bdc755-27sqr"
Sep 10 15:54:33.597728 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:54:33.597699    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-8cb756/ipp-route-8cb756-kserve-596bdc755-27sqr"]
Sep 10 15:54:33.599060 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:54:33.599032    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1a9082bf_20cc_48b1_92fa_c0bef5f5ab46.slice/crio-0383aea8116629dfed8561fd9206d089366d940adfa37e2beea42fa80652da90 WatchSource:0}: Error finding container 0383aea8116629dfed8561fd9206d089366d940adfa37e2beea42fa80652da90: Status 404 returned error can't find the container with id 0383aea8116629dfed8561fd9206d089366d940adfa37e2beea42fa80652da90
Sep 10 15:54:33.666308 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:54:33.666278    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-8cb756/ipp-route-8cb756-kserve-596bdc755-27sqr" event={"ID":"1a9082bf-20cc-48b1-92fa-c0bef5f5ab46","Type":"ContainerStarted","Data":"0383aea8116629dfed8561fd9206d089366d940adfa37e2beea42fa80652da90"}
Sep 10 15:54:34.673498 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:54:34.673465    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-8cb756/ipp-route-8cb756-kserve-596bdc755-27sqr" event={"ID":"1a9082bf-20cc-48b1-92fa-c0bef5f5ab46","Type":"ContainerStarted","Data":"f20a5c9c23df39e77b41d177dfb707544d13fedd7177e6c2e4a60492f9fb7e5b"}
Sep 10 15:54:34.694860 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:54:34.694809    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-8cb756/ipp-route-8cb756-kserve-596bdc755-27sqr" podStartSLOduration=2.69479281 podStartE2EDuration="2.69479281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:54:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:54:34.692513048 +0000 UTC m=+1650.434478151" watchObservedRunningTime="2026-09-10 15:54:34.69479281 +0000 UTC m=+1650.436757912"
Sep 10 15:54:37.204272 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:54:37.204240    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bs7hd"]
Sep 10 15:54:43.319357 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:54:43.319319    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-8cb756/ipp-route-8cb756-kserve-596bdc755-27sqr"
Sep 10 15:54:43.319818 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:54:43.319378    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-8cb756/ipp-route-8cb756-kserve-596bdc755-27sqr"
Sep 10 15:54:43.331961 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:54:43.331931    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-8cb756/ipp-route-8cb756-kserve-596bdc755-27sqr"
Sep 10 15:54:43.732151 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:54:43.732117    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-8cb756/ipp-route-8cb756-kserve-596bdc755-27sqr"
Sep 10 15:55:01.008337 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:55:01.008301    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bs7hd"]
Sep 10 15:55:09.045337 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:55:09.045293    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-e48e16-openshift-default-6bc647b4bb-f6j44"]
Sep 10 15:55:09.049617 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:55:09.049594    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-e48e16-openshift-default-6bc647b4bb-f6j44"
Sep 10 15:55:09.052609 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:55:09.052564    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-e48e16-openshift-default-dockercfg-bjc4t\""
Sep 10 15:55:09.060804 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:55:09.060781    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-e48e16-openshift-default-6bc647b4bb-f6j44"]
Sep 10 15:55:09.166339 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:55:09.166299    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjt6r\" (UniqueName: \"kubernetes.io/projected/beb30354-999e-491d-b6de-d3c734bc3b64-kube-api-access-cjt6r\") pod \"e2e-ipp-cleanup-e48e16-openshift-default-6bc647b4bb-f6j44\" (UID: \"beb30354-999e-491d-b6de-d3c734bc3b64\") " pod="openshift-ingress/e2e-ipp-cleanup-e48e16-openshift-default-6bc647b4bb-f6j44"
Sep 10 15:55:09.166339 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:55:09.166341    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/beb30354-999e-491d-b6de-d3c734bc3b64-istio-token\") pod \"e2e-ipp-cleanup-e48e16-openshift-default-6bc647b4bb-f6j44\" (UID: \"beb30354-999e-491d-b6de-d3c734bc3b64\") " pod="openshift-ingress/e2e-ipp-cleanup-e48e16-openshift-default-6bc647b4bb-f6j44"
Sep 10 15:55:09.166606 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:55:09.166375    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/beb30354-999e-491d-b6de-d3c734bc3b64-credential-socket\") pod \"e2e-ipp-cleanup-e48e16-openshift-default-6bc647b4bb-f6j44\" (UID: \"beb30354-999e-491d-b6de-d3c734bc3b64\") " pod="openshift-ingress/e2e-ipp-cleanup-e48e16-openshift-default-6bc647b4bb-f6j44"
Sep 10 15:55:09.166606 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:55:09.166458    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/beb30354-999e-491d-b6de-d3c734bc3b64-istio-podinfo\") pod \"e2e-ipp-cleanup-e48e16-openshift-default-6bc647b4bb-f6j44\" (UID: \"beb30354-999e-491d-b6de-d3c734bc3b64\") " pod="openshift-ingress/e2e-ipp-cleanup-e48e16-openshift-default-6bc647b4bb-f6j44"
Sep 10 15:55:09.166606 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:55:09.166487    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/beb30354-999e-491d-b6de-d3c734bc3b64-workload-certs\") pod \"e2e-ipp-cleanup-e48e16-openshift-default-6bc647b4bb-f6j44\" (UID: \"beb30354-999e-491d-b6de-d3c734bc3b64\") " pod="openshift-ingress/e2e-ipp-cleanup-e48e16-openshift-default-6bc647b4bb-f6j44"
Sep 10 15:55:09.166606 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:55:09.166504    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/beb30354-999e-491d-b6de-d3c734bc3b64-istio-data\") pod \"e2e-ipp-cleanup-e48e16-openshift-default-6bc647b4bb-f6j44\" (UID: \"beb30354-999e-491d-b6de-d3c734bc3b64\") " pod="openshift-ingress/e2e-ipp-cleanup-e48e16-openshift-default-6bc647b4bb-f6j44"
Sep 10 15:55:09.166606 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:55:09.166521    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/beb30354-999e-491d-b6de-d3c734bc3b64-istio-envoy\") pod \"e2e-ipp-cleanup-e48e16-openshift-default-6bc647b4bb-f6j44\" (UID: \"beb30354-999e-491d-b6de-d3c734bc3b64\") " pod="openshift-ingress/e2e-ipp-cleanup-e48e16-openshift-default-6bc647b4bb-f6j44"
Sep 10 15:55:09.166810 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:55:09.166634    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/beb30354-999e-491d-b6de-d3c734bc3b64-workload-socket\") pod \"e2e-ipp-cleanup-e48e16-openshift-default-6bc647b4bb-f6j44\" (UID: \"beb30354-999e-491d-b6de-d3c734bc3b64\") " pod="openshift-ingress/e2e-ipp-cleanup-e48e16-openshift-default-6bc647b4bb-f6j44"
Sep 10 15:55:09.166810 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:55:09.166667    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/beb30354-999e-491d-b6de-d3c734bc3b64-istiod-ca-cert\") pod \"e2e-ipp-cleanup-e48e16-openshift-default-6bc647b4bb-f6j44\" (UID: \"beb30354-999e-491d-b6de-d3c734bc3b64\") " pod="openshift-ingress/e2e-ipp-cleanup-e48e16-openshift-default-6bc647b4bb-f6j44"
Sep 10 15:55:09.267319 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:55:09.267280    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/beb30354-999e-491d-b6de-d3c734bc3b64-workload-certs\") pod \"e2e-ipp-cleanup-e48e16-openshift-default-6bc647b4bb-f6j44\" (UID: \"beb30354-999e-491d-b6de-d3c734bc3b64\") " pod="openshift-ingress/e2e-ipp-cleanup-e48e16-openshift-default-6bc647b4bb-f6j44"
Sep 10 15:55:09.267319 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:55:09.267315    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/beb30354-999e-491d-b6de-d3c734bc3b64-istio-data\") pod \"e2e-ipp-cleanup-e48e16-openshift-default-6bc647b4bb-f6j44\" (UID: \"beb30354-999e-491d-b6de-d3c734bc3b64\") " pod="openshift-ingress/e2e-ipp-cleanup-e48e16-openshift-default-6bc647b4bb-f6j44"
Sep 10 15:55:09.267612 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:55:09.267332    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/beb30354-999e-491d-b6de-d3c734bc3b64-istio-envoy\") pod \"e2e-ipp-cleanup-e48e16-openshift-default-6bc647b4bb-f6j44\" (UID: \"beb30354-999e-491d-b6de-d3c734bc3b64\") " pod="openshift-ingress/e2e-ipp-cleanup-e48e16-openshift-default-6bc647b4bb-f6j44"
Sep 10 15:55:09.267612 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:55:09.267398    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/beb30354-999e-491d-b6de-d3c734bc3b64-workload-socket\") pod \"e2e-ipp-cleanup-e48e16-openshift-default-6bc647b4bb-f6j44\" (UID: \"beb30354-999e-491d-b6de-d3c734bc3b64\") " pod="openshift-ingress/e2e-ipp-cleanup-e48e16-openshift-default-6bc647b4bb-f6j44"
Sep 10 15:55:09.267612 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:55:09.267426    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/beb30354-999e-491d-b6de-d3c734bc3b64-istiod-ca-cert\") pod \"e2e-ipp-cleanup-e48e16-openshift-default-6bc647b4bb-f6j44\" (UID: \"beb30354-999e-491d-b6de-d3c734bc3b64\") " pod="openshift-ingress/e2e-ipp-cleanup-e48e16-openshift-default-6bc647b4bb-f6j44"
Sep 10 15:55:09.267612 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:55:09.267467    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjt6r\" (UniqueName: \"kubernetes.io/projected/beb30354-999e-491d-b6de-d3c734bc3b64-kube-api-access-cjt6r\") pod \"e2e-ipp-cleanup-e48e16-openshift-default-6bc647b4bb-f6j44\" (UID: \"beb30354-999e-491d-b6de-d3c734bc3b64\") " pod="openshift-ingress/e2e-ipp-cleanup-e48e16-openshift-default-6bc647b4bb-f6j44"
Sep 10 15:55:09.267612 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:55:09.267491    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/beb30354-999e-491d-b6de-d3c734bc3b64-istio-token\") pod \"e2e-ipp-cleanup-e48e16-openshift-default-6bc647b4bb-f6j44\" (UID: \"beb30354-999e-491d-b6de-d3c734bc3b64\") " pod="openshift-ingress/e2e-ipp-cleanup-e48e16-openshift-default-6bc647b4bb-f6j44"
Sep 10 15:55:09.267612 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:55:09.267537    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/beb30354-999e-491d-b6de-d3c734bc3b64-credential-socket\") pod \"e2e-ipp-cleanup-e48e16-openshift-default-6bc647b4bb-f6j44\" (UID: \"beb30354-999e-491d-b6de-d3c734bc3b64\") " pod="openshift-ingress/e2e-ipp-cleanup-e48e16-openshift-default-6bc647b4bb-f6j44"
Sep 10 15:55:09.267612 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:55:09.267599    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/beb30354-999e-491d-b6de-d3c734bc3b64-istio-podinfo\") pod \"e2e-ipp-cleanup-e48e16-openshift-default-6bc647b4bb-f6j44\" (UID: \"beb30354-999e-491d-b6de-d3c734bc3b64\") " pod="openshift-ingress/e2e-ipp-cleanup-e48e16-openshift-default-6bc647b4bb-f6j44"
Sep 10 15:55:09.267975 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:55:09.267789    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/beb30354-999e-491d-b6de-d3c734bc3b64-workload-certs\") pod \"e2e-ipp-cleanup-e48e16-openshift-default-6bc647b4bb-f6j44\" (UID: \"beb30354-999e-491d-b6de-d3c734bc3b64\") " pod="openshift-ingress/e2e-ipp-cleanup-e48e16-openshift-default-6bc647b4bb-f6j44"
Sep 10 15:55:09.267975 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:55:09.267803    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/beb30354-999e-491d-b6de-d3c734bc3b64-workload-socket\") pod \"e2e-ipp-cleanup-e48e16-openshift-default-6bc647b4bb-f6j44\" (UID: \"beb30354-999e-491d-b6de-d3c734bc3b64\") " pod="openshift-ingress/e2e-ipp-cleanup-e48e16-openshift-default-6bc647b4bb-f6j44"
Sep 10 15:55:09.267975 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:55:09.267866    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/beb30354-999e-491d-b6de-d3c734bc3b64-istio-data\") pod \"e2e-ipp-cleanup-e48e16-openshift-default-6bc647b4bb-f6j44\" (UID: \"beb30354-999e-491d-b6de-d3c734bc3b64\") " pod="openshift-ingress/e2e-ipp-cleanup-e48e16-openshift-default-6bc647b4bb-f6j44"
Sep 10 15:55:09.268147 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:55:09.268122    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/beb30354-999e-491d-b6de-d3c734bc3b64-credential-socket\") pod \"e2e-ipp-cleanup-e48e16-openshift-default-6bc647b4bb-f6j44\" (UID: \"beb30354-999e-491d-b6de-d3c734bc3b64\") " pod="openshift-ingress/e2e-ipp-cleanup-e48e16-openshift-default-6bc647b4bb-f6j44"
Sep 10 15:55:09.268240 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:55:09.268213    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/beb30354-999e-491d-b6de-d3c734bc3b64-istiod-ca-cert\") pod \"e2e-ipp-cleanup-e48e16-openshift-default-6bc647b4bb-f6j44\" (UID: \"beb30354-999e-491d-b6de-d3c734bc3b64\") " pod="openshift-ingress/e2e-ipp-cleanup-e48e16-openshift-default-6bc647b4bb-f6j44"
Sep 10 15:55:09.269768 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:55:09.269744    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/beb30354-999e-491d-b6de-d3c734bc3b64-istio-envoy\") pod \"e2e-ipp-cleanup-e48e16-openshift-default-6bc647b4bb-f6j44\" (UID: \"beb30354-999e-491d-b6de-d3c734bc3b64\") " pod="openshift-ingress/e2e-ipp-cleanup-e48e16-openshift-default-6bc647b4bb-f6j44"
Sep 10 15:55:09.269976 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:55:09.269958    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/beb30354-999e-491d-b6de-d3c734bc3b64-istio-podinfo\") pod \"e2e-ipp-cleanup-e48e16-openshift-default-6bc647b4bb-f6j44\" (UID: \"beb30354-999e-491d-b6de-d3c734bc3b64\") " pod="openshift-ingress/e2e-ipp-cleanup-e48e16-openshift-default-6bc647b4bb-f6j44"
Sep 10 15:55:09.278520 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:55:09.278492    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjt6r\" (UniqueName: \"kubernetes.io/projected/beb30354-999e-491d-b6de-d3c734bc3b64-kube-api-access-cjt6r\") pod \"e2e-ipp-cleanup-e48e16-openshift-default-6bc647b4bb-f6j44\" (UID: \"beb30354-999e-491d-b6de-d3c734bc3b64\") " pod="openshift-ingress/e2e-ipp-cleanup-e48e16-openshift-default-6bc647b4bb-f6j44"
Sep 10 15:55:09.278647 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:55:09.278549    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/beb30354-999e-491d-b6de-d3c734bc3b64-istio-token\") pod \"e2e-ipp-cleanup-e48e16-openshift-default-6bc647b4bb-f6j44\" (UID: \"beb30354-999e-491d-b6de-d3c734bc3b64\") " pod="openshift-ingress/e2e-ipp-cleanup-e48e16-openshift-default-6bc647b4bb-f6j44"
Sep 10 15:55:09.361179 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:55:09.361100    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-e48e16-openshift-default-6bc647b4bb-f6j44"
Sep 10 15:55:09.640560 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:55:09.640527    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-e48e16-openshift-default-6bc647b4bb-f6j44"]
Sep 10 15:55:09.642499 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:55:09.642472    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbeb30354_999e_491d_b6de_d3c734bc3b64.slice/crio-f323f717e64e3237f6bdabf93966f63f8ac4dd9e82966bb02b94445b4b1daa48 WatchSource:0}: Error finding container f323f717e64e3237f6bdabf93966f63f8ac4dd9e82966bb02b94445b4b1daa48: Status 404 returned error can't find the container with id f323f717e64e3237f6bdabf93966f63f8ac4dd9e82966bb02b94445b4b1daa48
Sep 10 15:55:09.645158 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:55:09.645127    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 15:55:09.645256 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:55:09.645193    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 15:55:09.645256 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:55:09.645236    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 15:55:09.862556 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:55:09.862471    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-e48e16-openshift-default-6bc647b4bb-f6j44" event={"ID":"beb30354-999e-491d-b6de-d3c734bc3b64","Type":"ContainerStarted","Data":"6d524695bfd366c255943e58bb2f2170f517be9208fdac7d9cf11973e2cba4d8"}
Sep 10 15:55:09.862556 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:55:09.862529    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-e48e16-openshift-default-6bc647b4bb-f6j44" event={"ID":"beb30354-999e-491d-b6de-d3c734bc3b64","Type":"ContainerStarted","Data":"f323f717e64e3237f6bdabf93966f63f8ac4dd9e82966bb02b94445b4b1daa48"}
Sep 10 15:55:09.883534 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:55:09.883479    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-e48e16-openshift-default-6bc647b4bb-f6j44" podStartSLOduration=0.883462606 podStartE2EDuration="883.462606ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:55:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:55:09.882475243 +0000 UTC m=+1685.624440341" watchObservedRunningTime="2026-09-10 15:55:09.883462606 +0000 UTC m=+1685.625427708"
Sep 10 15:55:10.361944 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:55:10.361912    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-e48e16-openshift-default-6bc647b4bb-f6j44"
Sep 10 15:55:10.367903 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:55:10.367877    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-e48e16-openshift-default-6bc647b4bb-f6j44"
Sep 10 15:55:10.867814 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:55:10.867786    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-e48e16-openshift-default-6bc647b4bb-f6j44"
Sep 10 15:55:10.868942 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:55:10.868920    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-e48e16-openshift-default-6bc647b4bb-f6j44"
Sep 10 15:55:20.523705 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:55:20.523659    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bs7hd"]
Sep 10 15:55:30.992919 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:55:30.992886    2559 generic.go:356] "Generic (PLEG): container finished" podID="beb30354-999e-491d-b6de-d3c734bc3b64" containerID="6d524695bfd366c255943e58bb2f2170f517be9208fdac7d9cf11973e2cba4d8" exitCode=0
Sep 10 15:55:30.993348 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:55:30.992975    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-e48e16-openshift-default-6bc647b4bb-f6j44" event={"ID":"beb30354-999e-491d-b6de-d3c734bc3b64","Type":"ContainerDied","Data":"6d524695bfd366c255943e58bb2f2170f517be9208fdac7d9cf11973e2cba4d8"}
Sep 10 15:55:30.993391 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:55:30.993358    2559 scope.go:117] "RemoveContainer" containerID="6d524695bfd366c255943e58bb2f2170f517be9208fdac7d9cf11973e2cba4d8"
Sep 10 15:55:30.994427 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:55:30.994394    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 15:55:30.994477 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:55:30.994463    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 15:55:30.994514 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:55:30.994501    2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 15:55:32.000882 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:55:32.000848    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-e48e16-openshift-default-6bc647b4bb-f6j44" event={"ID":"beb30354-999e-491d-b6de-d3c734bc3b64","Type":"ContainerStarted","Data":"69a677378efe4c6fa71c5b5c1f7cc4e3f054e954ca318adb87fa5dc87e3476f6"}
Sep 10 15:55:32.362033 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:55:32.361935    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-e48e16-openshift-default-6bc647b4bb-f6j44"
Sep 10 15:55:32.362340 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:55:32.362315    2559 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-e48e16-openshift-default-6bc647b4bb-f6j44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Sep 10 15:55:32.362523 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:55:32.362497    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-e48e16-openshift-default-6bc647b4bb-f6j44" podUID="beb30354-999e-491d-b6de-d3c734bc3b64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Sep 10 15:55:33.361715 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:55:33.361654    2559 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-e48e16-openshift-default-6bc647b4bb-f6j44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Sep 10 15:55:33.362218 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:55:33.361763    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-e48e16-openshift-default-6bc647b4bb-f6j44" podUID="beb30354-999e-491d-b6de-d3c734bc3b64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Sep 10 15:55:34.362178 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:55:34.362128    2559 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-e48e16-openshift-default-6bc647b4bb-f6j44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Sep 10 15:55:34.362750 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:55:34.362207    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-e48e16-openshift-default-6bc647b4bb-f6j44" podUID="beb30354-999e-491d-b6de-d3c734bc3b64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Sep 10 15:55:35.362317 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:55:35.361874    2559 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-e48e16-openshift-default-6bc647b4bb-f6j44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Sep 10 15:55:35.362317 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:55:35.361939    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-e48e16-openshift-default-6bc647b4bb-f6j44" podUID="beb30354-999e-491d-b6de-d3c734bc3b64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Sep 10 15:55:36.362223 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:55:36.362184    2559 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-e48e16-openshift-default-6bc647b4bb-f6j44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Sep 10 15:55:36.362588 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:55:36.362540    2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-e48e16-openshift-default-6bc647b4bb-f6j44" podUID="beb30354-999e-491d-b6de-d3c734bc3b64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Sep 10 15:55:37.366067 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:55:37.366036    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-e48e16-openshift-default-6bc647b4bb-f6j44"
Sep 10 15:55:37.366480 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:55:37.366318    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-e48e16-openshift-default-6bc647b4bb-f6j44"
Sep 10 15:55:37.367135 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:55:37.367116    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-e48e16-openshift-default-6bc647b4bb-f6j44"
Sep 10 15:55:58.112360 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:55:58.112320    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bs7hd"]
Sep 10 15:56:00.903433 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:56:00.903401    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-e48e16-openshift-default-6bc647b4bb-f6j44"]
Sep 10 15:56:00.904061 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:56:00.903669    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-e48e16-openshift-default-6bc647b4bb-f6j44" podUID="beb30354-999e-491d-b6de-d3c734bc3b64" containerName="istio-proxy" containerID="cri-o://69a677378efe4c6fa71c5b5c1f7cc4e3f054e954ca318adb87fa5dc87e3476f6" gracePeriod=30
Sep 10 15:56:06.171399 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:56:06.171336    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-e48e16-openshift-default-6bc647b4bb-f6j44"
Sep 10 15:56:06.191724 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:56:06.191691    2559 generic.go:356] "Generic (PLEG): container finished" podID="beb30354-999e-491d-b6de-d3c734bc3b64" containerID="69a677378efe4c6fa71c5b5c1f7cc4e3f054e954ca318adb87fa5dc87e3476f6" exitCode=0
Sep 10 15:56:06.191880 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:56:06.191743    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-e48e16-openshift-default-6bc647b4bb-f6j44"
Sep 10 15:56:06.191880 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:56:06.191767    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-e48e16-openshift-default-6bc647b4bb-f6j44" event={"ID":"beb30354-999e-491d-b6de-d3c734bc3b64","Type":"ContainerDied","Data":"69a677378efe4c6fa71c5b5c1f7cc4e3f054e954ca318adb87fa5dc87e3476f6"}
Sep 10 15:56:06.191880 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:56:06.191809    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-e48e16-openshift-default-6bc647b4bb-f6j44" event={"ID":"beb30354-999e-491d-b6de-d3c734bc3b64","Type":"ContainerDied","Data":"f323f717e64e3237f6bdabf93966f63f8ac4dd9e82966bb02b94445b4b1daa48"}
Sep 10 15:56:06.191880 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:56:06.191832    2559 scope.go:117] "RemoveContainer" containerID="69a677378efe4c6fa71c5b5c1f7cc4e3f054e954ca318adb87fa5dc87e3476f6"
Sep 10 15:56:06.202282 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:56:06.202252    2559 scope.go:117] "RemoveContainer" containerID="6d524695bfd366c255943e58bb2f2170f517be9208fdac7d9cf11973e2cba4d8"
Sep 10 15:56:06.202495 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:56:06.202475    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cjt6r\" (UniqueName: \"kubernetes.io/projected/beb30354-999e-491d-b6de-d3c734bc3b64-kube-api-access-cjt6r\") pod \"beb30354-999e-491d-b6de-d3c734bc3b64\" (UID: \"beb30354-999e-491d-b6de-d3c734bc3b64\") "
Sep 10 15:56:06.202623 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:56:06.202511    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/beb30354-999e-491d-b6de-d3c734bc3b64-istio-podinfo\") pod \"beb30354-999e-491d-b6de-d3c734bc3b64\" (UID: \"beb30354-999e-491d-b6de-d3c734bc3b64\") "
Sep 10 15:56:06.202623 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:56:06.202538    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/beb30354-999e-491d-b6de-d3c734bc3b64-istio-data\") pod \"beb30354-999e-491d-b6de-d3c734bc3b64\" (UID: \"beb30354-999e-491d-b6de-d3c734bc3b64\") "
Sep 10 15:56:06.202623 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:56:06.202599    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/beb30354-999e-491d-b6de-d3c734bc3b64-istio-token\") pod \"beb30354-999e-491d-b6de-d3c734bc3b64\" (UID: \"beb30354-999e-491d-b6de-d3c734bc3b64\") "
Sep 10 15:56:06.202782 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:56:06.202626    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/beb30354-999e-491d-b6de-d3c734bc3b64-credential-socket\") pod \"beb30354-999e-491d-b6de-d3c734bc3b64\" (UID: \"beb30354-999e-491d-b6de-d3c734bc3b64\") "
Sep 10 15:56:06.202782 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:56:06.202654    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/beb30354-999e-491d-b6de-d3c734bc3b64-workload-certs\") pod \"beb30354-999e-491d-b6de-d3c734bc3b64\" (UID: \"beb30354-999e-491d-b6de-d3c734bc3b64\") "
Sep 10 15:56:06.202782 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:56:06.202681    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/beb30354-999e-491d-b6de-d3c734bc3b64-workload-socket\") pod \"beb30354-999e-491d-b6de-d3c734bc3b64\" (UID: \"beb30354-999e-491d-b6de-d3c734bc3b64\") "
Sep 10 15:56:06.202782 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:56:06.202713    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/beb30354-999e-491d-b6de-d3c734bc3b64-istiod-ca-cert\") pod \"beb30354-999e-491d-b6de-d3c734bc3b64\" (UID: \"beb30354-999e-491d-b6de-d3c734bc3b64\") "
Sep 10 15:56:06.202782 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:56:06.202759    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/beb30354-999e-491d-b6de-d3c734bc3b64-istio-envoy\") pod \"beb30354-999e-491d-b6de-d3c734bc3b64\" (UID: \"beb30354-999e-491d-b6de-d3c734bc3b64\") "
Sep 10 15:56:06.203025 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:56:06.202976    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/beb30354-999e-491d-b6de-d3c734bc3b64-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "beb30354-999e-491d-b6de-d3c734bc3b64" (UID: "beb30354-999e-491d-b6de-d3c734bc3b64"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:56:06.203115 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:56:06.203092    2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/beb30354-999e-491d-b6de-d3c734bc3b64-credential-socket\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:56:06.203115 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:56:06.203087    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/beb30354-999e-491d-b6de-d3c734bc3b64-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "beb30354-999e-491d-b6de-d3c734bc3b64" (UID: "beb30354-999e-491d-b6de-d3c734bc3b64"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:56:06.203370 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:56:06.203198    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/beb30354-999e-491d-b6de-d3c734bc3b64-istio-data" (OuterVolumeSpecName: "istio-data") pod "beb30354-999e-491d-b6de-d3c734bc3b64" (UID: "beb30354-999e-491d-b6de-d3c734bc3b64"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:56:06.203461 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:56:06.203377    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/beb30354-999e-491d-b6de-d3c734bc3b64-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "beb30354-999e-491d-b6de-d3c734bc3b64" (UID: "beb30354-999e-491d-b6de-d3c734bc3b64"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:56:06.203535 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:56:06.203495    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/beb30354-999e-491d-b6de-d3c734bc3b64-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "beb30354-999e-491d-b6de-d3c734bc3b64" (UID: "beb30354-999e-491d-b6de-d3c734bc3b64"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:56:06.205173 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:56:06.205141    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/beb30354-999e-491d-b6de-d3c734bc3b64-kube-api-access-cjt6r" (OuterVolumeSpecName: "kube-api-access-cjt6r") pod "beb30354-999e-491d-b6de-d3c734bc3b64" (UID: "beb30354-999e-491d-b6de-d3c734bc3b64"). InnerVolumeSpecName "kube-api-access-cjt6r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:56:06.205290 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:56:06.205271    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/beb30354-999e-491d-b6de-d3c734bc3b64-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "beb30354-999e-491d-b6de-d3c734bc3b64" (UID: "beb30354-999e-491d-b6de-d3c734bc3b64"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 15:56:06.205368 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:56:06.205284    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/beb30354-999e-491d-b6de-d3c734bc3b64-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "beb30354-999e-491d-b6de-d3c734bc3b64" (UID: "beb30354-999e-491d-b6de-d3c734bc3b64"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:56:06.205433 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:56:06.205391    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/beb30354-999e-491d-b6de-d3c734bc3b64-istio-token" (OuterVolumeSpecName: "istio-token") pod "beb30354-999e-491d-b6de-d3c734bc3b64" (UID: "beb30354-999e-491d-b6de-d3c734bc3b64"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:56:06.224356 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:56:06.224328    2559 scope.go:117] "RemoveContainer" containerID="69a677378efe4c6fa71c5b5c1f7cc4e3f054e954ca318adb87fa5dc87e3476f6"
Sep 10 15:56:06.224785 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:56:06.224756    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69a677378efe4c6fa71c5b5c1f7cc4e3f054e954ca318adb87fa5dc87e3476f6\": container with ID starting with 69a677378efe4c6fa71c5b5c1f7cc4e3f054e954ca318adb87fa5dc87e3476f6 not found: ID does not exist" containerID="69a677378efe4c6fa71c5b5c1f7cc4e3f054e954ca318adb87fa5dc87e3476f6"
Sep 10 15:56:06.224860 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:56:06.224802    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69a677378efe4c6fa71c5b5c1f7cc4e3f054e954ca318adb87fa5dc87e3476f6"} err="failed to get container status \"69a677378efe4c6fa71c5b5c1f7cc4e3f054e954ca318adb87fa5dc87e3476f6\": rpc error: code = NotFound desc = could not find container \"69a677378efe4c6fa71c5b5c1f7cc4e3f054e954ca318adb87fa5dc87e3476f6\": container with ID starting with 69a677378efe4c6fa71c5b5c1f7cc4e3f054e954ca318adb87fa5dc87e3476f6 not found: ID does not exist"
Sep 10 15:56:06.224860 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:56:06.224831    2559 scope.go:117] "RemoveContainer" containerID="6d524695bfd366c255943e58bb2f2170f517be9208fdac7d9cf11973e2cba4d8"
Sep 10 15:56:06.225156 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:56:06.225137    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d524695bfd366c255943e58bb2f2170f517be9208fdac7d9cf11973e2cba4d8\": container with ID starting with 6d524695bfd366c255943e58bb2f2170f517be9208fdac7d9cf11973e2cba4d8 not found: ID does not exist" containerID="6d524695bfd366c255943e58bb2f2170f517be9208fdac7d9cf11973e2cba4d8"
Sep 10 15:56:06.225205 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:56:06.225165    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d524695bfd366c255943e58bb2f2170f517be9208fdac7d9cf11973e2cba4d8"} err="failed to get container status \"6d524695bfd366c255943e58bb2f2170f517be9208fdac7d9cf11973e2cba4d8\": rpc error: code = NotFound desc = could not find container \"6d524695bfd366c255943e58bb2f2170f517be9208fdac7d9cf11973e2cba4d8\": container with ID starting with 6d524695bfd366c255943e58bb2f2170f517be9208fdac7d9cf11973e2cba4d8 not found: ID does not exist"
Sep 10 15:56:06.303899 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:56:06.303866    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjt6r\" (UniqueName: \"kubernetes.io/projected/beb30354-999e-491d-b6de-d3c734bc3b64-kube-api-access-cjt6r\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:56:06.303899 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:56:06.303897    2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/beb30354-999e-491d-b6de-d3c734bc3b64-istio-podinfo\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:56:06.304126 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:56:06.303911    2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/beb30354-999e-491d-b6de-d3c734bc3b64-istio-data\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:56:06.304126 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:56:06.303923    2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/beb30354-999e-491d-b6de-d3c734bc3b64-istio-token\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:56:06.304126 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:56:06.303934    2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/beb30354-999e-491d-b6de-d3c734bc3b64-workload-certs\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:56:06.304126 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:56:06.303945    2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/beb30354-999e-491d-b6de-d3c734bc3b64-workload-socket\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:56:06.304126 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:56:06.303955    2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/beb30354-999e-491d-b6de-d3c734bc3b64-istiod-ca-cert\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:56:06.304126 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:56:06.303966    2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/beb30354-999e-491d-b6de-d3c734bc3b64-istio-envoy\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:56:06.516697 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:56:06.516662    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-e48e16-openshift-default-6bc647b4bb-f6j44"]
Sep 10 15:56:06.523739 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:56:06.523710    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-e48e16-openshift-default-6bc647b4bb-f6j44"]
Sep 10 15:56:06.786784 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:56:06.786691    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="beb30354-999e-491d-b6de-d3c734bc3b64" path="/var/lib/kubelet/pods/beb30354-999e-491d-b6de-d3c734bc3b64/volumes"
Sep 10 15:56:18.098063 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:56:18.098024    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bs7hd"]
Sep 10 15:56:20.208233 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:56:20.208193    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8cb756-7bcbfc58d8-67pz2"]
Sep 10 15:56:20.208739 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:56:20.208448    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8cb756-7bcbfc58d8-67pz2" podUID="d252c941-0566-44bd-a79b-f433db7fbd51" containerName="maas-api" containerID="cri-o://550d7b331d1f7e068741830a52e36fcbba90ad849abcd43dea73c0c9d1a1ad87" gracePeriod=30
Sep 10 15:56:20.461145 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:56:20.461088    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8cb756-7bcbfc58d8-67pz2"
Sep 10 15:56:20.527265 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:56:20.527237    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tdblp\" (UniqueName: \"kubernetes.io/projected/d252c941-0566-44bd-a79b-f433db7fbd51-kube-api-access-tdblp\") pod \"d252c941-0566-44bd-a79b-f433db7fbd51\" (UID: \"d252c941-0566-44bd-a79b-f433db7fbd51\") "
Sep 10 15:56:20.527412 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:56:20.527271    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d252c941-0566-44bd-a79b-f433db7fbd51-openshift-service-ca\") pod \"d252c941-0566-44bd-a79b-f433db7fbd51\" (UID: \"d252c941-0566-44bd-a79b-f433db7fbd51\") "
Sep 10 15:56:20.527412 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:56:20.527324    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d252c941-0566-44bd-a79b-f433db7fbd51-maas-api-tls\") pod \"d252c941-0566-44bd-a79b-f433db7fbd51\" (UID: \"d252c941-0566-44bd-a79b-f433db7fbd51\") "
Sep 10 15:56:20.527412 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:56:20.527361    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d252c941-0566-44bd-a79b-f433db7fbd51-metrics-certs\") pod \"d252c941-0566-44bd-a79b-f433db7fbd51\" (UID: \"d252c941-0566-44bd-a79b-f433db7fbd51\") "
Sep 10 15:56:20.527735 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:56:20.527711    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d252c941-0566-44bd-a79b-f433db7fbd51-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d252c941-0566-44bd-a79b-f433db7fbd51" (UID: "d252c941-0566-44bd-a79b-f433db7fbd51"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:56:20.529401 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:56:20.529355    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d252c941-0566-44bd-a79b-f433db7fbd51-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d252c941-0566-44bd-a79b-f433db7fbd51" (UID: "d252c941-0566-44bd-a79b-f433db7fbd51"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:56:20.529488 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:56:20.529431    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d252c941-0566-44bd-a79b-f433db7fbd51-kube-api-access-tdblp" (OuterVolumeSpecName: "kube-api-access-tdblp") pod "d252c941-0566-44bd-a79b-f433db7fbd51" (UID: "d252c941-0566-44bd-a79b-f433db7fbd51"). InnerVolumeSpecName "kube-api-access-tdblp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:56:20.529488 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:56:20.529434    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d252c941-0566-44bd-a79b-f433db7fbd51-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d252c941-0566-44bd-a79b-f433db7fbd51" (UID: "d252c941-0566-44bd-a79b-f433db7fbd51"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:56:20.628856 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:56:20.628826    2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d252c941-0566-44bd-a79b-f433db7fbd51-metrics-certs\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:56:20.628856 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:56:20.628853    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdblp\" (UniqueName: \"kubernetes.io/projected/d252c941-0566-44bd-a79b-f433db7fbd51-kube-api-access-tdblp\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:56:20.629065 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:56:20.628867    2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d252c941-0566-44bd-a79b-f433db7fbd51-openshift-service-ca\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:56:20.629065 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:56:20.628878    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d252c941-0566-44bd-a79b-f433db7fbd51-maas-api-tls\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:56:21.275619 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:56:21.275571    2559 generic.go:356] "Generic (PLEG): container finished" podID="d252c941-0566-44bd-a79b-f433db7fbd51" containerID="550d7b331d1f7e068741830a52e36fcbba90ad849abcd43dea73c0c9d1a1ad87" exitCode=0
Sep 10 15:56:21.276059 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:56:21.275637    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8cb756-7bcbfc58d8-67pz2"
Sep 10 15:56:21.276059 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:56:21.275669    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8cb756-7bcbfc58d8-67pz2" event={"ID":"d252c941-0566-44bd-a79b-f433db7fbd51","Type":"ContainerDied","Data":"550d7b331d1f7e068741830a52e36fcbba90ad849abcd43dea73c0c9d1a1ad87"}
Sep 10 15:56:21.276059 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:56:21.275726    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8cb756-7bcbfc58d8-67pz2" event={"ID":"d252c941-0566-44bd-a79b-f433db7fbd51","Type":"ContainerDied","Data":"6e8478df7e95ed471484dd177f6a637dfe9b53c74db73cd32953130c213296c9"}
Sep 10 15:56:21.276059 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:56:21.275750    2559 scope.go:117] "RemoveContainer" containerID="550d7b331d1f7e068741830a52e36fcbba90ad849abcd43dea73c0c9d1a1ad87"
Sep 10 15:56:21.285042 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:56:21.285026    2559 scope.go:117] "RemoveContainer" containerID="550d7b331d1f7e068741830a52e36fcbba90ad849abcd43dea73c0c9d1a1ad87"
Sep 10 15:56:21.285313 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:56:21.285293    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"550d7b331d1f7e068741830a52e36fcbba90ad849abcd43dea73c0c9d1a1ad87\": container with ID starting with 550d7b331d1f7e068741830a52e36fcbba90ad849abcd43dea73c0c9d1a1ad87 not found: ID does not exist" containerID="550d7b331d1f7e068741830a52e36fcbba90ad849abcd43dea73c0c9d1a1ad87"
Sep 10 15:56:21.285398 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:56:21.285326    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"550d7b331d1f7e068741830a52e36fcbba90ad849abcd43dea73c0c9d1a1ad87"} err="failed to get container status \"550d7b331d1f7e068741830a52e36fcbba90ad849abcd43dea73c0c9d1a1ad87\": rpc error: code = NotFound desc = could not find container \"550d7b331d1f7e068741830a52e36fcbba90ad849abcd43dea73c0c9d1a1ad87\": container with ID starting with 550d7b331d1f7e068741830a52e36fcbba90ad849abcd43dea73c0c9d1a1ad87 not found: ID does not exist"
Sep 10 15:56:21.296617 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:56:21.296594    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8cb756-7bcbfc58d8-67pz2"]
Sep 10 15:56:21.301176 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:56:21.301153    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8cb756-7bcbfc58d8-67pz2"]
Sep 10 15:56:22.784919 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:56:22.784886    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d252c941-0566-44bd-a79b-f433db7fbd51" path="/var/lib/kubelet/pods/d252c941-0566-44bd-a79b-f433db7fbd51/volumes"
Sep 10 15:56:25.822401 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:56:25.822360    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-8cb756/ipp-route-8cb756-kserve-596bdc755-27sqr"]
Sep 10 15:56:31.494672 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:56:31.494634    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bs7hd"]
Sep 10 15:56:40.842514 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:56:40.842460    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-8cb756/ipp-route-8cb756-kserve-596bdc755-27sqr" podUID="1a9082bf-20cc-48b1-92fa-c0bef5f5ab46" containerName="main" containerID="cri-o://f20a5c9c23df39e77b41d177dfb707544d13fedd7177e6c2e4a60492f9fb7e5b" gracePeriod=45
Sep 10 15:56:41.097288 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:56:41.097226    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-8cb756/ipp-route-8cb756-kserve-596bdc755-27sqr"
Sep 10 15:56:41.214294 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:56:41.214260    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a9082bf-20cc-48b1-92fa-c0bef5f5ab46-dshm\") pod \"1a9082bf-20cc-48b1-92fa-c0bef5f5ab46\" (UID: \"1a9082bf-20cc-48b1-92fa-c0bef5f5ab46\") "
Sep 10 15:56:41.214507 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:56:41.214370    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a9082bf-20cc-48b1-92fa-c0bef5f5ab46-model-cache\") pod \"1a9082bf-20cc-48b1-92fa-c0bef5f5ab46\" (UID: \"1a9082bf-20cc-48b1-92fa-c0bef5f5ab46\") "
Sep 10 15:56:41.214507 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:56:41.214407    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a9082bf-20cc-48b1-92fa-c0bef5f5ab46-home\") pod \"1a9082bf-20cc-48b1-92fa-c0bef5f5ab46\" (UID: \"1a9082bf-20cc-48b1-92fa-c0bef5f5ab46\") "
Sep 10 15:56:41.214507 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:56:41.214453    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a9082bf-20cc-48b1-92fa-c0bef5f5ab46-tls-certs\") pod \"1a9082bf-20cc-48b1-92fa-c0bef5f5ab46\" (UID: \"1a9082bf-20cc-48b1-92fa-c0bef5f5ab46\") "
Sep 10 15:56:41.214507 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:56:41.214489    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a9082bf-20cc-48b1-92fa-c0bef5f5ab46-tmp-dir\") pod \"1a9082bf-20cc-48b1-92fa-c0bef5f5ab46\" (UID: \"1a9082bf-20cc-48b1-92fa-c0bef5f5ab46\") "
Sep 10 15:56:41.214763 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:56:41.214517    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9h86b\" (UniqueName: \"kubernetes.io/projected/1a9082bf-20cc-48b1-92fa-c0bef5f5ab46-kube-api-access-9h86b\") pod \"1a9082bf-20cc-48b1-92fa-c0bef5f5ab46\" (UID: \"1a9082bf-20cc-48b1-92fa-c0bef5f5ab46\") "
Sep 10 15:56:41.214763 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:56:41.214677    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a9082bf-20cc-48b1-92fa-c0bef5f5ab46-model-cache" (OuterVolumeSpecName: "model-cache") pod "1a9082bf-20cc-48b1-92fa-c0bef5f5ab46" (UID: "1a9082bf-20cc-48b1-92fa-c0bef5f5ab46"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:56:41.214763 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:56:41.214696    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a9082bf-20cc-48b1-92fa-c0bef5f5ab46-home" (OuterVolumeSpecName: "home") pod "1a9082bf-20cc-48b1-92fa-c0bef5f5ab46" (UID: "1a9082bf-20cc-48b1-92fa-c0bef5f5ab46"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:56:41.214898 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:56:41.214820    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a9082bf-20cc-48b1-92fa-c0bef5f5ab46-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1a9082bf-20cc-48b1-92fa-c0bef5f5ab46" (UID: "1a9082bf-20cc-48b1-92fa-c0bef5f5ab46"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:56:41.214941 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:56:41.214925    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a9082bf-20cc-48b1-92fa-c0bef5f5ab46-model-cache\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:56:41.214981 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:56:41.214946    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a9082bf-20cc-48b1-92fa-c0bef5f5ab46-home\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:56:41.214981 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:56:41.214959    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a9082bf-20cc-48b1-92fa-c0bef5f5ab46-tmp-dir\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:56:41.216562 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:56:41.216536    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a9082bf-20cc-48b1-92fa-c0bef5f5ab46-dshm" (OuterVolumeSpecName: "dshm") pod "1a9082bf-20cc-48b1-92fa-c0bef5f5ab46" (UID: "1a9082bf-20cc-48b1-92fa-c0bef5f5ab46"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:56:41.216694 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:56:41.216569    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a9082bf-20cc-48b1-92fa-c0bef5f5ab46-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1a9082bf-20cc-48b1-92fa-c0bef5f5ab46" (UID: "1a9082bf-20cc-48b1-92fa-c0bef5f5ab46"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:56:41.216694 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:56:41.216673    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a9082bf-20cc-48b1-92fa-c0bef5f5ab46-kube-api-access-9h86b" (OuterVolumeSpecName: "kube-api-access-9h86b") pod "1a9082bf-20cc-48b1-92fa-c0bef5f5ab46" (UID: "1a9082bf-20cc-48b1-92fa-c0bef5f5ab46"). InnerVolumeSpecName "kube-api-access-9h86b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:56:41.315682 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:56:41.315642    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a9082bf-20cc-48b1-92fa-c0bef5f5ab46-dshm\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:56:41.315682 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:56:41.315679    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a9082bf-20cc-48b1-92fa-c0bef5f5ab46-tls-certs\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:56:41.315902 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:56:41.315694    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9h86b\" (UniqueName: \"kubernetes.io/projected/1a9082bf-20cc-48b1-92fa-c0bef5f5ab46-kube-api-access-9h86b\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:56:41.393405 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:56:41.393303    2559 generic.go:356] "Generic (PLEG): container finished" podID="1a9082bf-20cc-48b1-92fa-c0bef5f5ab46" containerID="f20a5c9c23df39e77b41d177dfb707544d13fedd7177e6c2e4a60492f9fb7e5b" exitCode=0
Sep 10 15:56:41.393405 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:56:41.393359    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-8cb756/ipp-route-8cb756-kserve-596bdc755-27sqr"
Sep 10 15:56:41.393405 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:56:41.393389    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-8cb756/ipp-route-8cb756-kserve-596bdc755-27sqr" event={"ID":"1a9082bf-20cc-48b1-92fa-c0bef5f5ab46","Type":"ContainerDied","Data":"f20a5c9c23df39e77b41d177dfb707544d13fedd7177e6c2e4a60492f9fb7e5b"}
Sep 10 15:56:41.393721 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:56:41.393434    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-8cb756/ipp-route-8cb756-kserve-596bdc755-27sqr" event={"ID":"1a9082bf-20cc-48b1-92fa-c0bef5f5ab46","Type":"ContainerDied","Data":"0383aea8116629dfed8561fd9206d089366d940adfa37e2beea42fa80652da90"}
Sep 10 15:56:41.393721 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:56:41.393449    2559 scope.go:117] "RemoveContainer" containerID="f20a5c9c23df39e77b41d177dfb707544d13fedd7177e6c2e4a60492f9fb7e5b"
Sep 10 15:56:41.403476 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:56:41.403455    2559 scope.go:117] "RemoveContainer" containerID="f20a5c9c23df39e77b41d177dfb707544d13fedd7177e6c2e4a60492f9fb7e5b"
Sep 10 15:56:41.403781 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:56:41.403760    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f20a5c9c23df39e77b41d177dfb707544d13fedd7177e6c2e4a60492f9fb7e5b\": container with ID starting with f20a5c9c23df39e77b41d177dfb707544d13fedd7177e6c2e4a60492f9fb7e5b not found: ID does not exist" containerID="f20a5c9c23df39e77b41d177dfb707544d13fedd7177e6c2e4a60492f9fb7e5b"
Sep 10 15:56:41.403844 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:56:41.403790    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f20a5c9c23df39e77b41d177dfb707544d13fedd7177e6c2e4a60492f9fb7e5b"} err="failed to get container status \"f20a5c9c23df39e77b41d177dfb707544d13fedd7177e6c2e4a60492f9fb7e5b\": rpc error: code = NotFound desc = could not find container \"f20a5c9c23df39e77b41d177dfb707544d13fedd7177e6c2e4a60492f9fb7e5b\": container with ID starting with f20a5c9c23df39e77b41d177dfb707544d13fedd7177e6c2e4a60492f9fb7e5b not found: ID does not exist"
Sep 10 15:56:41.422124 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:56:41.422097    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-8cb756/ipp-route-8cb756-kserve-596bdc755-27sqr"]
Sep 10 15:56:41.427457 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:56:41.427436    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-8cb756/ipp-route-8cb756-kserve-596bdc755-27sqr"]
Sep 10 15:56:42.785890 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:56:42.785858    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a9082bf-20cc-48b1-92fa-c0bef5f5ab46" path="/var/lib/kubelet/pods/1a9082bf-20cc-48b1-92fa-c0bef5f5ab46/volumes"
Sep 10 15:57:26.812120 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:57:26.812083    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-6d929c-6bf78f68b5-9b7nn"]
Sep 10 15:57:26.812648 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:57:26.812361    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6d929c-6bf78f68b5-9b7nn" podUID="b878b42b-4947-4d62-a1c1-829389306f92" containerName="maas-api" containerID="cri-o://f84a59377514d9ef408120016e463361253558796ea8f0ce1ed7a011b84be89e" gracePeriod=30
Sep 10 15:57:27.063627 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:57:27.063559    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6d929c-6bf78f68b5-9b7nn"
Sep 10 15:57:27.230543 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:57:27.230503    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b878b42b-4947-4d62-a1c1-829389306f92-metrics-certs\") pod \"b878b42b-4947-4d62-a1c1-829389306f92\" (UID: \"b878b42b-4947-4d62-a1c1-829389306f92\") "
Sep 10 15:57:27.230768 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:57:27.230564    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b878b42b-4947-4d62-a1c1-829389306f92-openshift-service-ca\") pod \"b878b42b-4947-4d62-a1c1-829389306f92\" (UID: \"b878b42b-4947-4d62-a1c1-829389306f92\") "
Sep 10 15:57:27.230768 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:57:27.230601    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b878b42b-4947-4d62-a1c1-829389306f92-maas-api-tls\") pod \"b878b42b-4947-4d62-a1c1-829389306f92\" (UID: \"b878b42b-4947-4d62-a1c1-829389306f92\") "
Sep 10 15:57:27.230768 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:57:27.230633    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lbqqz\" (UniqueName: \"kubernetes.io/projected/b878b42b-4947-4d62-a1c1-829389306f92-kube-api-access-lbqqz\") pod \"b878b42b-4947-4d62-a1c1-829389306f92\" (UID: \"b878b42b-4947-4d62-a1c1-829389306f92\") "
Sep 10 15:57:27.230949 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:57:27.230930    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b878b42b-4947-4d62-a1c1-829389306f92-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b878b42b-4947-4d62-a1c1-829389306f92" (UID: "b878b42b-4947-4d62-a1c1-829389306f92"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:57:27.232756 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:57:27.232731    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b878b42b-4947-4d62-a1c1-829389306f92-kube-api-access-lbqqz" (OuterVolumeSpecName: "kube-api-access-lbqqz") pod "b878b42b-4947-4d62-a1c1-829389306f92" (UID: "b878b42b-4947-4d62-a1c1-829389306f92"). InnerVolumeSpecName "kube-api-access-lbqqz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:57:27.232756 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:57:27.232744    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b878b42b-4947-4d62-a1c1-829389306f92-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b878b42b-4947-4d62-a1c1-829389306f92" (UID: "b878b42b-4947-4d62-a1c1-829389306f92"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:57:27.232929 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:57:27.232774    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b878b42b-4947-4d62-a1c1-829389306f92-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b878b42b-4947-4d62-a1c1-829389306f92" (UID: "b878b42b-4947-4d62-a1c1-829389306f92"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:57:27.331642 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:57:27.331515    2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b878b42b-4947-4d62-a1c1-829389306f92-metrics-certs\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:57:27.331642 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:57:27.331560    2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b878b42b-4947-4d62-a1c1-829389306f92-openshift-service-ca\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:57:27.331642 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:57:27.331570    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b878b42b-4947-4d62-a1c1-829389306f92-maas-api-tls\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:57:27.331642 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:57:27.331604    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbqqz\" (UniqueName: \"kubernetes.io/projected/b878b42b-4947-4d62-a1c1-829389306f92-kube-api-access-lbqqz\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:57:27.646749 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:57:27.646652    2559 generic.go:356] "Generic (PLEG): container finished" podID="b878b42b-4947-4d62-a1c1-829389306f92" containerID="f84a59377514d9ef408120016e463361253558796ea8f0ce1ed7a011b84be89e" exitCode=0
Sep 10 15:57:27.646749 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:57:27.646705    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6d929c-6bf78f68b5-9b7nn"
Sep 10 15:57:27.646971 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:57:27.646738    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6d929c-6bf78f68b5-9b7nn" event={"ID":"b878b42b-4947-4d62-a1c1-829389306f92","Type":"ContainerDied","Data":"f84a59377514d9ef408120016e463361253558796ea8f0ce1ed7a011b84be89e"}
Sep 10 15:57:27.646971 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:57:27.646796    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6d929c-6bf78f68b5-9b7nn" event={"ID":"b878b42b-4947-4d62-a1c1-829389306f92","Type":"ContainerDied","Data":"92d2450700c4c39feac77e086b8347faed1c91af860f619a308a7f97f8b64338"}
Sep 10 15:57:27.646971 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:57:27.646819    2559 scope.go:117] "RemoveContainer" containerID="f84a59377514d9ef408120016e463361253558796ea8f0ce1ed7a011b84be89e"
Sep 10 15:57:27.659631 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:57:27.659611    2559 scope.go:117] "RemoveContainer" containerID="f84a59377514d9ef408120016e463361253558796ea8f0ce1ed7a011b84be89e"
Sep 10 15:57:27.659922 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:57:27.659902    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f84a59377514d9ef408120016e463361253558796ea8f0ce1ed7a011b84be89e\": container with ID starting with f84a59377514d9ef408120016e463361253558796ea8f0ce1ed7a011b84be89e not found: ID does not exist" containerID="f84a59377514d9ef408120016e463361253558796ea8f0ce1ed7a011b84be89e"
Sep 10 15:57:27.659982 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:57:27.659931    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f84a59377514d9ef408120016e463361253558796ea8f0ce1ed7a011b84be89e"} err="failed to get container status \"f84a59377514d9ef408120016e463361253558796ea8f0ce1ed7a011b84be89e\": rpc error: code = NotFound desc = could not find container \"f84a59377514d9ef408120016e463361253558796ea8f0ce1ed7a011b84be89e\": container with ID starting with f84a59377514d9ef408120016e463361253558796ea8f0ce1ed7a011b84be89e not found: ID does not exist"
Sep 10 15:57:27.674410 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:57:27.674388    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-6d929c-6bf78f68b5-9b7nn"]
Sep 10 15:57:27.679619 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:57:27.679599    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-6d929c-6bf78f68b5-9b7nn"]
Sep 10 15:57:28.785506 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:57:28.785474    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b878b42b-4947-4d62-a1c1-829389306f92" path="/var/lib/kubelet/pods/b878b42b-4947-4d62-a1c1-829389306f92/volumes"
Sep 10 15:57:30.008168 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:57:30.008129    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bs7hd"]
Sep 10 15:57:34.455120 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:57:34.455080    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-6d929c-openshift-default-5cd456f5b6-4j7rh"]
Sep 10 15:57:34.455591 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:57:34.455373    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-6d929c-openshift-default-5cd456f5b6-4j7rh" podUID="3f28530b-4e55-487a-8ed4-540d104609bb" containerName="istio-proxy" containerID="cri-o://efa31517c47cdd32fe277ca2a393555360d7595301b5792b4655835135a79dbe" gracePeriod=30
Sep 10 15:57:34.812415 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:57:34.812333    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c7257b-78l8l"]
Sep 10 15:57:34.812704 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:57:34.812692    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="beb30354-999e-491d-b6de-d3c734bc3b64" containerName="istio-proxy"
Sep 10 15:57:34.812761 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:57:34.812705    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="beb30354-999e-491d-b6de-d3c734bc3b64" containerName="istio-proxy"
Sep 10 15:57:34.812761 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:57:34.812720    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a9082bf-20cc-48b1-92fa-c0bef5f5ab46" containerName="main"
Sep 10 15:57:34.812761 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:57:34.812725    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a9082bf-20cc-48b1-92fa-c0bef5f5ab46" containerName="main"
Sep 10 15:57:34.812761 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:57:34.812733    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="beb30354-999e-491d-b6de-d3c734bc3b64" containerName="istio-proxy"
Sep 10 15:57:34.812761 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:57:34.812738    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="beb30354-999e-491d-b6de-d3c734bc3b64" containerName="istio-proxy"
Sep 10 15:57:34.812761 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:57:34.812745    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b878b42b-4947-4d62-a1c1-829389306f92" containerName="maas-api"
Sep 10 15:57:34.812761 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:57:34.812750    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b878b42b-4947-4d62-a1c1-829389306f92" containerName="maas-api"
Sep 10 15:57:34.812761 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:57:34.812759    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d252c941-0566-44bd-a79b-f433db7fbd51" containerName="maas-api"
Sep 10 15:57:34.813001 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:57:34.812766    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d252c941-0566-44bd-a79b-f433db7fbd51" containerName="maas-api"
Sep 10 15:57:34.813001 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:57:34.812819    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a9082bf-20cc-48b1-92fa-c0bef5f5ab46" containerName="main"
Sep 10 15:57:34.813001 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:57:34.812828    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="beb30354-999e-491d-b6de-d3c734bc3b64" containerName="istio-proxy"
Sep 10 15:57:34.813001 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:57:34.812833    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="beb30354-999e-491d-b6de-d3c734bc3b64" containerName="istio-proxy"
Sep 10 15:57:34.813001 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:57:34.812841    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d252c941-0566-44bd-a79b-f433db7fbd51" containerName="maas-api"
Sep 10 15:57:34.813001 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:57:34.812847    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="b878b42b-4947-4d62-a1c1-829389306f92" containerName="maas-api"
Sep 10 15:57:34.816930 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:57:34.816912    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c7257b-78l8l"
Sep 10 15:57:34.819374 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:57:34.819350    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 15:57:34.819503 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:57:34.819485    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vfhtz\""
Sep 10 15:57:34.823926 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:57:34.823899    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c7257b-78l8l"]
Sep 10 15:57:35.010475 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:57:35.010437    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b24b80bb-823e-4f47-bf4a-8c0dde5b171e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-c7257b-78l8l\" (UID: \"b24b80bb-823e-4f47-bf4a-8c0dde5b171e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c7257b-78l8l"
Sep 10 15:57:35.111809 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:57:35.111711    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b24b80bb-823e-4f47-bf4a-8c0dde5b171e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-c7257b-78l8l\" (UID: \"b24b80bb-823e-4f47-bf4a-8c0dde5b171e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c7257b-78l8l"
Sep 10 15:57:35.112334 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:57:35.112312    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b24b80bb-823e-4f47-bf4a-8c0dde5b171e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-c7257b-78l8l\" (UID: \"b24b80bb-823e-4f47-bf4a-8c0dde5b171e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c7257b-78l8l"
Sep 10 15:57:35.128569 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:57:35.128541    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c7257b-78l8l"
Sep 10 15:57:35.401597 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:57:35.401558    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c7257b-78l8l"]
Sep 10 15:57:35.403405 ip-10-0-130-30 kubenswrapper[2559]: W0910 15:57:35.403361    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb24b80bb_823e_4f47_bf4a_8c0dde5b171e.slice/crio-d746acc90d3ca089bea53afb287f54c0baa51acaa6c97bd1e0db30790a47e5af WatchSource:0}: Error finding container d746acc90d3ca089bea53afb287f54c0baa51acaa6c97bd1e0db30790a47e5af: Status 404 returned error can't find the container with id d746acc90d3ca089bea53afb287f54c0baa51acaa6c97bd1e0db30790a47e5af
Sep 10 15:57:35.692240 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:57:35.692198    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c7257b-78l8l" event={"ID":"b24b80bb-823e-4f47-bf4a-8c0dde5b171e","Type":"ContainerStarted","Data":"d746acc90d3ca089bea53afb287f54c0baa51acaa6c97bd1e0db30790a47e5af"}
Sep 10 15:57:36.699171 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:57:36.699135    2559 generic.go:356] "Generic (PLEG): container finished" podID="b24b80bb-823e-4f47-bf4a-8c0dde5b171e" containerID="76e2ba23d0846142a8d250ab1c2fcf5fb757ef2ad8c1bceeadb5c99d7a4d286d" exitCode=0
Sep 10 15:57:36.699549 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:57:36.699219    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c7257b-78l8l" event={"ID":"b24b80bb-823e-4f47-bf4a-8c0dde5b171e","Type":"ContainerDied","Data":"76e2ba23d0846142a8d250ab1c2fcf5fb757ef2ad8c1bceeadb5c99d7a4d286d"}
Sep 10 15:57:37.831295 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:57:37.831269    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c7257b-78l8l"
Sep 10 15:57:37.931225 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:57:37.931195    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b24b80bb-823e-4f47-bf4a-8c0dde5b171e-maas-api-service-ca\") pod \"b24b80bb-823e-4f47-bf4a-8c0dde5b171e\" (UID: \"b24b80bb-823e-4f47-bf4a-8c0dde5b171e\") "
Sep 10 15:57:37.931560 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:57:37.931537    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b24b80bb-823e-4f47-bf4a-8c0dde5b171e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b24b80bb-823e-4f47-bf4a-8c0dde5b171e" (UID: "b24b80bb-823e-4f47-bf4a-8c0dde5b171e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:57:38.031893 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:57:38.031801    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b24b80bb-823e-4f47-bf4a-8c0dde5b171e-maas-api-service-ca\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:57:38.711278 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:57:38.711234    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c7257b-78l8l" event={"ID":"b24b80bb-823e-4f47-bf4a-8c0dde5b171e","Type":"ContainerDied","Data":"d746acc90d3ca089bea53afb287f54c0baa51acaa6c97bd1e0db30790a47e5af"}
Sep 10 15:57:38.711278 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:57:38.711263    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c7257b-78l8l"
Sep 10 15:57:38.711487 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:57:38.711293    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d746acc90d3ca089bea53afb287f54c0baa51acaa6c97bd1e0db30790a47e5af"
Sep 10 15:57:39.699213 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:57:39.699182    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-6d929c-openshift-default-5cd456f5b6-4j7rh"
Sep 10 15:57:39.718677 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:57:39.718599    2559 generic.go:356] "Generic (PLEG): container finished" podID="3f28530b-4e55-487a-8ed4-540d104609bb" containerID="efa31517c47cdd32fe277ca2a393555360d7595301b5792b4655835135a79dbe" exitCode=0
Sep 10 15:57:39.718677 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:57:39.718641    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-6d929c-openshift-default-5cd456f5b6-4j7rh"
Sep 10 15:57:39.718828 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:57:39.718677    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6d929c-openshift-default-5cd456f5b6-4j7rh" event={"ID":"3f28530b-4e55-487a-8ed4-540d104609bb","Type":"ContainerDied","Data":"efa31517c47cdd32fe277ca2a393555360d7595301b5792b4655835135a79dbe"}
Sep 10 15:57:39.718828 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:57:39.718763    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6d929c-openshift-default-5cd456f5b6-4j7rh" event={"ID":"3f28530b-4e55-487a-8ed4-540d104609bb","Type":"ContainerDied","Data":"dfada672aa91f494454ebe61bdcbef673a4d8bbe234eb58f9c5f376b0cb92d42"}
Sep 10 15:57:39.718828 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:57:39.718788    2559 scope.go:117] "RemoveContainer" containerID="efa31517c47cdd32fe277ca2a393555360d7595301b5792b4655835135a79dbe"
Sep 10 15:57:39.730845 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:57:39.730823    2559 scope.go:117] "RemoveContainer" containerID="5599d6092e025ae2b3be10148088fdfe69ade76230dcf5d243a07881401a5736"
Sep 10 15:57:39.739536 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:57:39.739514    2559 scope.go:117] "RemoveContainer" containerID="efa31517c47cdd32fe277ca2a393555360d7595301b5792b4655835135a79dbe"
Sep 10 15:57:39.739837 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:57:39.739819    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efa31517c47cdd32fe277ca2a393555360d7595301b5792b4655835135a79dbe\": container with ID starting with efa31517c47cdd32fe277ca2a393555360d7595301b5792b4655835135a79dbe not found: ID does not exist" containerID="efa31517c47cdd32fe277ca2a393555360d7595301b5792b4655835135a79dbe"
Sep 10 15:57:39.739918 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:57:39.739844    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efa31517c47cdd32fe277ca2a393555360d7595301b5792b4655835135a79dbe"} err="failed to get container status \"efa31517c47cdd32fe277ca2a393555360d7595301b5792b4655835135a79dbe\": rpc error: code = NotFound desc = could not find container \"efa31517c47cdd32fe277ca2a393555360d7595301b5792b4655835135a79dbe\": container with ID starting with efa31517c47cdd32fe277ca2a393555360d7595301b5792b4655835135a79dbe not found: ID does not exist"
Sep 10 15:57:39.739918 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:57:39.739869    2559 scope.go:117] "RemoveContainer" containerID="5599d6092e025ae2b3be10148088fdfe69ade76230dcf5d243a07881401a5736"
Sep 10 15:57:39.740149 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:57:39.740135    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5599d6092e025ae2b3be10148088fdfe69ade76230dcf5d243a07881401a5736\": container with ID starting with 5599d6092e025ae2b3be10148088fdfe69ade76230dcf5d243a07881401a5736 not found: ID does not exist" containerID="5599d6092e025ae2b3be10148088fdfe69ade76230dcf5d243a07881401a5736"
Sep 10 15:57:39.740193 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:57:39.740153    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5599d6092e025ae2b3be10148088fdfe69ade76230dcf5d243a07881401a5736"} err="failed to get container status \"5599d6092e025ae2b3be10148088fdfe69ade76230dcf5d243a07881401a5736\": rpc error: code = NotFound desc = could not find container \"5599d6092e025ae2b3be10148088fdfe69ade76230dcf5d243a07881401a5736\": container with ID starting with 5599d6092e025ae2b3be10148088fdfe69ade76230dcf5d243a07881401a5736 not found: ID does not exist"
Sep 10 15:57:39.755584 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:57:39.755547    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f28530b-4e55-487a-8ed4-540d104609bb-istio-podinfo\") pod \"3f28530b-4e55-487a-8ed4-540d104609bb\" (UID: \"3f28530b-4e55-487a-8ed4-540d104609bb\") "
Sep 10 15:57:39.755691 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:57:39.755629    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f28530b-4e55-487a-8ed4-540d104609bb-credential-socket\") pod \"3f28530b-4e55-487a-8ed4-540d104609bb\" (UID: \"3f28530b-4e55-487a-8ed4-540d104609bb\") "
Sep 10 15:57:39.755691 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:57:39.755675    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f28530b-4e55-487a-8ed4-540d104609bb-istio-token\") pod \"3f28530b-4e55-487a-8ed4-540d104609bb\" (UID: \"3f28530b-4e55-487a-8ed4-540d104609bb\") "
Sep 10 15:57:39.755810 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:57:39.755723    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f28530b-4e55-487a-8ed4-540d104609bb-istio-envoy\") pod \"3f28530b-4e55-487a-8ed4-540d104609bb\" (UID: \"3f28530b-4e55-487a-8ed4-540d104609bb\") "
Sep 10 15:57:39.755810 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:57:39.755770    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f28530b-4e55-487a-8ed4-540d104609bb-istio-data\") pod \"3f28530b-4e55-487a-8ed4-540d104609bb\" (UID: \"3f28530b-4e55-487a-8ed4-540d104609bb\") "
Sep 10 15:57:39.755810 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:57:39.755788    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f28530b-4e55-487a-8ed4-540d104609bb-istiod-ca-cert\") pod \"3f28530b-4e55-487a-8ed4-540d104609bb\" (UID: \"3f28530b-4e55-487a-8ed4-540d104609bb\") "
Sep 10 15:57:39.755962 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:57:39.755809    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fx4dw\" (UniqueName: \"kubernetes.io/projected/3f28530b-4e55-487a-8ed4-540d104609bb-kube-api-access-fx4dw\") pod \"3f28530b-4e55-487a-8ed4-540d104609bb\" (UID: \"3f28530b-4e55-487a-8ed4-540d104609bb\") "
Sep 10 15:57:39.755962 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:57:39.755831    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f28530b-4e55-487a-8ed4-540d104609bb-workload-certs\") pod \"3f28530b-4e55-487a-8ed4-540d104609bb\" (UID: \"3f28530b-4e55-487a-8ed4-540d104609bb\") "
Sep 10 15:57:39.755962 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:57:39.755849    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f28530b-4e55-487a-8ed4-540d104609bb-workload-socket\") pod \"3f28530b-4e55-487a-8ed4-540d104609bb\" (UID: \"3f28530b-4e55-487a-8ed4-540d104609bb\") "
Sep 10 15:57:39.756111 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:57:39.755967    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f28530b-4e55-487a-8ed4-540d104609bb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3f28530b-4e55-487a-8ed4-540d104609bb" (UID: "3f28530b-4e55-487a-8ed4-540d104609bb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:57:39.756213 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:57:39.756164    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f28530b-4e55-487a-8ed4-540d104609bb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3f28530b-4e55-487a-8ed4-540d104609bb" (UID: "3f28530b-4e55-487a-8ed4-540d104609bb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:57:39.756213 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:57:39.756177    2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f28530b-4e55-487a-8ed4-540d104609bb-credential-socket\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:57:39.756362 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:57:39.756207    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f28530b-4e55-487a-8ed4-540d104609bb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3f28530b-4e55-487a-8ed4-540d104609bb" (UID: "3f28530b-4e55-487a-8ed4-540d104609bb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:57:39.756420 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:57:39.756370    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f28530b-4e55-487a-8ed4-540d104609bb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3f28530b-4e55-487a-8ed4-540d104609bb" (UID: "3f28530b-4e55-487a-8ed4-540d104609bb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:57:39.756420 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:57:39.756391    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f28530b-4e55-487a-8ed4-540d104609bb-istio-data" (OuterVolumeSpecName: "istio-data") pod "3f28530b-4e55-487a-8ed4-540d104609bb" (UID: "3f28530b-4e55-487a-8ed4-540d104609bb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:57:39.758041 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:57:39.758005    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3f28530b-4e55-487a-8ed4-540d104609bb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3f28530b-4e55-487a-8ed4-540d104609bb" (UID: "3f28530b-4e55-487a-8ed4-540d104609bb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 15:57:39.758233 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:57:39.758199    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f28530b-4e55-487a-8ed4-540d104609bb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3f28530b-4e55-487a-8ed4-540d104609bb" (UID: "3f28530b-4e55-487a-8ed4-540d104609bb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:57:39.758649 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:57:39.758627    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f28530b-4e55-487a-8ed4-540d104609bb-kube-api-access-fx4dw" (OuterVolumeSpecName: "kube-api-access-fx4dw") pod "3f28530b-4e55-487a-8ed4-540d104609bb" (UID: "3f28530b-4e55-487a-8ed4-540d104609bb"). InnerVolumeSpecName "kube-api-access-fx4dw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:57:39.758717 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:57:39.758650    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f28530b-4e55-487a-8ed4-540d104609bb-istio-token" (OuterVolumeSpecName: "istio-token") pod "3f28530b-4e55-487a-8ed4-540d104609bb" (UID: "3f28530b-4e55-487a-8ed4-540d104609bb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:57:39.856999 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:57:39.856958    2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f28530b-4e55-487a-8ed4-540d104609bb-istio-data\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:57:39.856999 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:57:39.856991    2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f28530b-4e55-487a-8ed4-540d104609bb-istiod-ca-cert\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:57:39.856999 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:57:39.857004    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fx4dw\" (UniqueName: \"kubernetes.io/projected/3f28530b-4e55-487a-8ed4-540d104609bb-kube-api-access-fx4dw\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:57:39.857325 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:57:39.857014    2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f28530b-4e55-487a-8ed4-540d104609bb-workload-certs\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:57:39.857325 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:57:39.857023    2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f28530b-4e55-487a-8ed4-540d104609bb-workload-socket\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:57:39.857325 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:57:39.857031    2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f28530b-4e55-487a-8ed4-540d104609bb-istio-podinfo\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:57:39.857325 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:57:39.857040    2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f28530b-4e55-487a-8ed4-540d104609bb-istio-token\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:57:39.857325 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:57:39.857052    2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f28530b-4e55-487a-8ed4-540d104609bb-istio-envoy\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:57:40.047019 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:57:40.046979    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-6d929c-openshift-default-5cd456f5b6-4j7rh"]
Sep 10 15:57:40.051383 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:57:40.051353    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-6d929c-openshift-default-5cd456f5b6-4j7rh"]
Sep 10 15:57:40.785457 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:57:40.785422    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f28530b-4e55-487a-8ed4-540d104609bb" path="/var/lib/kubelet/pods/3f28530b-4e55-487a-8ed4-540d104609bb/volumes"
Sep 10 15:57:45.807373 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:57:45.807336    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c7257b-78l8l"]
Sep 10 15:57:45.814621 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:57:45.814595    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c7257b-78l8l"]
Sep 10 15:57:46.784630 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:57:46.784573    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b24b80bb-823e-4f47-bf4a-8c0dde5b171e" path="/var/lib/kubelet/pods/b24b80bb-823e-4f47-bf4a-8c0dde5b171e/volumes"
Sep 10 15:57:48.504232 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:57:48.504193    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bs7hd"]
Sep 10 15:58:06.908810 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:58:06.908771    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bs7hd"]
Sep 10 15:58:12.047161 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:58:12.047127    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-3650b0-openshift-default-5b6b46d878-fcvqh"]
Sep 10 15:58:12.047599 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:58:12.047368    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-3650b0-openshift-default-5b6b46d878-fcvqh" podUID="4c8c63e9-de0d-4181-ae91-f9a5ba79c89b" containerName="istio-proxy" containerID="cri-o://a075235a0a77580834e2742720fe67b7c92229511da8b1275ec91fec13e84a50" gracePeriod=30
Sep 10 15:58:13.601397 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:58:13.601352    2559 patch_prober.go:28] interesting pod/e2e-shared-b-w3-3650b0-openshift-default-5b6b46d878-fcvqh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 10 15:58:13.601892 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:58:13.601427    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-3650b0-openshift-default-5b6b46d878-fcvqh" podUID="4c8c63e9-de0d-4181-ae91-f9a5ba79c89b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 10 15:58:17.288468 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:58:17.288444    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-3650b0-openshift-default-5b6b46d878-fcvqh"
Sep 10 15:58:17.383767 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:58:17.383686    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c8c63e9-de0d-4181-ae91-f9a5ba79c89b-credential-socket\") pod \"4c8c63e9-de0d-4181-ae91-f9a5ba79c89b\" (UID: \"4c8c63e9-de0d-4181-ae91-f9a5ba79c89b\") "
Sep 10 15:58:17.383767 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:58:17.383724    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c8c63e9-de0d-4181-ae91-f9a5ba79c89b-istiod-ca-cert\") pod \"4c8c63e9-de0d-4181-ae91-f9a5ba79c89b\" (UID: \"4c8c63e9-de0d-4181-ae91-f9a5ba79c89b\") "
Sep 10 15:58:17.383767 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:58:17.383746    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bh26t\" (UniqueName: \"kubernetes.io/projected/4c8c63e9-de0d-4181-ae91-f9a5ba79c89b-kube-api-access-bh26t\") pod \"4c8c63e9-de0d-4181-ae91-f9a5ba79c89b\" (UID: \"4c8c63e9-de0d-4181-ae91-f9a5ba79c89b\") "
Sep 10 15:58:17.384041 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:58:17.383813    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c8c63e9-de0d-4181-ae91-f9a5ba79c89b-workload-certs\") pod \"4c8c63e9-de0d-4181-ae91-f9a5ba79c89b\" (UID: \"4c8c63e9-de0d-4181-ae91-f9a5ba79c89b\") "
Sep 10 15:58:17.384041 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:58:17.383844    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c8c63e9-de0d-4181-ae91-f9a5ba79c89b-workload-socket\") pod \"4c8c63e9-de0d-4181-ae91-f9a5ba79c89b\" (UID: \"4c8c63e9-de0d-4181-ae91-f9a5ba79c89b\") "
Sep 10 15:58:17.384041 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:58:17.383892    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c8c63e9-de0d-4181-ae91-f9a5ba79c89b-istio-token\") pod \"4c8c63e9-de0d-4181-ae91-f9a5ba79c89b\" (UID: \"4c8c63e9-de0d-4181-ae91-f9a5ba79c89b\") "
Sep 10 15:58:17.384041 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:58:17.383924    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c8c63e9-de0d-4181-ae91-f9a5ba79c89b-istio-envoy\") pod \"4c8c63e9-de0d-4181-ae91-f9a5ba79c89b\" (UID: \"4c8c63e9-de0d-4181-ae91-f9a5ba79c89b\") "
Sep 10 15:58:17.384041 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:58:17.383954    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c8c63e9-de0d-4181-ae91-f9a5ba79c89b-istio-podinfo\") pod \"4c8c63e9-de0d-4181-ae91-f9a5ba79c89b\" (UID: \"4c8c63e9-de0d-4181-ae91-f9a5ba79c89b\") "
Sep 10 15:58:17.384041 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:58:17.384016    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c8c63e9-de0d-4181-ae91-f9a5ba79c89b-istio-data\") pod \"4c8c63e9-de0d-4181-ae91-f9a5ba79c89b\" (UID: \"4c8c63e9-de0d-4181-ae91-f9a5ba79c89b\") "
Sep 10 15:58:17.384041 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:58:17.384012    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c8c63e9-de0d-4181-ae91-f9a5ba79c89b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4c8c63e9-de0d-4181-ae91-f9a5ba79c89b" (UID: "4c8c63e9-de0d-4181-ae91-f9a5ba79c89b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:58:17.384319 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:58:17.384147    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c8c63e9-de0d-4181-ae91-f9a5ba79c89b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4c8c63e9-de0d-4181-ae91-f9a5ba79c89b" (UID: "4c8c63e9-de0d-4181-ae91-f9a5ba79c89b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:58:17.384319 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:58:17.384210    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c8c63e9-de0d-4181-ae91-f9a5ba79c89b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4c8c63e9-de0d-4181-ae91-f9a5ba79c89b" (UID: "4c8c63e9-de0d-4181-ae91-f9a5ba79c89b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:58:17.384319 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:58:17.384280    2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c8c63e9-de0d-4181-ae91-f9a5ba79c89b-credential-socket\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:58:17.384319 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:58:17.384297    2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c8c63e9-de0d-4181-ae91-f9a5ba79c89b-istiod-ca-cert\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:58:17.384540 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:58:17.384366    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c8c63e9-de0d-4181-ae91-f9a5ba79c89b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4c8c63e9-de0d-4181-ae91-f9a5ba79c89b" (UID: "4c8c63e9-de0d-4181-ae91-f9a5ba79c89b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:58:17.384540 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:58:17.384404    2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c8c63e9-de0d-4181-ae91-f9a5ba79c89b-workload-certs\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:58:17.384540 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:58:17.384500    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c8c63e9-de0d-4181-ae91-f9a5ba79c89b-istio-data" (OuterVolumeSpecName: "istio-data") pod "4c8c63e9-de0d-4181-ae91-f9a5ba79c89b" (UID: "4c8c63e9-de0d-4181-ae91-f9a5ba79c89b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:58:17.386233 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:58:17.386208    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c8c63e9-de0d-4181-ae91-f9a5ba79c89b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4c8c63e9-de0d-4181-ae91-f9a5ba79c89b" (UID: "4c8c63e9-de0d-4181-ae91-f9a5ba79c89b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:58:17.386347 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:58:17.386292    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c8c63e9-de0d-4181-ae91-f9a5ba79c89b-istio-token" (OuterVolumeSpecName: "istio-token") pod "4c8c63e9-de0d-4181-ae91-f9a5ba79c89b" (UID: "4c8c63e9-de0d-4181-ae91-f9a5ba79c89b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:58:17.386471 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:58:17.386452    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4c8c63e9-de0d-4181-ae91-f9a5ba79c89b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4c8c63e9-de0d-4181-ae91-f9a5ba79c89b" (UID: "4c8c63e9-de0d-4181-ae91-f9a5ba79c89b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 15:58:17.386518 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:58:17.386463    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c8c63e9-de0d-4181-ae91-f9a5ba79c89b-kube-api-access-bh26t" (OuterVolumeSpecName: "kube-api-access-bh26t") pod "4c8c63e9-de0d-4181-ae91-f9a5ba79c89b" (UID: "4c8c63e9-de0d-4181-ae91-f9a5ba79c89b"). InnerVolumeSpecName "kube-api-access-bh26t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:58:17.424366 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:58:17.424339    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bs7hd"]
Sep 10 15:58:17.484788 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:58:17.484764    2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c8c63e9-de0d-4181-ae91-f9a5ba79c89b-workload-socket\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:58:17.484788 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:58:17.484784    2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c8c63e9-de0d-4181-ae91-f9a5ba79c89b-istio-token\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:58:17.484788 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:58:17.484794    2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c8c63e9-de0d-4181-ae91-f9a5ba79c89b-istio-envoy\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:58:17.484986 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:58:17.484803    2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c8c63e9-de0d-4181-ae91-f9a5ba79c89b-istio-podinfo\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:58:17.484986 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:58:17.484810    2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c8c63e9-de0d-4181-ae91-f9a5ba79c89b-istio-data\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:58:17.484986 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:58:17.484818    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bh26t\" (UniqueName: \"kubernetes.io/projected/4c8c63e9-de0d-4181-ae91-f9a5ba79c89b-kube-api-access-bh26t\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 15:58:17.926590 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:58:17.926542    2559 generic.go:356] "Generic (PLEG): container finished" podID="4c8c63e9-de0d-4181-ae91-f9a5ba79c89b" containerID="a075235a0a77580834e2742720fe67b7c92229511da8b1275ec91fec13e84a50" exitCode=0
Sep 10 15:58:17.926774 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:58:17.926601    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-3650b0-openshift-default-5b6b46d878-fcvqh"
Sep 10 15:58:17.926774 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:58:17.926607    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-3650b0-openshift-default-5b6b46d878-fcvqh" event={"ID":"4c8c63e9-de0d-4181-ae91-f9a5ba79c89b","Type":"ContainerDied","Data":"a075235a0a77580834e2742720fe67b7c92229511da8b1275ec91fec13e84a50"}
Sep 10 15:58:17.926774 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:58:17.926657    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-3650b0-openshift-default-5b6b46d878-fcvqh" event={"ID":"4c8c63e9-de0d-4181-ae91-f9a5ba79c89b","Type":"ContainerDied","Data":"3f9d40d9140213abd4e8bc99a6ba7e9036399af2fc680a9a12a43dc5dfa3b03a"}
Sep 10 15:58:17.926774 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:58:17.926682    2559 scope.go:117] "RemoveContainer" containerID="a075235a0a77580834e2742720fe67b7c92229511da8b1275ec91fec13e84a50"
Sep 10 15:58:17.935893 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:58:17.935877    2559 scope.go:117] "RemoveContainer" containerID="3415437d37340eefa951ab8b9a3f7f8caa543151cfd743b12f834e86a770dddb"
Sep 10 15:58:17.945763 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:58:17.945731    2559 scope.go:117] "RemoveContainer" containerID="a075235a0a77580834e2742720fe67b7c92229511da8b1275ec91fec13e84a50"
Sep 10 15:58:17.946006 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:58:17.945988    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a075235a0a77580834e2742720fe67b7c92229511da8b1275ec91fec13e84a50\": container with ID starting with a075235a0a77580834e2742720fe67b7c92229511da8b1275ec91fec13e84a50 not found: ID does not exist" containerID="a075235a0a77580834e2742720fe67b7c92229511da8b1275ec91fec13e84a50"
Sep 10 15:58:17.946059 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:58:17.946015    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a075235a0a77580834e2742720fe67b7c92229511da8b1275ec91fec13e84a50"} err="failed to get container status \"a075235a0a77580834e2742720fe67b7c92229511da8b1275ec91fec13e84a50\": rpc error: code = NotFound desc = could not find container \"a075235a0a77580834e2742720fe67b7c92229511da8b1275ec91fec13e84a50\": container with ID starting with a075235a0a77580834e2742720fe67b7c92229511da8b1275ec91fec13e84a50 not found: ID does not exist"
Sep 10 15:58:17.946059 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:58:17.946029    2559 scope.go:117] "RemoveContainer" containerID="3415437d37340eefa951ab8b9a3f7f8caa543151cfd743b12f834e86a770dddb"
Sep 10 15:58:17.946281 ip-10-0-130-30 kubenswrapper[2559]: E0910 15:58:17.946263    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3415437d37340eefa951ab8b9a3f7f8caa543151cfd743b12f834e86a770dddb\": container with ID starting with 3415437d37340eefa951ab8b9a3f7f8caa543151cfd743b12f834e86a770dddb not found: ID does not exist" containerID="3415437d37340eefa951ab8b9a3f7f8caa543151cfd743b12f834e86a770dddb"
Sep 10 15:58:17.946343 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:58:17.946290    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3415437d37340eefa951ab8b9a3f7f8caa543151cfd743b12f834e86a770dddb"} err="failed to get container status \"3415437d37340eefa951ab8b9a3f7f8caa543151cfd743b12f834e86a770dddb\": rpc error: code = NotFound desc = could not find container \"3415437d37340eefa951ab8b9a3f7f8caa543151cfd743b12f834e86a770dddb\": container with ID starting with 3415437d37340eefa951ab8b9a3f7f8caa543151cfd743b12f834e86a770dddb not found: ID does not exist"
Sep 10 15:58:17.953424 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:58:17.953401    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-3650b0-openshift-default-5b6b46d878-fcvqh"]
Sep 10 15:58:17.966364 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:58:17.966345    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-3650b0-openshift-default-5b6b46d878-fcvqh"]
Sep 10 15:58:18.786857 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:58:18.786807    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4c8c63e9-de0d-4181-ae91-f9a5ba79c89b" path="/var/lib/kubelet/pods/4c8c63e9-de0d-4181-ae91-f9a5ba79c89b/volumes"
Sep 10 15:59:38.404505 ip-10-0-130-30 kubenswrapper[2559]: I0910 15:59:38.404468    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bs7hd"]
Sep 10 16:00:05.104226 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:00:05.104184    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bs7hd"]
Sep 10 16:00:14.501843 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:00:14.501801    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bs7hd"]
Sep 10 16:00:21.500998 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:00:21.500961    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bs7hd"]
Sep 10 16:00:27.814057 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:00:27.814018    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bs7hd"]
Sep 10 16:00:41.603338 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:00:41.603304    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bs7hd"]
Sep 10 16:00:51.409931 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:00:51.409894    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bs7hd"]
Sep 10 16:01:00.831293 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:01:00.831232    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7c7ft"]
Sep 10 16:01:00.831815 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:01:00.831482    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7c7ft" podUID="8f20ab32-7d9d-4055-a1f0-0e07cf692e5e" containerName="manager" containerID="cri-o://f385fa6a38e5450745bb0c0bb6c698b26ad410b06341075ffb3d7cb13ad6be90" gracePeriod=10
Sep 10 16:01:01.085117 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:01:01.085053    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7c7ft"
Sep 10 16:01:01.143664 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:01:01.143620    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8f20ab32-7d9d-4055-a1f0-0e07cf692e5e-extensions-socket-volume\") pod \"8f20ab32-7d9d-4055-a1f0-0e07cf692e5e\" (UID: \"8f20ab32-7d9d-4055-a1f0-0e07cf692e5e\") "
Sep 10 16:01:01.143838 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:01:01.143692    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rl79x\" (UniqueName: \"kubernetes.io/projected/8f20ab32-7d9d-4055-a1f0-0e07cf692e5e-kube-api-access-rl79x\") pod \"8f20ab32-7d9d-4055-a1f0-0e07cf692e5e\" (UID: \"8f20ab32-7d9d-4055-a1f0-0e07cf692e5e\") "
Sep 10 16:01:01.144065 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:01:01.144039    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f20ab32-7d9d-4055-a1f0-0e07cf692e5e-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "8f20ab32-7d9d-4055-a1f0-0e07cf692e5e" (UID: "8f20ab32-7d9d-4055-a1f0-0e07cf692e5e"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:01:01.145706 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:01:01.145685    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f20ab32-7d9d-4055-a1f0-0e07cf692e5e-kube-api-access-rl79x" (OuterVolumeSpecName: "kube-api-access-rl79x") pod "8f20ab32-7d9d-4055-a1f0-0e07cf692e5e" (UID: "8f20ab32-7d9d-4055-a1f0-0e07cf692e5e"). InnerVolumeSpecName "kube-api-access-rl79x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:01:01.244499 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:01:01.244464    2559 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8f20ab32-7d9d-4055-a1f0-0e07cf692e5e-extensions-socket-volume\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 16:01:01.244499 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:01:01.244492    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rl79x\" (UniqueName: \"kubernetes.io/projected/8f20ab32-7d9d-4055-a1f0-0e07cf692e5e-kube-api-access-rl79x\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 16:01:01.759423 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:01:01.759384    2559 generic.go:356] "Generic (PLEG): container finished" podID="8f20ab32-7d9d-4055-a1f0-0e07cf692e5e" containerID="f385fa6a38e5450745bb0c0bb6c698b26ad410b06341075ffb3d7cb13ad6be90" exitCode=0
Sep 10 16:01:01.759423 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:01:01.759428    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7c7ft"
Sep 10 16:01:01.759711 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:01:01.759470    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7c7ft" event={"ID":"8f20ab32-7d9d-4055-a1f0-0e07cf692e5e","Type":"ContainerDied","Data":"f385fa6a38e5450745bb0c0bb6c698b26ad410b06341075ffb3d7cb13ad6be90"}
Sep 10 16:01:01.759711 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:01:01.759527    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7c7ft" event={"ID":"8f20ab32-7d9d-4055-a1f0-0e07cf692e5e","Type":"ContainerDied","Data":"0c060a35c417c122f8fc22930046959a2daecc256d70532808fa0646130f9693"}
Sep 10 16:01:01.759711 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:01:01.759557    2559 scope.go:117] "RemoveContainer" containerID="f385fa6a38e5450745bb0c0bb6c698b26ad410b06341075ffb3d7cb13ad6be90"
Sep 10 16:01:01.770850 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:01:01.770823    2559 scope.go:117] "RemoveContainer" containerID="f385fa6a38e5450745bb0c0bb6c698b26ad410b06341075ffb3d7cb13ad6be90"
Sep 10 16:01:01.771414 ip-10-0-130-30 kubenswrapper[2559]: E0910 16:01:01.771381    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f385fa6a38e5450745bb0c0bb6c698b26ad410b06341075ffb3d7cb13ad6be90\": container with ID starting with f385fa6a38e5450745bb0c0bb6c698b26ad410b06341075ffb3d7cb13ad6be90 not found: ID does not exist" containerID="f385fa6a38e5450745bb0c0bb6c698b26ad410b06341075ffb3d7cb13ad6be90"
Sep 10 16:01:01.771531 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:01:01.771428    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f385fa6a38e5450745bb0c0bb6c698b26ad410b06341075ffb3d7cb13ad6be90"} err="failed to get container status \"f385fa6a38e5450745bb0c0bb6c698b26ad410b06341075ffb3d7cb13ad6be90\": rpc error: code = NotFound desc = could not find container \"f385fa6a38e5450745bb0c0bb6c698b26ad410b06341075ffb3d7cb13ad6be90\": container with ID starting with f385fa6a38e5450745bb0c0bb6c698b26ad410b06341075ffb3d7cb13ad6be90 not found: ID does not exist"
Sep 10 16:01:01.788795 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:01:01.788758    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7c7ft"]
Sep 10 16:01:01.793544 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:01:01.793511    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7c7ft"]
Sep 10 16:01:02.784538 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:01:02.784505    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f20ab32-7d9d-4055-a1f0-0e07cf692e5e" path="/var/lib/kubelet/pods/8f20ab32-7d9d-4055-a1f0-0e07cf692e5e/volumes"
Sep 10 16:02:01.545364 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:02:01.545329    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-tstxf"]
Sep 10 16:02:01.545838 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:02:01.545737    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f28530b-4e55-487a-8ed4-540d104609bb" containerName="istio-proxy"
Sep 10 16:02:01.545838 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:02:01.545750    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f28530b-4e55-487a-8ed4-540d104609bb" containerName="istio-proxy"
Sep 10 16:02:01.545838 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:02:01.545759    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f28530b-4e55-487a-8ed4-540d104609bb" containerName="istio-proxy"
Sep 10 16:02:01.545838 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:02:01.545765    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f28530b-4e55-487a-8ed4-540d104609bb" containerName="istio-proxy"
Sep 10 16:02:01.545838 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:02:01.545781    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f20ab32-7d9d-4055-a1f0-0e07cf692e5e" containerName="manager"
Sep 10 16:02:01.545838 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:02:01.545787    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f20ab32-7d9d-4055-a1f0-0e07cf692e5e" containerName="manager"
Sep 10 16:02:01.545838 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:02:01.545793    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c8c63e9-de0d-4181-ae91-f9a5ba79c89b" containerName="istio-proxy"
Sep 10 16:02:01.545838 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:02:01.545799    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c8c63e9-de0d-4181-ae91-f9a5ba79c89b" containerName="istio-proxy"
Sep 10 16:02:01.545838 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:02:01.545805    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b24b80bb-823e-4f47-bf4a-8c0dde5b171e" containerName="revoke-keys"
Sep 10 16:02:01.545838 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:02:01.545810    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b24b80bb-823e-4f47-bf4a-8c0dde5b171e" containerName="revoke-keys"
Sep 10 16:02:01.545838 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:02:01.545816    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c8c63e9-de0d-4181-ae91-f9a5ba79c89b" containerName="istio-proxy"
Sep 10 16:02:01.545838 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:02:01.545821    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c8c63e9-de0d-4181-ae91-f9a5ba79c89b" containerName="istio-proxy"
Sep 10 16:02:01.546213 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:02:01.545881    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f20ab32-7d9d-4055-a1f0-0e07cf692e5e" containerName="manager"
Sep 10 16:02:01.546213 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:02:01.545889    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f28530b-4e55-487a-8ed4-540d104609bb" containerName="istio-proxy"
Sep 10 16:02:01.546213 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:02:01.545895    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="b24b80bb-823e-4f47-bf4a-8c0dde5b171e" containerName="revoke-keys"
Sep 10 16:02:01.546213 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:02:01.545901    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c8c63e9-de0d-4181-ae91-f9a5ba79c89b" containerName="istio-proxy"
Sep 10 16:02:01.546213 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:02:01.545906    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c8c63e9-de0d-4181-ae91-f9a5ba79c89b" containerName="istio-proxy"
Sep 10 16:02:01.548866 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:02:01.548848    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-tstxf"
Sep 10 16:02:01.551491 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:02:01.551470    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-rk9mn\""
Sep 10 16:02:01.561694 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:02:01.561666    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-tstxf"]
Sep 10 16:02:01.666895 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:02:01.666845    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzc47\" (UniqueName: \"kubernetes.io/projected/dc07a35b-729a-43c2-88cb-e96578f3af02-kube-api-access-gzc47\") pod \"kuadrant-operator-controller-manager-5769f99d9f-tstxf\" (UID: \"dc07a35b-729a-43c2-88cb-e96578f3af02\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-tstxf"
Sep 10 16:02:01.667094 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:02:01.666911    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dc07a35b-729a-43c2-88cb-e96578f3af02-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-tstxf\" (UID: \"dc07a35b-729a-43c2-88cb-e96578f3af02\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-tstxf"
Sep 10 16:02:01.767334 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:02:01.767294    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzc47\" (UniqueName: \"kubernetes.io/projected/dc07a35b-729a-43c2-88cb-e96578f3af02-kube-api-access-gzc47\") pod \"kuadrant-operator-controller-manager-5769f99d9f-tstxf\" (UID: \"dc07a35b-729a-43c2-88cb-e96578f3af02\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-tstxf"
Sep 10 16:02:01.767521 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:02:01.767360    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dc07a35b-729a-43c2-88cb-e96578f3af02-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-tstxf\" (UID: \"dc07a35b-729a-43c2-88cb-e96578f3af02\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-tstxf"
Sep 10 16:02:01.767818 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:02:01.767799    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dc07a35b-729a-43c2-88cb-e96578f3af02-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-tstxf\" (UID: \"dc07a35b-729a-43c2-88cb-e96578f3af02\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-tstxf"
Sep 10 16:02:01.778525 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:02:01.778500    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzc47\" (UniqueName: \"kubernetes.io/projected/dc07a35b-729a-43c2-88cb-e96578f3af02-kube-api-access-gzc47\") pod \"kuadrant-operator-controller-manager-5769f99d9f-tstxf\" (UID: \"dc07a35b-729a-43c2-88cb-e96578f3af02\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-tstxf"
Sep 10 16:02:01.859396 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:02:01.859303    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-tstxf"
Sep 10 16:02:02.142586 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:02:02.142547    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-tstxf"]
Sep 10 16:02:02.144604 ip-10-0-130-30 kubenswrapper[2559]: W0910 16:02:02.144561    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc07a35b_729a_43c2_88cb_e96578f3af02.slice/crio-e5e0efa7dd7f913af10673625eb1e1b2054498451e94d26da493b4e6e41e1a2d WatchSource:0}: Error finding container e5e0efa7dd7f913af10673625eb1e1b2054498451e94d26da493b4e6e41e1a2d: Status 404 returned error can't find the container with id e5e0efa7dd7f913af10673625eb1e1b2054498451e94d26da493b4e6e41e1a2d
Sep 10 16:02:03.069556 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:02:03.069520    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-tstxf" event={"ID":"dc07a35b-729a-43c2-88cb-e96578f3af02","Type":"ContainerStarted","Data":"ea3e8cd0a7855d2a0a1dc15b68997c2b90b74868e1c5761051914b5e7dcacb5a"}
Sep 10 16:02:03.069974 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:02:03.069601    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-tstxf" event={"ID":"dc07a35b-729a-43c2-88cb-e96578f3af02","Type":"ContainerStarted","Data":"e5e0efa7dd7f913af10673625eb1e1b2054498451e94d26da493b4e6e41e1a2d"}
Sep 10 16:02:03.069974 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:02:03.069629    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-tstxf"
Sep 10 16:02:03.095172 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:02:03.095118    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-tstxf" podStartSLOduration=2.095102937 podStartE2EDuration="2.095102937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:02:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:02:03.09403307 +0000 UTC m=+2098.835998172" watchObservedRunningTime="2026-09-10 16:02:03.095102937 +0000 UTC m=+2098.837068039"
Sep 10 16:02:14.077997 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:02:14.077968    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-tstxf"
Sep 10 16:02:23.004085 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:02:23.004047    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bs7hd"]
Sep 10 16:02:29.521598 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:02:29.521543    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bs7hd"]
Sep 10 16:02:35.500014 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:02:35.499982    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-130b9555-kserve-6d8b778d79-cp84c"]
Sep 10 16:02:35.500512 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:02:35.500401    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f28530b-4e55-487a-8ed4-540d104609bb" containerName="istio-proxy"
Sep 10 16:02:35.503733 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:02:35.503712    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-130b9555-kserve-6d8b778d79-cp84c"
Sep 10 16:02:35.506347 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:02:35.506326    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-130b9555-kserve-self-signed-certs\""
Sep 10 16:02:35.511169 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:02:35.511145    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-130b9555-kserve-6d8b778d79-cp84c"]
Sep 10 16:02:35.664828 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:02:35.664792    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64019fec-ba3c-470b-8b33-afc0d81c0176-dshm\") pod \"e2e-same-id-a-130b9555-kserve-6d8b778d79-cp84c\" (UID: \"64019fec-ba3c-470b-8b33-afc0d81c0176\") " pod="llm/e2e-same-id-a-130b9555-kserve-6d8b778d79-cp84c"
Sep 10 16:02:35.664828 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:02:35.664835    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnpwp\" (UniqueName: \"kubernetes.io/projected/64019fec-ba3c-470b-8b33-afc0d81c0176-kube-api-access-wnpwp\") pod \"e2e-same-id-a-130b9555-kserve-6d8b778d79-cp84c\" (UID: \"64019fec-ba3c-470b-8b33-afc0d81c0176\") " pod="llm/e2e-same-id-a-130b9555-kserve-6d8b778d79-cp84c"
Sep 10 16:02:35.665114 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:02:35.664857    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64019fec-ba3c-470b-8b33-afc0d81c0176-home\") pod \"e2e-same-id-a-130b9555-kserve-6d8b778d79-cp84c\" (UID: \"64019fec-ba3c-470b-8b33-afc0d81c0176\") " pod="llm/e2e-same-id-a-130b9555-kserve-6d8b778d79-cp84c"
Sep 10 16:02:35.665114 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:02:35.664945    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64019fec-ba3c-470b-8b33-afc0d81c0176-tmp-dir\") pod \"e2e-same-id-a-130b9555-kserve-6d8b778d79-cp84c\" (UID: \"64019fec-ba3c-470b-8b33-afc0d81c0176\") " pod="llm/e2e-same-id-a-130b9555-kserve-6d8b778d79-cp84c"
Sep 10 16:02:35.665114 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:02:35.665020    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64019fec-ba3c-470b-8b33-afc0d81c0176-tls-certs\") pod \"e2e-same-id-a-130b9555-kserve-6d8b778d79-cp84c\" (UID: \"64019fec-ba3c-470b-8b33-afc0d81c0176\") " pod="llm/e2e-same-id-a-130b9555-kserve-6d8b778d79-cp84c"
Sep 10 16:02:35.665114 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:02:35.665065    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64019fec-ba3c-470b-8b33-afc0d81c0176-model-cache\") pod \"e2e-same-id-a-130b9555-kserve-6d8b778d79-cp84c\" (UID: \"64019fec-ba3c-470b-8b33-afc0d81c0176\") " pod="llm/e2e-same-id-a-130b9555-kserve-6d8b778d79-cp84c"
Sep 10 16:02:35.766199 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:02:35.766107    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64019fec-ba3c-470b-8b33-afc0d81c0176-dshm\") pod \"e2e-same-id-a-130b9555-kserve-6d8b778d79-cp84c\" (UID: \"64019fec-ba3c-470b-8b33-afc0d81c0176\") " pod="llm/e2e-same-id-a-130b9555-kserve-6d8b778d79-cp84c"
Sep 10 16:02:35.766199 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:02:35.766144    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnpwp\" (UniqueName: \"kubernetes.io/projected/64019fec-ba3c-470b-8b33-afc0d81c0176-kube-api-access-wnpwp\") pod \"e2e-same-id-a-130b9555-kserve-6d8b778d79-cp84c\" (UID: \"64019fec-ba3c-470b-8b33-afc0d81c0176\") " pod="llm/e2e-same-id-a-130b9555-kserve-6d8b778d79-cp84c"
Sep 10 16:02:35.766199 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:02:35.766165    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64019fec-ba3c-470b-8b33-afc0d81c0176-home\") pod \"e2e-same-id-a-130b9555-kserve-6d8b778d79-cp84c\" (UID: \"64019fec-ba3c-470b-8b33-afc0d81c0176\") " pod="llm/e2e-same-id-a-130b9555-kserve-6d8b778d79-cp84c"
Sep 10 16:02:35.766199 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:02:35.766192    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64019fec-ba3c-470b-8b33-afc0d81c0176-tmp-dir\") pod \"e2e-same-id-a-130b9555-kserve-6d8b778d79-cp84c\" (UID: \"64019fec-ba3c-470b-8b33-afc0d81c0176\") " pod="llm/e2e-same-id-a-130b9555-kserve-6d8b778d79-cp84c"
Sep 10 16:02:35.766538 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:02:35.766289    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64019fec-ba3c-470b-8b33-afc0d81c0176-tls-certs\") pod \"e2e-same-id-a-130b9555-kserve-6d8b778d79-cp84c\" (UID: \"64019fec-ba3c-470b-8b33-afc0d81c0176\") " pod="llm/e2e-same-id-a-130b9555-kserve-6d8b778d79-cp84c"
Sep 10 16:02:35.766538 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:02:35.766358    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64019fec-ba3c-470b-8b33-afc0d81c0176-model-cache\") pod \"e2e-same-id-a-130b9555-kserve-6d8b778d79-cp84c\" (UID: \"64019fec-ba3c-470b-8b33-afc0d81c0176\") " pod="llm/e2e-same-id-a-130b9555-kserve-6d8b778d79-cp84c"
Sep 10 16:02:35.766694 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:02:35.766557    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64019fec-ba3c-470b-8b33-afc0d81c0176-home\") pod \"e2e-same-id-a-130b9555-kserve-6d8b778d79-cp84c\" (UID: \"64019fec-ba3c-470b-8b33-afc0d81c0176\") " pod="llm/e2e-same-id-a-130b9555-kserve-6d8b778d79-cp84c"
Sep 10 16:02:35.766694 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:02:35.766664    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64019fec-ba3c-470b-8b33-afc0d81c0176-tmp-dir\") pod \"e2e-same-id-a-130b9555-kserve-6d8b778d79-cp84c\" (UID: \"64019fec-ba3c-470b-8b33-afc0d81c0176\") " pod="llm/e2e-same-id-a-130b9555-kserve-6d8b778d79-cp84c"
Sep 10 16:02:35.766771 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:02:35.766711    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64019fec-ba3c-470b-8b33-afc0d81c0176-model-cache\") pod \"e2e-same-id-a-130b9555-kserve-6d8b778d79-cp84c\" (UID: \"64019fec-ba3c-470b-8b33-afc0d81c0176\") " pod="llm/e2e-same-id-a-130b9555-kserve-6d8b778d79-cp84c"
Sep 10 16:02:35.768498 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:02:35.768477    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64019fec-ba3c-470b-8b33-afc0d81c0176-dshm\") pod \"e2e-same-id-a-130b9555-kserve-6d8b778d79-cp84c\" (UID: \"64019fec-ba3c-470b-8b33-afc0d81c0176\") " pod="llm/e2e-same-id-a-130b9555-kserve-6d8b778d79-cp84c"
Sep 10 16:02:35.768694 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:02:35.768675    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64019fec-ba3c-470b-8b33-afc0d81c0176-tls-certs\") pod \"e2e-same-id-a-130b9555-kserve-6d8b778d79-cp84c\" (UID: \"64019fec-ba3c-470b-8b33-afc0d81c0176\") " pod="llm/e2e-same-id-a-130b9555-kserve-6d8b778d79-cp84c"
Sep 10 16:02:35.778379 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:02:35.778355    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnpwp\" (UniqueName: \"kubernetes.io/projected/64019fec-ba3c-470b-8b33-afc0d81c0176-kube-api-access-wnpwp\") pod \"e2e-same-id-a-130b9555-kserve-6d8b778d79-cp84c\" (UID: \"64019fec-ba3c-470b-8b33-afc0d81c0176\") " pod="llm/e2e-same-id-a-130b9555-kserve-6d8b778d79-cp84c"
Sep 10 16:02:35.814740 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:02:35.814710    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-130b9555-kserve-6d8b778d79-cp84c"
Sep 10 16:02:36.088101 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:02:36.088073    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-130b9555-kserve-6d8b778d79-cp84c"]
Sep 10 16:02:36.089770 ip-10-0-130-30 kubenswrapper[2559]: W0910 16:02:36.089736    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod64019fec_ba3c_470b_8b33_afc0d81c0176.slice/crio-a4caa482a2071ac3fd1c521f34ebb0ddf41efb5d8532cb63073414b0227a3b9e WatchSource:0}: Error finding container a4caa482a2071ac3fd1c521f34ebb0ddf41efb5d8532cb63073414b0227a3b9e: Status 404 returned error can't find the container with id a4caa482a2071ac3fd1c521f34ebb0ddf41efb5d8532cb63073414b0227a3b9e
Sep 10 16:02:36.251175 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:02:36.251139    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-130b9555-kserve-6d8b778d79-cp84c" event={"ID":"64019fec-ba3c-470b-8b33-afc0d81c0176","Type":"ContainerStarted","Data":"8d45a95ea1a78a937d154d99cdc943e5a03f31bcc0083695535b59be827d1932"}
Sep 10 16:02:36.251359 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:02:36.251188    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-130b9555-kserve-6d8b778d79-cp84c" event={"ID":"64019fec-ba3c-470b-8b33-afc0d81c0176","Type":"ContainerStarted","Data":"a4caa482a2071ac3fd1c521f34ebb0ddf41efb5d8532cb63073414b0227a3b9e"}
Sep 10 16:02:36.271750 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:02:36.271700    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-130b9555-kserve-6d8b778d79-cp84c" podStartSLOduration=1.271683868 podStartE2EDuration="1.271683868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:02:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:02:36.26930731 +0000 UTC m=+2132.011272412" watchObservedRunningTime="2026-09-10 16:02:36.271683868 +0000 UTC m=+2132.013648971"
Sep 10 16:02:39.007018 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:02:39.006980    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bs7hd"]
Sep 10 16:02:45.815120 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:02:45.815082    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-130b9555-kserve-6d8b778d79-cp84c"
Sep 10 16:02:45.815567 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:02:45.815157    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-130b9555-kserve-6d8b778d79-cp84c"
Sep 10 16:02:45.827756 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:02:45.827731    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-130b9555-kserve-6d8b778d79-cp84c"
Sep 10 16:02:46.319844 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:02:46.319817    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-130b9555-kserve-6d8b778d79-cp84c"
Sep 10 16:02:49.608151 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:02:49.608106    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bs7hd"]
Sep 10 16:03:26.672665 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:03:26.672628    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-130b9555-kserve-6d8b778d79-cp84c"]
Sep 10 16:03:29.061423 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:03:29.061385    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bs7hd"]
Sep 10 16:03:35.105163 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:03:35.105124    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bs7hd"]
Sep 10 16:03:40.999316 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:03:40.999279    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bs7hd"]
Sep 10 16:03:41.692511 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:03:41.692452    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-130b9555-kserve-6d8b778d79-cp84c" podUID="64019fec-ba3c-470b-8b33-afc0d81c0176" containerName="main" containerID="cri-o://8d45a95ea1a78a937d154d99cdc943e5a03f31bcc0083695535b59be827d1932" gracePeriod=45
Sep 10 16:03:41.949027 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:03:41.948952    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-130b9555-kserve-6d8b778d79-cp84c"
Sep 10 16:03:42.056244 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:03:42.056206    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wnpwp\" (UniqueName: \"kubernetes.io/projected/64019fec-ba3c-470b-8b33-afc0d81c0176-kube-api-access-wnpwp\") pod \"64019fec-ba3c-470b-8b33-afc0d81c0176\" (UID: \"64019fec-ba3c-470b-8b33-afc0d81c0176\") "
Sep 10 16:03:42.056764 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:03:42.056251    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64019fec-ba3c-470b-8b33-afc0d81c0176-tls-certs\") pod \"64019fec-ba3c-470b-8b33-afc0d81c0176\" (UID: \"64019fec-ba3c-470b-8b33-afc0d81c0176\") "
Sep 10 16:03:42.056764 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:03:42.056304    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64019fec-ba3c-470b-8b33-afc0d81c0176-dshm\") pod \"64019fec-ba3c-470b-8b33-afc0d81c0176\" (UID: \"64019fec-ba3c-470b-8b33-afc0d81c0176\") "
Sep 10 16:03:42.056764 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:03:42.056365    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64019fec-ba3c-470b-8b33-afc0d81c0176-home\") pod \"64019fec-ba3c-470b-8b33-afc0d81c0176\" (UID: \"64019fec-ba3c-470b-8b33-afc0d81c0176\") "
Sep 10 16:03:42.056764 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:03:42.056400    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64019fec-ba3c-470b-8b33-afc0d81c0176-model-cache\") pod \"64019fec-ba3c-470b-8b33-afc0d81c0176\" (UID: \"64019fec-ba3c-470b-8b33-afc0d81c0176\") "
Sep 10 16:03:42.056764 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:03:42.056471    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64019fec-ba3c-470b-8b33-afc0d81c0176-tmp-dir\") pod \"64019fec-ba3c-470b-8b33-afc0d81c0176\" (UID: \"64019fec-ba3c-470b-8b33-afc0d81c0176\") "
Sep 10 16:03:42.056764 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:03:42.056720    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64019fec-ba3c-470b-8b33-afc0d81c0176-home" (OuterVolumeSpecName: "home") pod "64019fec-ba3c-470b-8b33-afc0d81c0176" (UID: "64019fec-ba3c-470b-8b33-afc0d81c0176"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:03:42.057068 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:03:42.056779    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64019fec-ba3c-470b-8b33-afc0d81c0176-model-cache" (OuterVolumeSpecName: "model-cache") pod "64019fec-ba3c-470b-8b33-afc0d81c0176" (UID: "64019fec-ba3c-470b-8b33-afc0d81c0176"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:03:42.057068 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:03:42.056928    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64019fec-ba3c-470b-8b33-afc0d81c0176-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "64019fec-ba3c-470b-8b33-afc0d81c0176" (UID: "64019fec-ba3c-470b-8b33-afc0d81c0176"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:03:42.058445 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:03:42.058410    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64019fec-ba3c-470b-8b33-afc0d81c0176-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "64019fec-ba3c-470b-8b33-afc0d81c0176" (UID: "64019fec-ba3c-470b-8b33-afc0d81c0176"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 16:03:42.058915 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:03:42.058896    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64019fec-ba3c-470b-8b33-afc0d81c0176-kube-api-access-wnpwp" (OuterVolumeSpecName: "kube-api-access-wnpwp") pod "64019fec-ba3c-470b-8b33-afc0d81c0176" (UID: "64019fec-ba3c-470b-8b33-afc0d81c0176"). InnerVolumeSpecName "kube-api-access-wnpwp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:03:42.059013 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:03:42.058898    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64019fec-ba3c-470b-8b33-afc0d81c0176-dshm" (OuterVolumeSpecName: "dshm") pod "64019fec-ba3c-470b-8b33-afc0d81c0176" (UID: "64019fec-ba3c-470b-8b33-afc0d81c0176"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:03:42.157712 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:03:42.157678    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64019fec-ba3c-470b-8b33-afc0d81c0176-home\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 16:03:42.157712 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:03:42.157706    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64019fec-ba3c-470b-8b33-afc0d81c0176-model-cache\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 16:03:42.157712 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:03:42.157717    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64019fec-ba3c-470b-8b33-afc0d81c0176-tmp-dir\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 16:03:42.157954 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:03:42.157726    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnpwp\" (UniqueName: \"kubernetes.io/projected/64019fec-ba3c-470b-8b33-afc0d81c0176-kube-api-access-wnpwp\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 16:03:42.157954 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:03:42.157735    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64019fec-ba3c-470b-8b33-afc0d81c0176-tls-certs\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 16:03:42.157954 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:03:42.157746    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64019fec-ba3c-470b-8b33-afc0d81c0176-dshm\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 16:03:42.604906 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:03:42.604875    2559 generic.go:356] "Generic (PLEG): container finished" podID="64019fec-ba3c-470b-8b33-afc0d81c0176" containerID="8d45a95ea1a78a937d154d99cdc943e5a03f31bcc0083695535b59be827d1932" exitCode=0
Sep 10 16:03:42.605104 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:03:42.604921    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-130b9555-kserve-6d8b778d79-cp84c"
Sep 10 16:03:42.605104 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:03:42.604959    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-130b9555-kserve-6d8b778d79-cp84c" event={"ID":"64019fec-ba3c-470b-8b33-afc0d81c0176","Type":"ContainerDied","Data":"8d45a95ea1a78a937d154d99cdc943e5a03f31bcc0083695535b59be827d1932"}
Sep 10 16:03:42.605104 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:03:42.605007    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-130b9555-kserve-6d8b778d79-cp84c" event={"ID":"64019fec-ba3c-470b-8b33-afc0d81c0176","Type":"ContainerDied","Data":"a4caa482a2071ac3fd1c521f34ebb0ddf41efb5d8532cb63073414b0227a3b9e"}
Sep 10 16:03:42.605104 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:03:42.605022    2559 scope.go:117] "RemoveContainer" containerID="8d45a95ea1a78a937d154d99cdc943e5a03f31bcc0083695535b59be827d1932"
Sep 10 16:03:42.614644 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:03:42.614615    2559 scope.go:117] "RemoveContainer" containerID="8d45a95ea1a78a937d154d99cdc943e5a03f31bcc0083695535b59be827d1932"
Sep 10 16:03:42.614924 ip-10-0-130-30 kubenswrapper[2559]: E0910 16:03:42.614904    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d45a95ea1a78a937d154d99cdc943e5a03f31bcc0083695535b59be827d1932\": container with ID starting with 8d45a95ea1a78a937d154d99cdc943e5a03f31bcc0083695535b59be827d1932 not found: ID does not exist" containerID="8d45a95ea1a78a937d154d99cdc943e5a03f31bcc0083695535b59be827d1932"
Sep 10 16:03:42.614980 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:03:42.614934    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d45a95ea1a78a937d154d99cdc943e5a03f31bcc0083695535b59be827d1932"} err="failed to get container status \"8d45a95ea1a78a937d154d99cdc943e5a03f31bcc0083695535b59be827d1932\": rpc error: code = NotFound desc = could not find container \"8d45a95ea1a78a937d154d99cdc943e5a03f31bcc0083695535b59be827d1932\": container with ID starting with 8d45a95ea1a78a937d154d99cdc943e5a03f31bcc0083695535b59be827d1932 not found: ID does not exist"
Sep 10 16:03:42.634891 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:03:42.634861    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-130b9555-kserve-6d8b778d79-cp84c"]
Sep 10 16:03:42.640547 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:03:42.640519    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-130b9555-kserve-6d8b778d79-cp84c"]
Sep 10 16:03:42.784709 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:03:42.784673    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64019fec-ba3c-470b-8b33-afc0d81c0176" path="/var/lib/kubelet/pods/64019fec-ba3c-470b-8b33-afc0d81c0176/volumes"
Sep 10 16:03:47.649499 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:03:47.649454    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bs7hd"]
Sep 10 16:03:47.934743 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:03:47.934706    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-66f7f9db8f-5f6qx"]
Sep 10 16:03:47.935289 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:03:47.935260    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64019fec-ba3c-470b-8b33-afc0d81c0176" containerName="main"
Sep 10 16:03:47.935289 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:03:47.935285    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="64019fec-ba3c-470b-8b33-afc0d81c0176" containerName="main"
Sep 10 16:03:47.935500 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:03:47.935376    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="64019fec-ba3c-470b-8b33-afc0d81c0176" containerName="main"
Sep 10 16:03:47.939270 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:03:47.939252    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-66f7f9db8f-5f6qx"
Sep 10 16:03:47.943071 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:03:47.943046    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 10 16:03:47.943192 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:03:47.943093    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 10 16:03:47.943192 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:03:47.943127    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 16:03:47.943192 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:03:47.943169    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-5qzmv\""
Sep 10 16:03:47.943419 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:03:47.943402    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 10 16:03:47.950776 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:03:47.950725    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-66f7f9db8f-5f6qx"]
Sep 10 16:03:48.012185 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:03:48.012149    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgtmq\" (UniqueName: \"kubernetes.io/projected/b539f448-ba5c-4073-ab4c-d92c206cea3c-kube-api-access-jgtmq\") pod \"maas-api-66f7f9db8f-5f6qx\" (UID: \"b539f448-ba5c-4073-ab4c-d92c206cea3c\") " pod="odh-ai-gateway-infra/maas-api-66f7f9db8f-5f6qx"
Sep 10 16:03:48.012363 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:03:48.012195    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b539f448-ba5c-4073-ab4c-d92c206cea3c-openshift-service-ca\") pod \"maas-api-66f7f9db8f-5f6qx\" (UID: \"b539f448-ba5c-4073-ab4c-d92c206cea3c\") " pod="odh-ai-gateway-infra/maas-api-66f7f9db8f-5f6qx"
Sep 10 16:03:48.012363 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:03:48.012288    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b539f448-ba5c-4073-ab4c-d92c206cea3c-maas-api-tls\") pod \"maas-api-66f7f9db8f-5f6qx\" (UID: \"b539f448-ba5c-4073-ab4c-d92c206cea3c\") " pod="odh-ai-gateway-infra/maas-api-66f7f9db8f-5f6qx"
Sep 10 16:03:48.012363 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:03:48.012312    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b539f448-ba5c-4073-ab4c-d92c206cea3c-metrics-certs\") pod \"maas-api-66f7f9db8f-5f6qx\" (UID: \"b539f448-ba5c-4073-ab4c-d92c206cea3c\") " pod="odh-ai-gateway-infra/maas-api-66f7f9db8f-5f6qx"
Sep 10 16:03:48.112929 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:03:48.112892    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b539f448-ba5c-4073-ab4c-d92c206cea3c-maas-api-tls\") pod \"maas-api-66f7f9db8f-5f6qx\" (UID: \"b539f448-ba5c-4073-ab4c-d92c206cea3c\") " pod="odh-ai-gateway-infra/maas-api-66f7f9db8f-5f6qx"
Sep 10 16:03:48.112929 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:03:48.112928    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b539f448-ba5c-4073-ab4c-d92c206cea3c-metrics-certs\") pod \"maas-api-66f7f9db8f-5f6qx\" (UID: \"b539f448-ba5c-4073-ab4c-d92c206cea3c\") " pod="odh-ai-gateway-infra/maas-api-66f7f9db8f-5f6qx"
Sep 10 16:03:48.113186 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:03:48.112985    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgtmq\" (UniqueName: \"kubernetes.io/projected/b539f448-ba5c-4073-ab4c-d92c206cea3c-kube-api-access-jgtmq\") pod \"maas-api-66f7f9db8f-5f6qx\" (UID: \"b539f448-ba5c-4073-ab4c-d92c206cea3c\") " pod="odh-ai-gateway-infra/maas-api-66f7f9db8f-5f6qx"
Sep 10 16:03:48.113186 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:03:48.113010    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b539f448-ba5c-4073-ab4c-d92c206cea3c-openshift-service-ca\") pod \"maas-api-66f7f9db8f-5f6qx\" (UID: \"b539f448-ba5c-4073-ab4c-d92c206cea3c\") " pod="odh-ai-gateway-infra/maas-api-66f7f9db8f-5f6qx"
Sep 10 16:03:48.113686 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:03:48.113660    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b539f448-ba5c-4073-ab4c-d92c206cea3c-openshift-service-ca\") pod \"maas-api-66f7f9db8f-5f6qx\" (UID: \"b539f448-ba5c-4073-ab4c-d92c206cea3c\") " pod="odh-ai-gateway-infra/maas-api-66f7f9db8f-5f6qx"
Sep 10 16:03:48.115422 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:03:48.115394    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b539f448-ba5c-4073-ab4c-d92c206cea3c-metrics-certs\") pod \"maas-api-66f7f9db8f-5f6qx\" (UID: \"b539f448-ba5c-4073-ab4c-d92c206cea3c\") " pod="odh-ai-gateway-infra/maas-api-66f7f9db8f-5f6qx"
Sep 10 16:03:48.115422 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:03:48.115415    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b539f448-ba5c-4073-ab4c-d92c206cea3c-maas-api-tls\") pod \"maas-api-66f7f9db8f-5f6qx\" (UID: \"b539f448-ba5c-4073-ab4c-d92c206cea3c\") " pod="odh-ai-gateway-infra/maas-api-66f7f9db8f-5f6qx"
Sep 10 16:03:48.123568 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:03:48.123540    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgtmq\" (UniqueName: \"kubernetes.io/projected/b539f448-ba5c-4073-ab4c-d92c206cea3c-kube-api-access-jgtmq\") pod \"maas-api-66f7f9db8f-5f6qx\" (UID: \"b539f448-ba5c-4073-ab4c-d92c206cea3c\") " pod="odh-ai-gateway-infra/maas-api-66f7f9db8f-5f6qx"
Sep 10 16:03:48.252523 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:03:48.252430    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-66f7f9db8f-5f6qx"
Sep 10 16:03:48.533664 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:03:48.533640    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-66f7f9db8f-5f6qx"]
Sep 10 16:03:48.535418 ip-10-0-130-30 kubenswrapper[2559]: W0910 16:03:48.535391    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb539f448_ba5c_4073_ab4c_d92c206cea3c.slice/crio-6cf15867378a0ed46ee8cf8fbd6c018547e5d2acd4e4aa055b452704453b9733 WatchSource:0}: Error finding container 6cf15867378a0ed46ee8cf8fbd6c018547e5d2acd4e4aa055b452704453b9733: Status 404 returned error can't find the container with id 6cf15867378a0ed46ee8cf8fbd6c018547e5d2acd4e4aa055b452704453b9733
Sep 10 16:03:48.536708 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:03:48.536691    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 16:03:48.640018 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:03:48.639981    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-66f7f9db8f-5f6qx" event={"ID":"b539f448-ba5c-4073-ab4c-d92c206cea3c","Type":"ContainerStarted","Data":"6cf15867378a0ed46ee8cf8fbd6c018547e5d2acd4e4aa055b452704453b9733"}
Sep 10 16:03:49.646642 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:03:49.646604    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-66f7f9db8f-5f6qx" event={"ID":"b539f448-ba5c-4073-ab4c-d92c206cea3c","Type":"ContainerStarted","Data":"821d4f31a55d883a02463e0b146241e5c00844e359cc527c6f9e453d3763d9dc"}
Sep 10 16:03:49.647063 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:03:49.646712    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-66f7f9db8f-5f6qx"
Sep 10 16:03:49.667600 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:03:49.667540    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-66f7f9db8f-5f6qx" podStartSLOduration=1.6392537489999999 podStartE2EDuration="2.667525483s" totalImagesPullingTime="1.028271734s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:03:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:03:49.666657881 +0000 UTC m=+2205.408622987" watchObservedRunningTime="2026-09-10 16:03:49.667525483 +0000 UTC m=+2205.409490617"
Sep 10 16:03:55.606459 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:03:55.606411    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bs7hd"]
Sep 10 16:03:55.658198 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:03:55.658167    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-66f7f9db8f-5f6qx"
Sep 10 16:03:56.309477 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:03:56.309444    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5df4475445-j29wd"]
Sep 10 16:03:56.312924 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:03:56.312908    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5df4475445-j29wd"
Sep 10 16:03:56.329256 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:03:56.329230    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5df4475445-j29wd"]
Sep 10 16:03:56.384962 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:03:56.384924    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7eb435b8-5bd8-4062-a79a-bab6490adb71-metrics-certs\") pod \"maas-api-5df4475445-j29wd\" (UID: \"7eb435b8-5bd8-4062-a79a-bab6490adb71\") " pod="odh-ai-gateway-infra/maas-api-5df4475445-j29wd"
Sep 10 16:03:56.384962 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:03:56.384963    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7eb435b8-5bd8-4062-a79a-bab6490adb71-maas-api-tls\") pod \"maas-api-5df4475445-j29wd\" (UID: \"7eb435b8-5bd8-4062-a79a-bab6490adb71\") " pod="odh-ai-gateway-infra/maas-api-5df4475445-j29wd"
Sep 10 16:03:56.385182 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:03:56.385131    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjpr7\" (UniqueName: \"kubernetes.io/projected/7eb435b8-5bd8-4062-a79a-bab6490adb71-kube-api-access-tjpr7\") pod \"maas-api-5df4475445-j29wd\" (UID: \"7eb435b8-5bd8-4062-a79a-bab6490adb71\") " pod="odh-ai-gateway-infra/maas-api-5df4475445-j29wd"
Sep 10 16:03:56.385182 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:03:56.385171    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7eb435b8-5bd8-4062-a79a-bab6490adb71-openshift-service-ca\") pod \"maas-api-5df4475445-j29wd\" (UID: \"7eb435b8-5bd8-4062-a79a-bab6490adb71\") " pod="odh-ai-gateway-infra/maas-api-5df4475445-j29wd"
Sep 10 16:03:56.486275 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:03:56.486243    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjpr7\" (UniqueName: \"kubernetes.io/projected/7eb435b8-5bd8-4062-a79a-bab6490adb71-kube-api-access-tjpr7\") pod \"maas-api-5df4475445-j29wd\" (UID: \"7eb435b8-5bd8-4062-a79a-bab6490adb71\") " pod="odh-ai-gateway-infra/maas-api-5df4475445-j29wd"
Sep 10 16:03:56.486482 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:03:56.486282    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7eb435b8-5bd8-4062-a79a-bab6490adb71-openshift-service-ca\") pod \"maas-api-5df4475445-j29wd\" (UID: \"7eb435b8-5bd8-4062-a79a-bab6490adb71\") " pod="odh-ai-gateway-infra/maas-api-5df4475445-j29wd"
Sep 10 16:03:56.486482 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:03:56.486305    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7eb435b8-5bd8-4062-a79a-bab6490adb71-metrics-certs\") pod \"maas-api-5df4475445-j29wd\" (UID: \"7eb435b8-5bd8-4062-a79a-bab6490adb71\") " pod="odh-ai-gateway-infra/maas-api-5df4475445-j29wd"
Sep 10 16:03:56.486482 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:03:56.486324    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7eb435b8-5bd8-4062-a79a-bab6490adb71-maas-api-tls\") pod \"maas-api-5df4475445-j29wd\" (UID: \"7eb435b8-5bd8-4062-a79a-bab6490adb71\") " pod="odh-ai-gateway-infra/maas-api-5df4475445-j29wd"
Sep 10 16:03:56.487083 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:03:56.487055    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7eb435b8-5bd8-4062-a79a-bab6490adb71-openshift-service-ca\") pod \"maas-api-5df4475445-j29wd\" (UID: \"7eb435b8-5bd8-4062-a79a-bab6490adb71\") " pod="odh-ai-gateway-infra/maas-api-5df4475445-j29wd"
Sep 10 16:03:56.488918 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:03:56.488899    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7eb435b8-5bd8-4062-a79a-bab6490adb71-maas-api-tls\") pod \"maas-api-5df4475445-j29wd\" (UID: \"7eb435b8-5bd8-4062-a79a-bab6490adb71\") " pod="odh-ai-gateway-infra/maas-api-5df4475445-j29wd"
Sep 10 16:03:56.489009 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:03:56.488966    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7eb435b8-5bd8-4062-a79a-bab6490adb71-metrics-certs\") pod \"maas-api-5df4475445-j29wd\" (UID: \"7eb435b8-5bd8-4062-a79a-bab6490adb71\") " pod="odh-ai-gateway-infra/maas-api-5df4475445-j29wd"
Sep 10 16:03:56.495934 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:03:56.495909    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjpr7\" (UniqueName: \"kubernetes.io/projected/7eb435b8-5bd8-4062-a79a-bab6490adb71-kube-api-access-tjpr7\") pod \"maas-api-5df4475445-j29wd\" (UID: \"7eb435b8-5bd8-4062-a79a-bab6490adb71\") " pod="odh-ai-gateway-infra/maas-api-5df4475445-j29wd"
Sep 10 16:03:56.622466 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:03:56.622377    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5df4475445-j29wd"
Sep 10 16:03:56.897916 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:03:56.897887    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5df4475445-j29wd"]
Sep 10 16:03:56.900037 ip-10-0-130-30 kubenswrapper[2559]: W0910 16:03:56.900014    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7eb435b8_5bd8_4062_a79a_bab6490adb71.slice/crio-566cdca98d33418842344bb1cc3b155336b90daf3103dcfd22c23323fb58299c WatchSource:0}: Error finding container 566cdca98d33418842344bb1cc3b155336b90daf3103dcfd22c23323fb58299c: Status 404 returned error can't find the container with id 566cdca98d33418842344bb1cc3b155336b90daf3103dcfd22c23323fb58299c
Sep 10 16:03:57.695407 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:03:57.695371    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5df4475445-j29wd" event={"ID":"7eb435b8-5bd8-4062-a79a-bab6490adb71","Type":"ContainerStarted","Data":"fc1a01f1c5dc8477469666357d003b6fc4093b7c765317c51c39f689e1f12b8c"}
Sep 10 16:03:57.695929 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:03:57.695428    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5df4475445-j29wd" event={"ID":"7eb435b8-5bd8-4062-a79a-bab6490adb71","Type":"ContainerStarted","Data":"566cdca98d33418842344bb1cc3b155336b90daf3103dcfd22c23323fb58299c"}
Sep 10 16:03:57.695929 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:03:57.695503    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5df4475445-j29wd"
Sep 10 16:03:57.715362 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:03:57.715314    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5df4475445-j29wd" podStartSLOduration=1.364388419 podStartE2EDuration="1.715298672s" totalImagesPullingTime="350.910253ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:03:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:03:57.714155783 +0000 UTC m=+2213.456120887" watchObservedRunningTime="2026-09-10 16:03:57.715298672 +0000 UTC m=+2213.457263775"
Sep 10 16:04:02.412700 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:04:02.412663    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bs7hd"]
Sep 10 16:04:03.707156 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:04:03.707126    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5df4475445-j29wd"
Sep 10 16:04:03.763312 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:04:03.763275    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-66f7f9db8f-5f6qx"]
Sep 10 16:04:03.763563 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:04:03.763518    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-66f7f9db8f-5f6qx" podUID="b539f448-ba5c-4073-ab4c-d92c206cea3c" containerName="maas-api" containerID="cri-o://821d4f31a55d883a02463e0b146241e5c00844e359cc527c6f9e453d3763d9dc" gracePeriod=30
Sep 10 16:04:05.110947 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:04:05.110924    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-66f7f9db8f-5f6qx"
Sep 10 16:04:05.151086 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:04:05.150994    2559 scope.go:117] "RemoveContainer" containerID="76e2ba23d0846142a8d250ab1c2fcf5fb757ef2ad8c1bceeadb5c99d7a4d286d"
Sep 10 16:04:05.262397 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:04:05.262358    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b539f448-ba5c-4073-ab4c-d92c206cea3c-metrics-certs\") pod \"b539f448-ba5c-4073-ab4c-d92c206cea3c\" (UID: \"b539f448-ba5c-4073-ab4c-d92c206cea3c\") "
Sep 10 16:04:05.262630 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:04:05.262424    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b539f448-ba5c-4073-ab4c-d92c206cea3c-maas-api-tls\") pod \"b539f448-ba5c-4073-ab4c-d92c206cea3c\" (UID: \"b539f448-ba5c-4073-ab4c-d92c206cea3c\") "
Sep 10 16:04:05.262630 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:04:05.262521    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b539f448-ba5c-4073-ab4c-d92c206cea3c-openshift-service-ca\") pod \"b539f448-ba5c-4073-ab4c-d92c206cea3c\" (UID: \"b539f448-ba5c-4073-ab4c-d92c206cea3c\") "
Sep 10 16:04:05.262630 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:04:05.262557    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jgtmq\" (UniqueName: \"kubernetes.io/projected/b539f448-ba5c-4073-ab4c-d92c206cea3c-kube-api-access-jgtmq\") pod \"b539f448-ba5c-4073-ab4c-d92c206cea3c\" (UID: \"b539f448-ba5c-4073-ab4c-d92c206cea3c\") "
Sep 10 16:04:05.262915 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:04:05.262892    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b539f448-ba5c-4073-ab4c-d92c206cea3c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b539f448-ba5c-4073-ab4c-d92c206cea3c" (UID: "b539f448-ba5c-4073-ab4c-d92c206cea3c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 16:04:05.264681 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:04:05.264644    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b539f448-ba5c-4073-ab4c-d92c206cea3c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b539f448-ba5c-4073-ab4c-d92c206cea3c" (UID: "b539f448-ba5c-4073-ab4c-d92c206cea3c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 16:04:05.264795 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:04:05.264689    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b539f448-ba5c-4073-ab4c-d92c206cea3c-kube-api-access-jgtmq" (OuterVolumeSpecName: "kube-api-access-jgtmq") pod "b539f448-ba5c-4073-ab4c-d92c206cea3c" (UID: "b539f448-ba5c-4073-ab4c-d92c206cea3c"). InnerVolumeSpecName "kube-api-access-jgtmq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:04:05.264795 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:04:05.264753    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b539f448-ba5c-4073-ab4c-d92c206cea3c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b539f448-ba5c-4073-ab4c-d92c206cea3c" (UID: "b539f448-ba5c-4073-ab4c-d92c206cea3c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 16:04:05.364278 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:04:05.364236    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b539f448-ba5c-4073-ab4c-d92c206cea3c-maas-api-tls\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 16:04:05.364278 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:04:05.364272    2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b539f448-ba5c-4073-ab4c-d92c206cea3c-openshift-service-ca\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 16:04:05.364278 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:04:05.364285    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgtmq\" (UniqueName: \"kubernetes.io/projected/b539f448-ba5c-4073-ab4c-d92c206cea3c-kube-api-access-jgtmq\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 16:04:05.364525 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:04:05.364297    2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b539f448-ba5c-4073-ab4c-d92c206cea3c-metrics-certs\") on node \"ip-10-0-130-30.ec2.internal\" DevicePath \"\""
Sep 10 16:04:05.741115 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:04:05.741076    2559 generic.go:356] "Generic (PLEG): container finished" podID="b539f448-ba5c-4073-ab4c-d92c206cea3c" containerID="821d4f31a55d883a02463e0b146241e5c00844e359cc527c6f9e453d3763d9dc" exitCode=0
Sep 10 16:04:05.741304 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:04:05.741136    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-66f7f9db8f-5f6qx" event={"ID":"b539f448-ba5c-4073-ab4c-d92c206cea3c","Type":"ContainerDied","Data":"821d4f31a55d883a02463e0b146241e5c00844e359cc527c6f9e453d3763d9dc"}
Sep 10 16:04:05.741304 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:04:05.741156    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-66f7f9db8f-5f6qx"
Sep 10 16:04:05.741304 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:04:05.741222    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-66f7f9db8f-5f6qx" event={"ID":"b539f448-ba5c-4073-ab4c-d92c206cea3c","Type":"ContainerDied","Data":"6cf15867378a0ed46ee8cf8fbd6c018547e5d2acd4e4aa055b452704453b9733"}
Sep 10 16:04:05.741304 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:04:05.741254    2559 scope.go:117] "RemoveContainer" containerID="821d4f31a55d883a02463e0b146241e5c00844e359cc527c6f9e453d3763d9dc"
Sep 10 16:04:05.750741 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:04:05.750721    2559 scope.go:117] "RemoveContainer" containerID="821d4f31a55d883a02463e0b146241e5c00844e359cc527c6f9e453d3763d9dc"
Sep 10 16:04:05.751075 ip-10-0-130-30 kubenswrapper[2559]: E0910 16:04:05.751045    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"821d4f31a55d883a02463e0b146241e5c00844e359cc527c6f9e453d3763d9dc\": container with ID starting with 821d4f31a55d883a02463e0b146241e5c00844e359cc527c6f9e453d3763d9dc not found: ID does not exist" containerID="821d4f31a55d883a02463e0b146241e5c00844e359cc527c6f9e453d3763d9dc"
Sep 10 16:04:05.751188 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:04:05.751081    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"821d4f31a55d883a02463e0b146241e5c00844e359cc527c6f9e453d3763d9dc"} err="failed to get container status \"821d4f31a55d883a02463e0b146241e5c00844e359cc527c6f9e453d3763d9dc\": rpc error: code = NotFound desc = could not find container \"821d4f31a55d883a02463e0b146241e5c00844e359cc527c6f9e453d3763d9dc\": container with ID starting with 821d4f31a55d883a02463e0b146241e5c00844e359cc527c6f9e453d3763d9dc not found: ID does not exist"
Sep 10 16:04:05.771310 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:04:05.771280    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-66f7f9db8f-5f6qx"]
Sep 10 16:04:05.782762 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:04:05.782736    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-66f7f9db8f-5f6qx"]
Sep 10 16:04:06.784909 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:04:06.784867    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b539f448-ba5c-4073-ab4c-d92c206cea3c" path="/var/lib/kubelet/pods/b539f448-ba5c-4073-ab4c-d92c206cea3c/volumes"
Sep 10 16:04:08.309411 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:04:08.309374    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bs7hd"]
Sep 10 16:04:14.818564 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:04:14.818525    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-bs7hd"]
Sep 10 16:04:23.621081 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:04:23.621052    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-pw7kv_1315ff10-e055-458c-9e83-7199ef2c576a/manager/0.log"
Sep 10 16:04:23.749211 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:04:23.749182    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-6w8hg_49ace7c7-c5ee-46fc-8f39-3334100ea261/manager/0.log"
Sep 10 16:04:23.988325 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:04:23.988298    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-wffnd_fab62d93-c65f-4c84-92cf-8923ebd51112/server/0.log"
Sep 10 16:04:24.235223 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:04:24.235196    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-797ffb6d9c-6rqgv_72e35df7-9069-4147-8025-408e7c8d08e1/copy-manifests/0.log"
Sep 10 16:04:24.247779 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:04:24.247719    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-797ffb6d9c-6rqgv_72e35df7-9069-4147-8025-408e7c8d08e1/manager/0.log"
Sep 10 16:04:24.965452 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:04:24.965423    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-5df4475445-j29wd_7eb435b8-5bd8-4062-a79a-bab6490adb71/maas-api/0.log"
Sep 10 16:04:25.094193 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:04:25.094165    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29817585-wqsgn_f1f6c3c4-a9b8-4783-9a5a-218e1d82d727/cleanup/0.log"
Sep 10 16:04:26.270921 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:04:26.270889    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-zl8r9_b57d4eb5-1764-4f58-8fe7-137e9b02a2a2/manager/0.log"
Sep 10 16:04:26.391643 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:04:26.391612    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-vc2w4_24787bdd-6822-40e8-b6b4-c701ce34f0dc/manager/0.log"
Sep 10 16:04:26.513591 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:04:26.513552    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-m6j75_173ab4ec-9a00-40f5-b459-a6b8ac84779a/kuadrant-console-plugin/0.log"
Sep 10 16:04:26.647297 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:04:26.647221    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-tstxf_dc07a35b-729a-43c2-88cb-e96578f3af02/manager/0.log"
Sep 10 16:04:26.798369 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:04:26.798299    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-bs7hd_3f6d47db-a6d6-49ee-88a8-0239dbdee32f/limitador/0.log"
Sep 10 16:04:26.927081 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:04:26.927054    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-7gjkr_1d313e05-146a-428f-8e9c-8709bd11293c/manager/0.log"
Sep 10 16:04:27.667431 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:04:27.667405    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-785ff899f9-wx2md_2c775c3c-0dff-4fbd-bb68-67e4fcf59820/kube-auth-proxy/0.log"
Sep 10 16:04:28.735266 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:04:28.735231    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-8nw6r_6ce9541a-aa1f-417b-a31a-263a328bae29/main/0.log"
Sep 10 16:04:28.971037 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:04:28.971008    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwcp7h_1f5629ad-b3bb-4286-bd99-97ea74ff324c/main/0.log"
Sep 10 16:04:29.219344 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:04:29.219313    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-wmk4q_1f6fc26b-0880-4957-a0fd-199b0b7aa389/main/0.log"
Sep 10 16:04:37.882726 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:04:37.882696    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-7kktn_96b96370-410d-447c-8fb0-63f56ab52310/global-pull-secret-syncer/0.log"
Sep 10 16:04:38.074270 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:04:38.074237    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-wgbw2_7a9f7d3b-8a21-40ed-ab47-067f97ed537a/konnectivity-agent/0.log"
Sep 10 16:04:38.131852 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:04:38.131824    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-30.ec2.internal_5aa341fc62b3c7ff684a265510f7e3fe/haproxy/0.log"
Sep 10 16:04:42.465275 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:04:42.465246    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-zl8r9_b57d4eb5-1764-4f58-8fe7-137e9b02a2a2/manager/0.log"
Sep 10 16:04:42.542810 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:04:42.542779    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-vc2w4_24787bdd-6822-40e8-b6b4-c701ce34f0dc/manager/0.log"
Sep 10 16:04:42.593238 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:04:42.593202    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-m6j75_173ab4ec-9a00-40f5-b459-a6b8ac84779a/kuadrant-console-plugin/0.log"
Sep 10 16:04:42.661012 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:04:42.660976    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-tstxf_dc07a35b-729a-43c2-88cb-e96578f3af02/manager/0.log"
Sep 10 16:04:42.705949 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:04:42.705916    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-bs7hd_3f6d47db-a6d6-49ee-88a8-0239dbdee32f/limitador/0.log"
Sep 10 16:04:42.775381 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:04:42.775293    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-7gjkr_1d313e05-146a-428f-8e9c-8709bd11293c/manager/0.log"
Sep 10 16:04:44.480718 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:04:44.480691    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cdcc96965-np6wn_e12b28c6-2050-4b61-9b4d-e87606ac4a97/cluster-monitoring-operator/0.log"
Sep 10 16:04:44.795965 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:04:44.795887    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hwn4s_59556fec-b586-41af-b85f-0e370544f35a/node-exporter/0.log"
Sep 10 16:04:44.835658 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:04:44.835631    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hwn4s_59556fec-b586-41af-b85f-0e370544f35a/kube-rbac-proxy/0.log"
Sep 10 16:04:44.875157 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:04:44.875134    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hwn4s_59556fec-b586-41af-b85f-0e370544f35a/init-textfile/0.log"
Sep 10 16:04:48.223139 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:04:48.223104    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-cs24j/perf-node-gather-daemonset-lh98c"]
Sep 10 16:04:48.223528 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:04:48.223463    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b539f448-ba5c-4073-ab4c-d92c206cea3c" containerName="maas-api"
Sep 10 16:04:48.223528 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:04:48.223475    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b539f448-ba5c-4073-ab4c-d92c206cea3c" containerName="maas-api"
Sep 10 16:04:48.223631 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:04:48.223534    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="b539f448-ba5c-4073-ab4c-d92c206cea3c" containerName="maas-api"
Sep 10 16:04:48.226560 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:04:48.226536    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cs24j/perf-node-gather-daemonset-lh98c"
Sep 10 16:04:48.229176 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:04:48.229147    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-cs24j\"/\"kube-root-ca.crt\""
Sep 10 16:04:48.229176 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:04:48.229150    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-cs24j\"/\"openshift-service-ca.crt\""
Sep 10 16:04:48.229413 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:04:48.229211    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-cs24j\"/\"default-dockercfg-sr2kk\""
Sep 10 16:04:48.241586 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:04:48.241550    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cs24j/perf-node-gather-daemonset-lh98c"]
Sep 10 16:04:48.327943 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:04:48.327907    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d61e8c53-bfae-48e3-88a4-d1c1d51841d5-sys\") pod \"perf-node-gather-daemonset-lh98c\" (UID: \"d61e8c53-bfae-48e3-88a4-d1c1d51841d5\") " pod="openshift-must-gather-cs24j/perf-node-gather-daemonset-lh98c"
Sep 10 16:04:48.328122 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:04:48.327963    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d61e8c53-bfae-48e3-88a4-d1c1d51841d5-proc\") pod \"perf-node-gather-daemonset-lh98c\" (UID: \"d61e8c53-bfae-48e3-88a4-d1c1d51841d5\") " pod="openshift-must-gather-cs24j/perf-node-gather-daemonset-lh98c"
Sep 10 16:04:48.328122 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:04:48.328031    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d61e8c53-bfae-48e3-88a4-d1c1d51841d5-podres\") pod \"perf-node-gather-daemonset-lh98c\" (UID: \"d61e8c53-bfae-48e3-88a4-d1c1d51841d5\") " pod="openshift-must-gather-cs24j/perf-node-gather-daemonset-lh98c"
Sep 10 16:04:48.328122 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:04:48.328069    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d61e8c53-bfae-48e3-88a4-d1c1d51841d5-lib-modules\") pod \"perf-node-gather-daemonset-lh98c\" (UID: \"d61e8c53-bfae-48e3-88a4-d1c1d51841d5\") " pod="openshift-must-gather-cs24j/perf-node-gather-daemonset-lh98c"
Sep 10 16:04:48.328122 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:04:48.328106    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75zlb\" (UniqueName: \"kubernetes.io/projected/d61e8c53-bfae-48e3-88a4-d1c1d51841d5-kube-api-access-75zlb\") pod \"perf-node-gather-daemonset-lh98c\" (UID: \"d61e8c53-bfae-48e3-88a4-d1c1d51841d5\") " pod="openshift-must-gather-cs24j/perf-node-gather-daemonset-lh98c"
Sep 10 16:04:48.429214 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:04:48.429170    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75zlb\" (UniqueName: \"kubernetes.io/projected/d61e8c53-bfae-48e3-88a4-d1c1d51841d5-kube-api-access-75zlb\") pod \"perf-node-gather-daemonset-lh98c\" (UID: \"d61e8c53-bfae-48e3-88a4-d1c1d51841d5\") " pod="openshift-must-gather-cs24j/perf-node-gather-daemonset-lh98c"
Sep 10 16:04:48.429391 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:04:48.429222    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d61e8c53-bfae-48e3-88a4-d1c1d51841d5-sys\") pod \"perf-node-gather-daemonset-lh98c\" (UID: \"d61e8c53-bfae-48e3-88a4-d1c1d51841d5\") " pod="openshift-must-gather-cs24j/perf-node-gather-daemonset-lh98c"
Sep 10 16:04:48.429391 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:04:48.429256    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d61e8c53-bfae-48e3-88a4-d1c1d51841d5-proc\") pod \"perf-node-gather-daemonset-lh98c\" (UID: \"d61e8c53-bfae-48e3-88a4-d1c1d51841d5\") " pod="openshift-must-gather-cs24j/perf-node-gather-daemonset-lh98c"
Sep 10 16:04:48.429391 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:04:48.429324    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d61e8c53-bfae-48e3-88a4-d1c1d51841d5-proc\") pod \"perf-node-gather-daemonset-lh98c\" (UID: \"d61e8c53-bfae-48e3-88a4-d1c1d51841d5\") " pod="openshift-must-gather-cs24j/perf-node-gather-daemonset-lh98c"
Sep 10 16:04:48.429391 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:04:48.429365    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d61e8c53-bfae-48e3-88a4-d1c1d51841d5-sys\") pod \"perf-node-gather-daemonset-lh98c\" (UID: \"d61e8c53-bfae-48e3-88a4-d1c1d51841d5\") " pod="openshift-must-gather-cs24j/perf-node-gather-daemonset-lh98c"
Sep 10 16:04:48.429540 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:04:48.429405    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d61e8c53-bfae-48e3-88a4-d1c1d51841d5-podres\") pod \"perf-node-gather-daemonset-lh98c\" (UID: \"d61e8c53-bfae-48e3-88a4-d1c1d51841d5\") " pod="openshift-must-gather-cs24j/perf-node-gather-daemonset-lh98c"
Sep 10 16:04:48.429540 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:04:48.429464    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d61e8c53-bfae-48e3-88a4-d1c1d51841d5-lib-modules\") pod \"perf-node-gather-daemonset-lh98c\" (UID: \"d61e8c53-bfae-48e3-88a4-d1c1d51841d5\") " pod="openshift-must-gather-cs24j/perf-node-gather-daemonset-lh98c"
Sep 10 16:04:48.429644 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:04:48.429559    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d61e8c53-bfae-48e3-88a4-d1c1d51841d5-podres\") pod \"perf-node-gather-daemonset-lh98c\" (UID: \"d61e8c53-bfae-48e3-88a4-d1c1d51841d5\") " pod="openshift-must-gather-cs24j/perf-node-gather-daemonset-lh98c"
Sep 10 16:04:48.429680 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:04:48.429640    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d61e8c53-bfae-48e3-88a4-d1c1d51841d5-lib-modules\") pod \"perf-node-gather-daemonset-lh98c\" (UID: \"d61e8c53-bfae-48e3-88a4-d1c1d51841d5\") " pod="openshift-must-gather-cs24j/perf-node-gather-daemonset-lh98c"
Sep 10 16:04:48.438161 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:04:48.438142    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-75zlb\" (UniqueName: \"kubernetes.io/projected/d61e8c53-bfae-48e3-88a4-d1c1d51841d5-kube-api-access-75zlb\") pod \"perf-node-gather-daemonset-lh98c\" (UID: \"d61e8c53-bfae-48e3-88a4-d1c1d51841d5\") " pod="openshift-must-gather-cs24j/perf-node-gather-daemonset-lh98c"
Sep 10 16:04:48.536413 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:04:48.536312    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cs24j/perf-node-gather-daemonset-lh98c"
Sep 10 16:04:48.809238 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:04:48.809214    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cs24j/perf-node-gather-daemonset-lh98c"]
Sep 10 16:04:48.811840 ip-10-0-130-30 kubenswrapper[2559]: W0910 16:04:48.811805    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd61e8c53_bfae_48e3_88a4_d1c1d51841d5.slice/crio-6051a9e8f221011809d8ed46f599ba6e42d916fe7b637f5d50ff18a10904533a WatchSource:0}: Error finding container 6051a9e8f221011809d8ed46f599ba6e42d916fe7b637f5d50ff18a10904533a: Status 404 returned error can't find the container with id 6051a9e8f221011809d8ed46f599ba6e42d916fe7b637f5d50ff18a10904533a
Sep 10 16:04:48.969025 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:04:48.968996    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cs24j/perf-node-gather-daemonset-lh98c" event={"ID":"d61e8c53-bfae-48e3-88a4-d1c1d51841d5","Type":"ContainerStarted","Data":"e30a7e433ebe696f695efc9ccf06af68b98f0493b45802091907fdab2cd0cf30"}
Sep 10 16:04:48.969154 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:04:48.969048    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cs24j/perf-node-gather-daemonset-lh98c" event={"ID":"d61e8c53-bfae-48e3-88a4-d1c1d51841d5","Type":"ContainerStarted","Data":"6051a9e8f221011809d8ed46f599ba6e42d916fe7b637f5d50ff18a10904533a"}
Sep 10 16:04:48.969154 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:04:48.969071    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-cs24j/perf-node-gather-daemonset-lh98c"
Sep 10 16:04:48.995990 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:04:48.995945    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-cs24j/perf-node-gather-daemonset-lh98c" podStartSLOduration=0.99593044 podStartE2EDuration="995.93044ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 16:04:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:04:48.994517336 +0000 UTC m=+2264.736482438" watchObservedRunningTime="2026-09-10 16:04:48.99593044 +0000 UTC m=+2264.737895544"
Sep 10 16:04:49.374970 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:04:49.374941    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-695c895bd6-bk47h_31f6539c-3d4f-4bfc-a1d6-44d44e6d6e4a/volume-data-source-validator/0.log"
Sep 10 16:04:50.864693 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:04:50.864659    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-tj846_66850670-e07c-4df2-bfe0-fd3b5fcb3308/dns/0.log"
Sep 10 16:04:50.902384 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:04:50.902359    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-tj846_66850670-e07c-4df2-bfe0-fd3b5fcb3308/kube-rbac-proxy/0.log"
Sep 10 16:04:51.021159 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:04:51.021133    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-md7dv_c56c45fc-a485-48b6-a88a-355ba442927f/dns-node-resolver/0.log"
Sep 10 16:04:51.804547 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:04:51.804507    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-pzn2d_6db07abf-0c03-4565-a661-9df53fbe5957/node-ca/0.log"
Sep 10 16:04:53.621699 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:04:53.621667    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-785ff899f9-wx2md_2c775c3c-0dff-4fbd-bb68-67e4fcf59820/kube-auth-proxy/0.log"
Sep 10 16:04:54.754945 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:04:54.754913    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5dnj9_280d97b4-f718-4bf7-a173-11822ff796fd/serve-healthcheck-canary/0.log"
Sep 10 16:04:54.985101 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:04:54.985072    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-cs24j/perf-node-gather-daemonset-lh98c"
Sep 10 16:04:55.591183 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:04:55.591157    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-f69vm_f7537246-aff7-4408-be0d-7a1d69496837/kube-rbac-proxy/0.log"
Sep 10 16:04:55.634615 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:04:55.634564    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-f69vm_f7537246-aff7-4408-be0d-7a1d69496837/exporter/0.log"
Sep 10 16:04:55.663826 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:04:55.663798    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-f69vm_f7537246-aff7-4408-be0d-7a1d69496837/extractor/0.log"
Sep 10 16:04:58.341038 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:04:58.341006    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-pw7kv_1315ff10-e055-458c-9e83-7199ef2c576a/manager/0.log"
Sep 10 16:04:58.576372 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:04:58.576341    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-6w8hg_49ace7c7-c5ee-46fc-8f39-3334100ea261/manager/0.log"
Sep 10 16:04:58.676789 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:04:58.676758    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-wffnd_fab62d93-c65f-4c84-92cf-8923ebd51112/server/0.log"
Sep 10 16:04:58.885486 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:04:58.885384    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-797ffb6d9c-6rqgv_72e35df7-9069-4147-8025-408e7c8d08e1/manager/0.log"
Sep 10 16:04:58.915638 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:04:58.915613    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-797ffb6d9c-6rqgv_72e35df7-9069-4147-8025-408e7c8d08e1/copy-manifests/0.log"
Sep 10 16:04:59.265927 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:04:59.265896    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-785ff899f9-wx2md_2c775c3c-0dff-4fbd-bb68-67e4fcf59820/kube-auth-proxy/0.log"
Sep 10 16:05:00.667021 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:05:00.666993    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-v25lg_ea32fcf8-df19-4d72-82e9-9d48dcd4701d/openshift-lws-operator/0.log"
Sep 10 16:05:01.722354 ip-10-0-130-30 kubenswrapper[2559]: I0910 16:05:01.722321    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-785ff899f9-wx2md_2c775c3c-0dff-4fbd-bb68-67e4fcf59820/kube-auth-proxy/0.log"