Sep 10 19:08:45.268244 ip-10-0-132-222 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 10 19:08:45.268252 ip-10-0-132-222 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 10 19:08:45.268259 ip-10-0-132-222 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 10 19:08:45.268486 ip-10-0-132-222 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 10 19:08:55.421352 ip-10-0-132-222 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 10 19:08:55.421371 ip-10-0-132-222 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 1170f11a311d4e528c0c7b220d7d041d --
Sep 10 19:10:35.613787 ip-10-0-132-222 systemd[1]: Starting Kubernetes Kubelet...
Sep 10 19:10:36.009206 ip-10-0-132-222 kubenswrapper[2546]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 19:10:36.009206 ip-10-0-132-222 kubenswrapper[2546]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 10 19:10:36.009206 ip-10-0-132-222 kubenswrapper[2546]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 19:10:36.009206 ip-10-0-132-222 kubenswrapper[2546]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 10 19:10:36.009206 ip-10-0-132-222 kubenswrapper[2546]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 19:10:36.010015 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.009227    2546 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 10 19:10:36.016159 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.016134    2546 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 19:10:36.016159 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.016155    2546 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 19:10:36.016159 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.016162    2546 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 19:10:36.016368 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.016168    2546 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 19:10:36.016368 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.016174    2546 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 19:10:36.016368 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.016178    2546 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 19:10:36.016368 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.016184    2546 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 19:10:36.016368 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.016187    2546 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 19:10:36.016368 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.016192    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 19:10:36.016368 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.016196    2546 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 19:10:36.016368 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.016200    2546 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 19:10:36.016368 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.016204    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 19:10:36.016368 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.016208    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 19:10:36.016368 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.016212    2546 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 19:10:36.016368 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.016216    2546 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 19:10:36.016368 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.016220    2546 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 19:10:36.016368 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.016224    2546 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 19:10:36.016368 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.016229    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 19:10:36.016368 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.016233    2546 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 19:10:36.016368 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.016238    2546 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 19:10:36.016368 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.016242    2546 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 19:10:36.016368 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.016247    2546 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 19:10:36.016368 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.016251    2546 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 19:10:36.016992 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.016255    2546 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 19:10:36.016992 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.016259    2546 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 19:10:36.016992 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.016263    2546 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 19:10:36.016992 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.016267    2546 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 19:10:36.016992 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.016271    2546 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 19:10:36.016992 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.016275    2546 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 19:10:36.016992 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.016279    2546 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 19:10:36.016992 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.016283    2546 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 19:10:36.016992 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.016287    2546 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 19:10:36.016992 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.016291    2546 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 19:10:36.016992 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.016295    2546 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 19:10:36.016992 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.016299    2546 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 19:10:36.016992 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.016304    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 19:10:36.016992 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.016309    2546 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 19:10:36.016992 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.016314    2546 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 19:10:36.016992 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.016319    2546 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 19:10:36.016992 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.016323    2546 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 19:10:36.016992 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.016327    2546 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 19:10:36.016992 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.016331    2546 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 19:10:36.017597 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.016336    2546 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 19:10:36.017597 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.016340    2546 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 19:10:36.017597 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.016346    2546 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 19:10:36.017597 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.016350    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 19:10:36.017597 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.016354    2546 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 19:10:36.017597 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.016361    2546 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 19:10:36.017597 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.016368    2546 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 19:10:36.017597 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.016372    2546 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 19:10:36.017597 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.016377    2546 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 19:10:36.017597 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.016381    2546 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 19:10:36.017597 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.016386    2546 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 19:10:36.017597 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.016392    2546 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 19:10:36.017597 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.016397    2546 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 19:10:36.017597 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.016401    2546 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 19:10:36.017597 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.016405    2546 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 19:10:36.017597 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.016409    2546 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 19:10:36.017597 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.016414    2546 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 19:10:36.017597 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.016418    2546 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 19:10:36.017597 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.016422    2546 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 19:10:36.018417 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.016427    2546 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 19:10:36.018417 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.016434    2546 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 19:10:36.018417 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.016438    2546 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 19:10:36.018417 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.016442    2546 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 19:10:36.018417 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.016446    2546 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 19:10:36.018417 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.016450    2546 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 19:10:36.018417 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.016469    2546 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 19:10:36.018417 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.016475    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 19:10:36.018417 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.016479    2546 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 19:10:36.018417 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.016483    2546 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 19:10:36.018417 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.016490    2546 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 19:10:36.018417 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.016496    2546 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 19:10:36.018417 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.016501    2546 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 19:10:36.018417 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.016506    2546 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 19:10:36.018417 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.016511    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 19:10:36.018417 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.016515    2546 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 19:10:36.018417 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.016521    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 19:10:36.018417 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.016526    2546 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 19:10:36.018417 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.016531    2546 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 19:10:36.019233 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.016535    2546 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 19:10:36.019233 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.016539    2546 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 19:10:36.019233 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.016543    2546 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 19:10:36.019233 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.016548    2546 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 19:10:36.019233 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.016552    2546 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 19:10:36.019233 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.016557    2546 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 19:10:36.019233 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.016561    2546 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 19:10:36.019233 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.016565    2546 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 19:10:36.019233 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.016570    2546 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 19:10:36.019233 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.016573    2546 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 19:10:36.019233 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.016577    2546 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 19:10:36.019233 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.016581    2546 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 19:10:36.019233 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.016586    2546 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 19:10:36.019233 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.016590    2546 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 19:10:36.019233 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.016594    2546 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 19:10:36.019233 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.016598    2546 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 19:10:36.019233 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.016602    2546 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 19:10:36.019233 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.016606    2546 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 19:10:36.019233 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.016610    2546 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 19:10:36.019233 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.018443    2546 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 19:10:36.020090 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.018455    2546 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 19:10:36.020090 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.018481    2546 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 19:10:36.020090 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.018486    2546 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 19:10:36.020090 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.018491    2546 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 19:10:36.020090 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.018495    2546 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 19:10:36.020090 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.018500    2546 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 19:10:36.020090 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.018504    2546 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 19:10:36.020090 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.018508    2546 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 19:10:36.020090 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.018512    2546 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 19:10:36.020090 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.018516    2546 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 19:10:36.020090 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.018520    2546 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 19:10:36.020090 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.018524    2546 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 19:10:36.020090 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.018528    2546 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 19:10:36.020090 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.018533    2546 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 19:10:36.020090 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.018537    2546 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 19:10:36.020090 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.018541    2546 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 19:10:36.020090 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.018545    2546 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 19:10:36.020090 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.018549    2546 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 19:10:36.020090 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.018555    2546 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 19:10:36.020090 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.018559    2546 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 19:10:36.020973 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.018563    2546 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 19:10:36.020973 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.018567    2546 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 19:10:36.020973 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.018571    2546 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 19:10:36.020973 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.018576    2546 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 19:10:36.020973 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.018580    2546 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 19:10:36.020973 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.018584    2546 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 19:10:36.020973 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.018589    2546 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 19:10:36.020973 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.018593    2546 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 19:10:36.020973 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.018597    2546 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 19:10:36.020973 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.018601    2546 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 19:10:36.020973 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.018608    2546 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 19:10:36.020973 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.018612    2546 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 19:10:36.020973 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.018616    2546 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 19:10:36.020973 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.018621    2546 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 19:10:36.020973 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.018625    2546 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 19:10:36.020973 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.018629    2546 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 19:10:36.020973 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.018634    2546 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 19:10:36.020973 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.018638    2546 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 19:10:36.020973 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.018642    2546 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 19:10:36.020973 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.018647    2546 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 19:10:36.021823 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.018654    2546 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 19:10:36.021823 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.018660    2546 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 19:10:36.021823 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.018665    2546 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 19:10:36.021823 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.018670    2546 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 19:10:36.021823 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.018675    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 19:10:36.021823 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.018680    2546 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 19:10:36.021823 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.018684    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 19:10:36.021823 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.018689    2546 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 19:10:36.021823 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.018694    2546 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 19:10:36.021823 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.018698    2546 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 19:10:36.021823 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.018702    2546 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 19:10:36.021823 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.018707    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 19:10:36.021823 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.018711    2546 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 19:10:36.021823 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.018715    2546 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 19:10:36.021823 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.018720    2546 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 19:10:36.021823 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.018724    2546 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 19:10:36.021823 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.018728    2546 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 19:10:36.021823 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.018732    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 19:10:36.021823 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.018737    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 19:10:36.022357 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.018741    2546 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 19:10:36.022357 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.018745    2546 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 19:10:36.022357 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.018750    2546 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 19:10:36.022357 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.018754    2546 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 19:10:36.022357 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.018758    2546 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 19:10:36.022357 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.018763    2546 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 19:10:36.022357 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.018766    2546 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 19:10:36.022357 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.018771    2546 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 19:10:36.022357 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.018775    2546 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 19:10:36.022357 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.018782    2546 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 19:10:36.022357 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.018787    2546 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 19:10:36.022357 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.018791    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 19:10:36.022357 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.018795    2546 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 19:10:36.022357 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.018803    2546 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 19:10:36.022357 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.018810    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 19:10:36.022357 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.018815    2546 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 19:10:36.022357 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.018820    2546 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 19:10:36.022357 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.018824    2546 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 19:10:36.022357 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.018828    2546 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 19:10:36.022859 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.018832    2546 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 19:10:36.022859 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.018838    2546 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 19:10:36.022859 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.018843    2546 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 19:10:36.022859 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.018848    2546 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 19:10:36.022859 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.018852    2546 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 19:10:36.022859 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.018856    2546 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 19:10:36.022859 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.018861    2546 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 19:10:36.022859 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.018865    2546 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 19:10:36.022859 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.018869    2546 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 19:10:36.022859 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.018874    2546 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 19:10:36.022859 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.018878    2546 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 19:10:36.022859 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.018882    2546 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 19:10:36.022859 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.018886    2546 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 19:10:36.022859 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.018890    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 19:10:36.022859 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.018894    2546 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 19:10:36.022859 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.018898    2546 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 19:10:36.022859 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.018903    2546 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 19:10:36.022859 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.018907    2546 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 19:10:36.022859 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.018912    2546 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 19:10:36.023328 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.018916    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 19:10:36.023328 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.019530    2546 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 19:10:36.023328 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.019539    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 19:10:36.023328 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.019546    2546 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 19:10:36.023328 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.019550    2546 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 19:10:36.023328 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.019555    2546 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 19:10:36.023328 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.019559    2546 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 19:10:36.023328 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.019563    2546 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 19:10:36.023328 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.019568    2546 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 19:10:36.023328 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.019572    2546 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 19:10:36.023328 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.019577    2546 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 19:10:36.023328 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.019581    2546 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 19:10:36.023328 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.019585    2546 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 19:10:36.023328 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.019589    2546 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 19:10:36.023328 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.019594    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 19:10:36.023328 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.019598    2546 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 19:10:36.023328 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.019602    2546 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 19:10:36.023328 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.019606    2546 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 19:10:36.023328 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.019610    2546 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 19:10:36.023328 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.019615    2546 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 19:10:36.023901 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.019619    2546 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 19:10:36.023901 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.019623    2546 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 19:10:36.023901 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.019628    2546 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 19:10:36.023901 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.019632    2546 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 19:10:36.023901 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.019636    2546 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 19:10:36.023901 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.019642    2546 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 19:10:36.023901 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.019646    2546 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 19:10:36.023901 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.019650    2546 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 19:10:36.023901 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.019655    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 19:10:36.023901 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.019659    2546 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 19:10:36.023901 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.019663    2546 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 19:10:36.023901 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.019670    2546 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 19:10:36.023901 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.019675    2546 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 19:10:36.023901 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.019680    2546 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 19:10:36.023901 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.019684    2546 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 19:10:36.023901 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.019691    2546 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 19:10:36.023901 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.019695    2546 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 19:10:36.023901 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.019700    2546 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 19:10:36.023901 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.019704    2546 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 19:10:36.024413 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.019708    2546 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 19:10:36.024413 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.019712    2546 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 19:10:36.024413 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.019716    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 19:10:36.024413 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.019721    2546 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 19:10:36.024413 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.019725    2546 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 19:10:36.024413 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.019729    2546 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 19:10:36.024413 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.019733    2546 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 19:10:36.024413 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.019737    2546 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 19:10:36.024413 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.019741    2546 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 19:10:36.024413 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.019745    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 19:10:36.024413 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.019751    2546 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 19:10:36.024413 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.019757    2546 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 19:10:36.024413 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.019761    2546 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 19:10:36.024413 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.019766    2546 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 19:10:36.024413 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.019771    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 19:10:36.024413 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.019774    2546 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 19:10:36.024413 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.019779    2546 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 19:10:36.024413 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.019783    2546 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 19:10:36.024413 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.019790    2546 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 19:10:36.024953 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.019797    2546 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 19:10:36.024953 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.019802    2546 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 19:10:36.024953 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.019807    2546 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 19:10:36.024953 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.019811    2546 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 19:10:36.024953 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.019815    2546 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 19:10:36.024953 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.019819    2546 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 19:10:36.024953 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.019823    2546 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 19:10:36.024953 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.019828    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 19:10:36.024953 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.019832    2546 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 19:10:36.024953 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.019839    2546 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 19:10:36.024953 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.019845    2546 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 19:10:36.024953 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.019849    2546 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 19:10:36.024953 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.019853    2546 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 19:10:36.024953 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.019857    2546 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 19:10:36.024953 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.019861    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 19:10:36.024953 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.019865    2546 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 19:10:36.024953 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.019869    2546 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 19:10:36.024953 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.019873    2546 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 19:10:36.024953 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.019878    2546 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 19:10:36.025484 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.019882    2546 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 19:10:36.025484 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.019886    2546 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 19:10:36.025484 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.019890    2546 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 19:10:36.025484 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.019894    2546 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 19:10:36.025484 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.019898    2546 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 19:10:36.025484 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.019903    2546 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 19:10:36.025484 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.019907    2546 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 19:10:36.025484 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.019911    2546 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 19:10:36.025484 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.019916    2546 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 19:10:36.025484 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.019920    2546 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 19:10:36.025484 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.019924    2546 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 19:10:36.025484 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.019928    2546 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 19:10:36.025484 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.019936    2546 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 19:10:36.025484 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.019940    2546 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 19:10:36.025484 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.019944    2546 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 19:10:36.025484 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.019948    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 19:10:36.025484 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.019952    2546 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 19:10:36.025484 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.019956    2546 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 19:10:36.025484 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.019960    2546 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 19:10:36.025484 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.019964    2546 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 19:10:36.026049 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.019968    2546 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 19:10:36.026049 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.019972    2546 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 19:10:36.026049 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.019977    2546 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 19:10:36.026049 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020087    2546 flags.go:64] FLAG: --address="0.0.0.0"
Sep 10 19:10:36.026049 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020137    2546 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 10 19:10:36.026049 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020147    2546 flags.go:64] FLAG: --anonymous-auth="true"
Sep 10 19:10:36.026049 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020154    2546 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 10 19:10:36.026049 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020161    2546 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 10 19:10:36.026049 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020166    2546 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 10 19:10:36.026049 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020173    2546 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 10 19:10:36.026049 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020180    2546 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 10 19:10:36.026049 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020185    2546 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 10 19:10:36.026049 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020190    2546 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 10 19:10:36.026049 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020195    2546 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 10 19:10:36.026049 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020201    2546 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 10 19:10:36.026049 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020206    2546 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 10 19:10:36.026049 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020211    2546 flags.go:64] FLAG: --cgroup-root=""
Sep 10 19:10:36.026049 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020216    2546 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 10 19:10:36.026049 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020221    2546 flags.go:64] FLAG: --client-ca-file=""
Sep 10 19:10:36.026049 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020225    2546 flags.go:64] FLAG: --cloud-provider="external"
Sep 10 19:10:36.026049 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020230    2546 flags.go:64] FLAG: --cluster-dns="[]"
Sep 10 19:10:36.026049 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020238    2546 flags.go:64] FLAG: --cluster-domain=""
Sep 10 19:10:36.026049 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020242    2546 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 10 19:10:36.026049 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020248    2546 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 10 19:10:36.026678 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020255    2546 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 10 19:10:36.026678 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020260    2546 flags.go:64] FLAG: --container-log-max-files="5"
Sep 10 19:10:36.026678 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020266    2546 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 10 19:10:36.026678 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020271    2546 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 10 19:10:36.026678 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020276    2546 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 10 19:10:36.026678 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020282    2546 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 10 19:10:36.026678 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020287    2546 flags.go:64] FLAG: --contention-profiling="false"
Sep 10 19:10:36.026678 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020292    2546 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 10 19:10:36.026678 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020297    2546 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 10 19:10:36.026678 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020302    2546 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 10 19:10:36.026678 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020308    2546 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 10 19:10:36.026678 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020314    2546 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 10 19:10:36.026678 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020319    2546 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 10 19:10:36.026678 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020324    2546 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 10 19:10:36.026678 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020329    2546 flags.go:64] FLAG: --enable-load-reader="false"
Sep 10 19:10:36.026678 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020334    2546 flags.go:64] FLAG: --enable-server="true"
Sep 10 19:10:36.026678 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020339    2546 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 10 19:10:36.026678 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020346    2546 flags.go:64] FLAG: --event-burst="100"
Sep 10 19:10:36.026678 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020351    2546 flags.go:64] FLAG: --event-qps="50"
Sep 10 19:10:36.026678 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020356    2546 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 10 19:10:36.026678 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020361    2546 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 10 19:10:36.026678 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020366    2546 flags.go:64] FLAG: --eviction-hard=""
Sep 10 19:10:36.026678 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020372    2546 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 10 19:10:36.026678 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020378    2546 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 10 19:10:36.027260 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020383    2546 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 10 19:10:36.027260 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020387    2546 flags.go:64] FLAG: --eviction-soft=""
Sep 10 19:10:36.027260 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020392    2546 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 10 19:10:36.027260 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020397    2546 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 10 19:10:36.027260 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020401    2546 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 10 19:10:36.027260 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020406    2546 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 10 19:10:36.027260 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020412    2546 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 10 19:10:36.027260 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020417    2546 flags.go:64] FLAG: --fail-swap-on="true"
Sep 10 19:10:36.027260 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020423    2546 flags.go:64] FLAG: --feature-gates=""
Sep 10 19:10:36.027260 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020430    2546 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 10 19:10:36.027260 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020434    2546 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 10 19:10:36.027260 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020439    2546 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 10 19:10:36.027260 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020444    2546 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 10 19:10:36.027260 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020450    2546 flags.go:64] FLAG: --healthz-port="10248"
Sep 10 19:10:36.027260 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020455    2546 flags.go:64] FLAG: --help="false"
Sep 10 19:10:36.027260 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020476    2546 flags.go:64] FLAG: --hostname-override="ip-10-0-132-222.ec2.internal"
Sep 10 19:10:36.027260 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020482    2546 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 10 19:10:36.027260 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020487    2546 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 10 19:10:36.027260 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020492    2546 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 10 19:10:36.027260 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020498    2546 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 10 19:10:36.027260 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020503    2546 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 10 19:10:36.027260 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020508    2546 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 10 19:10:36.027260 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020513    2546 flags.go:64] FLAG: --image-service-endpoint=""
Sep 10 19:10:36.028209 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020519    2546 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 10 19:10:36.028209 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020524    2546 flags.go:64] FLAG: --kube-api-burst="100"
Sep 10 19:10:36.028209 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020529    2546 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 10 19:10:36.028209 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020534    2546 flags.go:64] FLAG: --kube-api-qps="50"
Sep 10 19:10:36.028209 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020539    2546 flags.go:64] FLAG: --kube-reserved=""
Sep 10 19:10:36.028209 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020544    2546 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 10 19:10:36.028209 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020548    2546 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 10 19:10:36.028209 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020553    2546 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 10 19:10:36.028209 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020558    2546 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 10 19:10:36.028209 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020563    2546 flags.go:64] FLAG: --lock-file=""
Sep 10 19:10:36.028209 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020568    2546 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 10 19:10:36.028209 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020573    2546 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 10 19:10:36.028209 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020578    2546 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 10 19:10:36.028209 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020587    2546 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 10 19:10:36.028209 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020591    2546 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 10 19:10:36.028209 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020596    2546 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 10 19:10:36.028209 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020602    2546 flags.go:64] FLAG: --logging-format="text"
Sep 10 19:10:36.028209 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020608    2546 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 10 19:10:36.028209 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020614    2546 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 10 19:10:36.028209 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020619    2546 flags.go:64] FLAG: --manifest-url=""
Sep 10 19:10:36.028209 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020623    2546 flags.go:64] FLAG: --manifest-url-header=""
Sep 10 19:10:36.028209 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020630    2546 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 10 19:10:36.028209 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020635    2546 flags.go:64] FLAG: --max-open-files="1000000"
Sep 10 19:10:36.028209 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020641    2546 flags.go:64] FLAG: --max-pods="110"
Sep 10 19:10:36.028209 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020646    2546 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 10 19:10:36.028925 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020651    2546 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 10 19:10:36.028925 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020657    2546 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 10 19:10:36.028925 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020662    2546 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 10 19:10:36.028925 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020666    2546 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 10 19:10:36.028925 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020671    2546 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 10 19:10:36.028925 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020676    2546 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 10 19:10:36.028925 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020688    2546 flags.go:64] FLAG: --node-status-max-images="50"
Sep 10 19:10:36.028925 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020693    2546 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 10 19:10:36.028925 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020697    2546 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 10 19:10:36.028925 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020702    2546 flags.go:64] FLAG: --pod-cidr=""
Sep 10 19:10:36.028925 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020707    2546 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 10 19:10:36.028925 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020717    2546 flags.go:64] FLAG: --pod-manifest-path=""
Sep 10 19:10:36.028925 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020721    2546 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 10 19:10:36.028925 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020726    2546 flags.go:64] FLAG: --pods-per-core="0"
Sep 10 19:10:36.028925 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020731    2546 flags.go:64] FLAG: --port="10250"
Sep 10 19:10:36.028925 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020736    2546 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 10 19:10:36.028925 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020741    2546 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0cfc745f1ec4c92c2"
Sep 10 19:10:36.028925 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020746    2546 flags.go:64] FLAG: --qos-reserved=""
Sep 10 19:10:36.028925 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020751    2546 flags.go:64] FLAG: --read-only-port="10255"
Sep 10 19:10:36.028925 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020756    2546 flags.go:64] FLAG: --register-node="true"
Sep 10 19:10:36.028925 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020760    2546 flags.go:64] FLAG: --register-with-taints=""
Sep 10 19:10:36.028925 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020766    2546 flags.go:64] FLAG: --registry-burst="10"
Sep 10 19:10:36.028925 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020771    2546 flags.go:64] FLAG: --registry-qps="5"
Sep 10 19:10:36.028925 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020776    2546 flags.go:64] FLAG: --reserved-cpus=""
Sep 10 19:10:36.029552 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020782    2546 flags.go:64] FLAG: --reserved-memory=""
Sep 10 19:10:36.029552 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020789    2546 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 10 19:10:36.029552 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020795    2546 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 10 19:10:36.029552 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020800    2546 flags.go:64] FLAG: --rotate-certificates="false"
Sep 10 19:10:36.029552 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020804    2546 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 10 19:10:36.029552 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020809    2546 flags.go:64] FLAG: --runonce="false"
Sep 10 19:10:36.029552 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020813    2546 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 10 19:10:36.029552 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020819    2546 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 10 19:10:36.029552 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020824    2546 flags.go:64] FLAG: --seccomp-default="false"
Sep 10 19:10:36.029552 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020829    2546 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 10 19:10:36.029552 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020833    2546 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 10 19:10:36.029552 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020838    2546 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 10 19:10:36.029552 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020843    2546 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 10 19:10:36.029552 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020848    2546 flags.go:64] FLAG: --storage-driver-password="root"
Sep 10 19:10:36.029552 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020853    2546 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 10 19:10:36.029552 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020858    2546 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 10 19:10:36.029552 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020863    2546 flags.go:64] FLAG: --storage-driver-user="root"
Sep 10 19:10:36.029552 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020867    2546 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 10 19:10:36.029552 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020872    2546 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 10 19:10:36.029552 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020877    2546 flags.go:64] FLAG: --system-cgroups=""
Sep 10 19:10:36.029552 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020882    2546 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 10 19:10:36.029552 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020890    2546 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 10 19:10:36.029552 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020895    2546 flags.go:64] FLAG: --tls-cert-file=""
Sep 10 19:10:36.029552 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020900    2546 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 10 19:10:36.029552 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020906    2546 flags.go:64] FLAG: --tls-min-version=""
Sep 10 19:10:36.030175 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020911    2546 flags.go:64] FLAG: --tls-private-key-file=""
Sep 10 19:10:36.030175 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020916    2546 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 10 19:10:36.030175 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020920    2546 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 10 19:10:36.030175 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020925    2546 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 10 19:10:36.030175 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020930    2546 flags.go:64] FLAG: --v="2"
Sep 10 19:10:36.030175 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020936    2546 flags.go:64] FLAG: --version="false"
Sep 10 19:10:36.030175 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020943    2546 flags.go:64] FLAG: --vmodule=""
Sep 10 19:10:36.030175 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020949    2546 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 10 19:10:36.030175 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.020957    2546 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 10 19:10:36.030175 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.021160    2546 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 19:10:36.030175 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.021168    2546 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 19:10:36.030175 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.021174    2546 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 19:10:36.030175 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.021178    2546 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 19:10:36.030175 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.021183    2546 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 19:10:36.030175 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.021187    2546 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 19:10:36.030175 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.021192    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 19:10:36.030175 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.021196    2546 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 19:10:36.030175 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.021202    2546 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 19:10:36.030175 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.021206    2546 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 19:10:36.030175 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.021210    2546 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 19:10:36.030175 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.021214    2546 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 19:10:36.030175 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.021219    2546 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 19:10:36.030770 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.021223    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 19:10:36.030770 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.021227    2546 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 19:10:36.030770 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.021231    2546 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 19:10:36.030770 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.021235    2546 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 19:10:36.030770 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.021239    2546 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 19:10:36.030770 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.021243    2546 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 19:10:36.030770 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.021248    2546 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 19:10:36.030770 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.021252    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 19:10:36.030770 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.021256    2546 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 19:10:36.030770 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.021260    2546 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 19:10:36.030770 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.021264    2546 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 19:10:36.030770 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.021268    2546 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 19:10:36.030770 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.021273    2546 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 19:10:36.030770 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.021277    2546 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 19:10:36.030770 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.021281    2546 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 19:10:36.030770 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.021285    2546 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 19:10:36.030770 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.021289    2546 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 19:10:36.030770 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.021293    2546 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 19:10:36.030770 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.021298    2546 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 19:10:36.030770 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.021314    2546 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 19:10:36.031275 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.021318    2546 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 19:10:36.031275 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.021322    2546 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 19:10:36.031275 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.021326    2546 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 19:10:36.031275 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.021330    2546 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 19:10:36.031275 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.021334    2546 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 19:10:36.031275 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.021338    2546 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 19:10:36.031275 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.021345    2546 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 19:10:36.031275 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.021352    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 19:10:36.031275 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.021357    2546 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 19:10:36.031275 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.021361    2546 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 19:10:36.031275 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.021365    2546 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 19:10:36.031275 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.021369    2546 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 19:10:36.031275 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.021373    2546 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 19:10:36.031275 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.021380    2546 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 19:10:36.031275 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.021386    2546 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 19:10:36.031275 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.021390    2546 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 19:10:36.031275 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.021395    2546 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 19:10:36.031275 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.021399    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 19:10:36.031275 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.021404    2546 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 19:10:36.031775 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.021408    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 19:10:36.031775 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.021413    2546 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 19:10:36.031775 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.021417    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 19:10:36.031775 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.021421    2546 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 19:10:36.031775 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.021426    2546 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 19:10:36.031775 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.021430    2546 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 19:10:36.031775 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.021434    2546 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 19:10:36.031775 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.021438    2546 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 19:10:36.031775 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.021442    2546 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 19:10:36.031775 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.021446    2546 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 19:10:36.031775 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.021450    2546 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 19:10:36.031775 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.021455    2546 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 19:10:36.031775 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.021475    2546 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 19:10:36.031775 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.021480    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 19:10:36.031775 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.021485    2546 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 19:10:36.031775 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.021489    2546 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 19:10:36.031775 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.021494    2546 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 19:10:36.031775 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.021498    2546 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 19:10:36.031775 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.021502    2546 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 19:10:36.031775 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.021506    2546 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 19:10:36.032297 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.021512    2546 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 19:10:36.032297 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.021517    2546 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 19:10:36.032297 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.021521    2546 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 19:10:36.032297 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.021525    2546 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 19:10:36.032297 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.021530    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 19:10:36.032297 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.021534    2546 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 19:10:36.032297 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.021540    2546 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 19:10:36.032297 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.021546    2546 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 19:10:36.032297 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.021551    2546 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 19:10:36.032297 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.021555    2546 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 19:10:36.032297 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.021559    2546 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 19:10:36.032297 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.021564    2546 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 19:10:36.032297 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.021568    2546 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 19:10:36.032297 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.021572    2546 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 19:10:36.032297 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.021576    2546 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 19:10:36.032297 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.021580    2546 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 19:10:36.032297 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.021584    2546 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 19:10:36.032297 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.021588    2546 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 19:10:36.032297 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.021592    2546 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 19:10:36.032777 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.021596    2546 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 19:10:36.032777 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.021601    2546 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 19:10:36.032777 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.021606    2546 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 19:10:36.032777 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.021610    2546 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 19:10:36.032777 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.021614    2546 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 19:10:36.032777 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.021618    2546 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 19:10:36.032777 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.021623    2546 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 19:10:36.032777 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.021628    2546 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 19:10:36.032777 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.021635    2546 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 19:10:36.032777 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.029962    2546 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 10 19:10:36.032777 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.029980    2546 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 10 19:10:36.032777 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.030756    2546 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 10 19:10:36.032777 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.030767    2546 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 10 19:10:36.032777 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.030825    2546 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 19:10:36.032777 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.030831    2546 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 19:10:36.033159 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.030835    2546 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 19:10:36.033159 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.030838    2546 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 19:10:36.033159 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.030842    2546 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 19:10:36.033159 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.030845    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 19:10:36.033159 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.030848    2546 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 19:10:36.033159 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.030851    2546 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 19:10:36.033159 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.030853    2546 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 19:10:36.033159 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.030856    2546 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 19:10:36.033159 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.030859    2546 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 19:10:36.033159 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.030862    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 19:10:36.033159 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.030864    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 19:10:36.033159 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.030867    2546 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 19:10:36.033159 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.030870    2546 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 19:10:36.033159 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.030872    2546 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 19:10:36.033159 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.030875    2546 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 19:10:36.033159 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.030878    2546 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 19:10:36.033159 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.030880    2546 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 19:10:36.033159 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.030883    2546 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 19:10:36.033159 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.030886    2546 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 19:10:36.033159 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.030889    2546 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 19:10:36.033716 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.030891    2546 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 19:10:36.033716 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.030895    2546 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 19:10:36.033716 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.030906    2546 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 19:10:36.033716 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.030909    2546 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 19:10:36.033716 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.030913    2546 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 19:10:36.033716 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.030918    2546 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 19:10:36.033716 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.030921    2546 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 19:10:36.033716 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.030925    2546 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 19:10:36.033716 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.030928    2546 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 19:10:36.033716 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.030931    2546 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 19:10:36.033716 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.030934    2546 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 19:10:36.033716 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.030937    2546 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 19:10:36.033716 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.030941    2546 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 19:10:36.033716 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.030944    2546 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 19:10:36.033716 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.030946    2546 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 19:10:36.033716 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.030949    2546 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 19:10:36.033716 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.030951    2546 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 19:10:36.033716 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.030954    2546 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 19:10:36.033716 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.030957    2546 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 19:10:36.034219 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.030961    2546 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 19:10:36.034219 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.030965    2546 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 19:10:36.034219 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.030967    2546 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 19:10:36.034219 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.030970    2546 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 19:10:36.034219 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.030973    2546 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 19:10:36.034219 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.030977    2546 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 19:10:36.034219 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.030979    2546 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 19:10:36.034219 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.030982    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 19:10:36.034219 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.030985    2546 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 19:10:36.034219 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.030987    2546 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 19:10:36.034219 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.030990    2546 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 19:10:36.034219 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.030992    2546 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 19:10:36.034219 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.030995    2546 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 19:10:36.034219 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.030999    2546 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 19:10:36.034219 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031003    2546 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 19:10:36.034219 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031006    2546 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 19:10:36.034219 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031009    2546 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 19:10:36.034219 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031012    2546 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 19:10:36.034219 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031015    2546 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 19:10:36.034219 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031018    2546 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 19:10:36.034734 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031021    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 19:10:36.034734 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031024    2546 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 19:10:36.034734 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031027    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 19:10:36.034734 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031030    2546 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 19:10:36.034734 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031032    2546 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 19:10:36.034734 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031035    2546 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 19:10:36.034734 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031038    2546 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 19:10:36.034734 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031040    2546 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 19:10:36.034734 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031043    2546 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 19:10:36.034734 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031045    2546 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 19:10:36.034734 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031048    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 19:10:36.034734 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031051    2546 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 19:10:36.034734 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031053    2546 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 19:10:36.034734 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031056    2546 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 19:10:36.034734 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031059    2546 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 19:10:36.034734 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031061    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 19:10:36.034734 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031064    2546 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 19:10:36.034734 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031066    2546 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 19:10:36.034734 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031070    2546 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 19:10:36.034734 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031073    2546 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 19:10:36.035312 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031075    2546 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 19:10:36.035312 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031078    2546 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 19:10:36.035312 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031081    2546 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 19:10:36.035312 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031083    2546 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 19:10:36.035312 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031086    2546 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 19:10:36.035312 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031089    2546 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 19:10:36.035312 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031092    2546 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 19:10:36.035312 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031094    2546 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 19:10:36.035312 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031097    2546 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 19:10:36.035312 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031100    2546 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 19:10:36.035312 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031102    2546 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 19:10:36.035312 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031105    2546 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 19:10:36.035312 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031108    2546 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 19:10:36.035312 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031111    2546 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 19:10:36.035312 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031113    2546 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 19:10:36.035312 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031116    2546 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 19:10:36.035312 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031118    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 19:10:36.035312 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031121    2546 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 19:10:36.035882 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.031126    2546 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 19:10:36.035882 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031249    2546 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 19:10:36.035882 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031254    2546 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 19:10:36.035882 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031257    2546 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 19:10:36.035882 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031260    2546 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 19:10:36.035882 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031263    2546 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 19:10:36.035882 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031266    2546 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 19:10:36.035882 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031269    2546 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 19:10:36.035882 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031272    2546 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 19:10:36.035882 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031275    2546 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 19:10:36.035882 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031278    2546 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 19:10:36.035882 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031281    2546 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 19:10:36.035882 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031284    2546 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 19:10:36.035882 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031288    2546 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 19:10:36.036229 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031291    2546 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 19:10:36.036229 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031294    2546 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 19:10:36.036229 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031296    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 19:10:36.036229 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031299    2546 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 19:10:36.036229 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031302    2546 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 19:10:36.036229 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031305    2546 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 19:10:36.036229 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031309    2546 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 19:10:36.036229 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031312    2546 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 19:10:36.036229 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031316    2546 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 19:10:36.036229 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031319    2546 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 19:10:36.036229 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031322    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 19:10:36.036229 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031325    2546 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 19:10:36.036229 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031328    2546 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 19:10:36.036229 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031331    2546 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 19:10:36.036229 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031334    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 19:10:36.036229 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031336    2546 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 19:10:36.036229 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031339    2546 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 19:10:36.036229 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031343    2546 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 19:10:36.036229 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031345    2546 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 19:10:36.036888 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031348    2546 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 19:10:36.036888 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031351    2546 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 19:10:36.036888 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031354    2546 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 19:10:36.036888 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031356    2546 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 19:10:36.036888 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031359    2546 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 19:10:36.036888 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031362    2546 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 19:10:36.036888 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031364    2546 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 19:10:36.036888 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031367    2546 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 19:10:36.036888 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031369    2546 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 19:10:36.036888 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031373    2546 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 19:10:36.036888 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031375    2546 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 19:10:36.036888 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031378    2546 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 19:10:36.036888 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031381    2546 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 19:10:36.036888 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031385    2546 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 19:10:36.036888 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031389    2546 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 19:10:36.036888 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031392    2546 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 19:10:36.036888 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031395    2546 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 19:10:36.036888 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031398    2546 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 19:10:36.036888 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031401    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 19:10:36.036888 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031404    2546 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 19:10:36.037638 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031406    2546 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 19:10:36.037638 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031409    2546 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 19:10:36.037638 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031412    2546 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 19:10:36.037638 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031414    2546 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 19:10:36.037638 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031417    2546 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 19:10:36.037638 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031419    2546 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 19:10:36.037638 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031422    2546 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 19:10:36.037638 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031425    2546 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 19:10:36.037638 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031427    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 19:10:36.037638 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031430    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 19:10:36.037638 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031432    2546 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 19:10:36.037638 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031435    2546 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 19:10:36.037638 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031438    2546 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 19:10:36.037638 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031440    2546 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 19:10:36.037638 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031443    2546 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 19:10:36.037638 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031446    2546 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 19:10:36.037638 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031448    2546 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 19:10:36.037638 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031451    2546 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 19:10:36.037638 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031453    2546 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 19:10:36.038141 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031471    2546 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 19:10:36.038141 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031476    2546 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 19:10:36.038141 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031479    2546 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 19:10:36.038141 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031482    2546 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 19:10:36.038141 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031484    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 19:10:36.038141 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031488    2546 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 19:10:36.038141 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031490    2546 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 19:10:36.038141 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031506    2546 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 19:10:36.038141 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031510    2546 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 19:10:36.038141 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031514    2546 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 19:10:36.038141 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031517    2546 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 19:10:36.038141 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031519    2546 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 19:10:36.038141 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031522    2546 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 19:10:36.038141 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031525    2546 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 19:10:36.038141 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031527    2546 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 19:10:36.038141 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031530    2546 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 19:10:36.038141 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031532    2546 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 19:10:36.038141 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031535    2546 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 19:10:36.038141 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031537    2546 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 19:10:36.038141 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031540    2546 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 19:10:36.038697 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031542    2546 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 19:10:36.038697 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031545    2546 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 19:10:36.038697 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031547    2546 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 19:10:36.038697 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031550    2546 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 19:10:36.038697 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031553    2546 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 19:10:36.038697 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031555    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 19:10:36.038697 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031558    2546 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 19:10:36.038697 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:36.031560    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 19:10:36.038697 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.031566    2546 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 19:10:36.038697 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.031688    2546 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 10 19:10:36.038697 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.033751    2546 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 10 19:10:36.038697 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.034691    2546 server.go:1019] "Starting client certificate rotation"
Sep 10 19:10:36.038697 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.034797    2546 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 10 19:10:36.038697 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.034838    2546 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 10 19:10:36.039099 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.037035    2546 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 10 19:10:36.042189 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.042171    2546 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 10 19:10:36.053286 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.053268    2546 log.go:25] "Validated CRI v1 runtime API"
Sep 10 19:10:36.055981 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.055964    2546 log.go:25] "Validated CRI v1 image API"
Sep 10 19:10:36.058312 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.058288    2546 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 10 19:10:36.061096 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.061070    2546 fs.go:135] Filesystem UUIDs: map[0077bd99-4347-41ac-9cad-3e48dec9b54b:/dev/nvme0n1p4 6eef52c7-319c-48ed-bf51-030a09ace4ba:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Sep 10 19:10:36.061163 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.061095    2546 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 10 19:10:36.062404 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.062389    2546 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 10 19:10:36.067070 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.066953    2546 manager.go:216] Machine: {Timestamp:2026-09-10 19:10:36.066442913 +0000 UTC m=+0.334372217 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100688 MemoryCapacity:33164484608 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2fc041f2e78dfe1f3970acb1d31ede SystemUUID:ec2fc041-f2e7-8dfe-1f39-70acb1d31ede BootID:1170f11a-311d-4e52-8c0c-7b220d7d041d Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:77:40:75:be:77 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:77:40:75:be:77 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:da:9f:66:0d:60:fb Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164484608 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 10 19:10:36.067070 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.067058    2546 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 10 19:10:36.067197 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.067143    2546 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 10 19:10:36.069749 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.069714    2546 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 10 19:10:36.069892 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.069753    2546 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-222.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 10 19:10:36.070417 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.070407    2546 topology_manager.go:138] "Creating topology manager with none policy"
Sep 10 19:10:36.070446 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.070420    2546 container_manager_linux.go:309] "Creating device plugin manager"
Sep 10 19:10:36.070446 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.070428    2546 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 10 19:10:36.071115 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.071104    2546 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 10 19:10:36.072308 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.072296    2546 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 10 19:10:36.076181 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.076167    2546 state_mem.go:36] "Initialized new in-memory state store"
Sep 10 19:10:36.076299 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.076290    2546 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 10 19:10:36.076335 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.076320    2546 kubelet.go:486] "Attempting to sync node with API server"
Sep 10 19:10:36.076335 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.076330    2546 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 10 19:10:36.076388 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.076345    2546 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 10 19:10:36.076388 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.076354    2546 kubelet.go:398] "Adding apiserver pod source"
Sep 10 19:10:36.076388 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.076363    2546 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 10 19:10:36.076774 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.076760    2546 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 10 19:10:36.076821 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.076786    2546 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 10 19:10:36.078720 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.078700    2546 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 10 19:10:36.081106 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.081091    2546 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 10 19:10:36.081174 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.081121    2546 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 10 19:10:36.081282 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.081272    2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 10 19:10:36.081328 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.081286    2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 10 19:10:36.081328 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.081293    2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 10 19:10:36.081328 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.081298    2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 10 19:10:36.081328 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.081312    2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 10 19:10:36.081328 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.081318    2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 10 19:10:36.081835 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.081825    2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 10 19:10:36.081875 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.081836    2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 10 19:10:36.081875 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.081843    2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 10 19:10:36.081875 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.081849    2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 10 19:10:36.081875 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.081857    2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 10 19:10:36.081971 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.081878    2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 10 19:10:36.082778 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.082768    2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 10 19:10:36.082778 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.082778    2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 10 19:10:36.085795 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.085777    2546 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-222.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 19:10:36.085871 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:10:36.085790    2546 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 10 19:10:36.085871 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:10:36.085793    2546 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-222.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 10 19:10:36.087014 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.087001    2546 server.go:1268] "Started kubelet"
Sep 10 19:10:36.087155 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.087104    2546 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 10 19:10:36.087213 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.087172    2546 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 10 19:10:36.087261 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.087235    2546 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 10 19:10:36.087314 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.087196    2546 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-lp6cq"
Sep 10 19:10:36.088262 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.088237    2546 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 10 19:10:36.088299 ip-10-0-132-222 systemd[1]: Started Kubernetes Kubelet.
Sep 10 19:10:36.089432 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.089417    2546 server.go:310] "Adding debug handlers to kubelet server"
Sep 10 19:10:36.093729 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:10:36.093703    2546 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 10 19:10:36.094044 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:10:36.093072    2546 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-222.ec2.internal.18d40b5aaf2ed33b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-222.ec2.internal,UID:ip-10-0-132-222.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-132-222.ec2.internal,},FirstTimestamp:2026-09-10 19:10:36.086973243 +0000 UTC m=+0.354902547,LastTimestamp:2026-09-10 19:10:36.086973243 +0000 UTC m=+0.354902547,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-222.ec2.internal,}"
Sep 10 19:10:36.094498 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.094484    2546 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-lp6cq"
Sep 10 19:10:36.094638 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.094624    2546 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 10 19:10:36.094695 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.094628    2546 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 10 19:10:36.095373 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:10:36.095353    2546 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-222.ec2.internal\" not found"
Sep 10 19:10:36.095476 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.095433    2546 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 10 19:10:36.095476 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.095449    2546 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 10 19:10:36.095615 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.095550    2546 reconstruct.go:97] "Volume reconstruction finished"
Sep 10 19:10:36.095615 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.095559    2546 reconciler.go:29] "Reconciler: start to sync state"
Sep 10 19:10:36.095615 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.095599    2546 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 10 19:10:36.095751 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.095655    2546 factory.go:55] Registering systemd factory
Sep 10 19:10:36.095751 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.095668    2546 factory.go:223] Registration of the systemd container factory successfully
Sep 10 19:10:36.095911 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.095896    2546 factory.go:153] Registering CRI-O factory
Sep 10 19:10:36.095958 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.095915    2546 factory.go:223] Registration of the crio container factory successfully
Sep 10 19:10:36.095996 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.095958    2546 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 10 19:10:36.095996 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.095973    2546 factory.go:103] Registering Raw factory
Sep 10 19:10:36.095996 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.095984    2546 manager.go:1196] Started watching for new ooms in manager
Sep 10 19:10:36.096334 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.096322    2546 manager.go:349] Starting recovery of all containers
Sep 10 19:10:36.103128 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.103106    2546 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 19:10:36.107230 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:10:36.107207    2546 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-132-222.ec2.internal\" not found" node="ip-10-0-132-222.ec2.internal"
Sep 10 19:10:36.107339 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.107266    2546 manager.go:354] Recovery completed
Sep 10 19:10:36.111782 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.111765    2546 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 19:10:36.113669 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.113648    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-222.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 19:10:36.113733 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.113681    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-222.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 19:10:36.113733 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.113692    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-222.ec2.internal" event="NodeHasSufficientPID"
Sep 10 19:10:36.114221 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.114208    2546 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 10 19:10:36.114281 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.114220    2546 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 10 19:10:36.114281 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.114239    2546 state_mem.go:36] "Initialized new in-memory state store"
Sep 10 19:10:36.114361 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.114223    2546 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 19:10:36.114979 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.114964    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-222.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 19:10:36.115046 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.114990    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-222.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 19:10:36.115046 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.114999    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-222.ec2.internal" event="NodeHasSufficientPID"
Sep 10 19:10:36.115292 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.115277    2546 policy_none.go:49] "None policy: Start"
Sep 10 19:10:36.115292 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.115295    2546 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 10 19:10:36.115383 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.115305    2546 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 10 19:10:36.116228 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.116217    2546 policy_none.go:47] "Start"
Sep 10 19:10:36.141137 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.141114    2546 manager.go:337] "Starting Device Plugin manager"
Sep 10 19:10:36.141237 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:10:36.141162    2546 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 10 19:10:36.141237 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.141175    2546 server.go:87] "Starting device plugin registration server"
Sep 10 19:10:36.141454 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.141442    2546 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 10 19:10:36.141531 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.141457    2546 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 10 19:10:36.141681 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.141606    2546 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 10 19:10:36.141779 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.141683    2546 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 10 19:10:36.141779 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.141692    2546 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 10 19:10:36.142312 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:10:36.142291    2546 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 10 19:10:36.142379 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:10:36.142353    2546 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-222.ec2.internal\" not found"
Sep 10 19:10:36.234113 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.234071    2546 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 10 19:10:36.235275 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.235257    2546 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 10 19:10:36.235380 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.235320    2546 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 10 19:10:36.235380 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.235349    2546 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 10 19:10:36.235504 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:10:36.235396    2546 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 10 19:10:36.239556 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.239539    2546 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 19:10:36.241558 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.241545    2546 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 19:10:36.242429 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.242405    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-222.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 19:10:36.242580 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.242444    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-222.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 19:10:36.242580 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.242455    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-222.ec2.internal" event="NodeHasSufficientPID"
Sep 10 19:10:36.242580 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.242496    2546 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-222.ec2.internal"
Sep 10 19:10:36.257147 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.257120    2546 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-132-222.ec2.internal"
Sep 10 19:10:36.257147 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:10:36.257150    2546 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-222.ec2.internal\": node \"ip-10-0-132-222.ec2.internal\" not found"
Sep 10 19:10:36.290213 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:10:36.290011    2546 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-222.ec2.internal\" not found"
Sep 10 19:10:36.335887 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.335853    2546 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-132-222.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-222.ec2.internal"]
Sep 10 19:10:36.336042 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.335931    2546 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 19:10:36.337556 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.337542    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-222.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 19:10:36.337635 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.337572    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-222.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 19:10:36.337635 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.337586    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-222.ec2.internal" event="NodeHasSufficientPID"
Sep 10 19:10:36.338797 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.338785    2546 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 19:10:36.338925 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.338911    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-222.ec2.internal"
Sep 10 19:10:36.338975 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.338936    2546 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 19:10:36.339579 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.339562    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-222.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 19:10:36.339659 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.339594    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-222.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 19:10:36.339659 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.339600    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-222.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 19:10:36.339659 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.339609    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-222.ec2.internal" event="NodeHasSufficientPID"
Sep 10 19:10:36.339659 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.339617    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-222.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 19:10:36.339659 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.339630    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-222.ec2.internal" event="NodeHasSufficientPID"
Sep 10 19:10:36.340781 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.340765    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-222.ec2.internal"
Sep 10 19:10:36.340826 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.340795    2546 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 19:10:36.341510 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.341493    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-222.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 19:10:36.341589 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.341522    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-222.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 19:10:36.341589 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.341536    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-222.ec2.internal" event="NodeHasSufficientPID"
Sep 10 19:10:36.373163 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:10:36.373142    2546 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-222.ec2.internal\" not found" node="ip-10-0-132-222.ec2.internal"
Sep 10 19:10:36.377562 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:10:36.377546    2546 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-222.ec2.internal\" not found" node="ip-10-0-132-222.ec2.internal"
Sep 10 19:10:36.391237 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:10:36.391215    2546 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-222.ec2.internal\" not found"
Sep 10 19:10:36.397563 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.397544    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3c00c3d431cb7c2cd59dc800908e3a9e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-222.ec2.internal\" (UID: \"3c00c3d431cb7c2cd59dc800908e3a9e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-222.ec2.internal"
Sep 10 19:10:36.397644 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.397575    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/658d2ad482679f9690b7cc4c32ebd70f-config\") pod \"kube-apiserver-proxy-ip-10-0-132-222.ec2.internal\" (UID: \"658d2ad482679f9690b7cc4c32ebd70f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-222.ec2.internal"
Sep 10 19:10:36.397644 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.397601    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3c00c3d431cb7c2cd59dc800908e3a9e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-222.ec2.internal\" (UID: \"3c00c3d431cb7c2cd59dc800908e3a9e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-222.ec2.internal"
Sep 10 19:10:36.492049 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:10:36.492010    2546 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-222.ec2.internal\" not found"
Sep 10 19:10:36.498433 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.498414    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/658d2ad482679f9690b7cc4c32ebd70f-config\") pod \"kube-apiserver-proxy-ip-10-0-132-222.ec2.internal\" (UID: \"658d2ad482679f9690b7cc4c32ebd70f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-222.ec2.internal"
Sep 10 19:10:36.498514 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.498439    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3c00c3d431cb7c2cd59dc800908e3a9e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-222.ec2.internal\" (UID: \"3c00c3d431cb7c2cd59dc800908e3a9e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-222.ec2.internal"
Sep 10 19:10:36.498514 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.498479    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3c00c3d431cb7c2cd59dc800908e3a9e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-222.ec2.internal\" (UID: \"3c00c3d431cb7c2cd59dc800908e3a9e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-222.ec2.internal"
Sep 10 19:10:36.498514 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.498507    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3c00c3d431cb7c2cd59dc800908e3a9e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-222.ec2.internal\" (UID: \"3c00c3d431cb7c2cd59dc800908e3a9e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-222.ec2.internal"
Sep 10 19:10:36.498631 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.498513    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/658d2ad482679f9690b7cc4c32ebd70f-config\") pod \"kube-apiserver-proxy-ip-10-0-132-222.ec2.internal\" (UID: \"658d2ad482679f9690b7cc4c32ebd70f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-222.ec2.internal"
Sep 10 19:10:36.498631 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:36.498530    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3c00c3d431cb7c2cd59dc800908e3a9e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-222.ec2.internal\" (UID: \"3c00c3d431cb7c2cd59dc800908e3a9e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-222.ec2.internal"
Sep 10 19:10:36.592380 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:10:36.592290    2546 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-222.ec2.internal\" not found"
Sep 10 19:10:37.714796 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:37.714752    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-222.ec2.internal"
Sep 10 19:10:37.719527 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:37.719511    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-222.ec2.internal"
Sep 10 19:10:37.732273 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:10:37.732249    2546 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-222.ec2.internal\" not found"
Sep 10 19:10:37.832871 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:10:37.832819    2546 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-222.ec2.internal\" not found"
Sep 10 19:10:37.933392 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:10:37.933297    2546 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-222.ec2.internal\" not found"
Sep 10 19:10:38.015497 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:38.015467    2546 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 19:10:38.033451 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:10:38.033419    2546 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-222.ec2.internal\" not found"
Sep 10 19:10:38.073278 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:38.073242    2546 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 10 19:10:38.073763 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:38.073419    2546 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 10 19:10:38.073763 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:38.073431    2546 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 10 19:10:38.073763 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:38.073449    2546 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 10 19:10:38.134768 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:10:38.134730    2546 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-222.ec2.internal\" not found"
Sep 10 19:10:38.138102 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:38.138064    2546 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-09 19:05:37 +0000 UTC" deadline="2028-02-16 03:48:40.569971321 +0000 UTC"
Sep 10 19:10:38.138102 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:38.138099    2546 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12560h38m2.431875401s"
Sep 10 19:10:38.235616 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:10:38.235534    2546 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-222.ec2.internal\" not found"
Sep 10 19:10:38.266920 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:38.266871    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod658d2ad482679f9690b7cc4c32ebd70f.slice/crio-e62937064c0184f1051e506b56037c4757905d324c01e253560468c2c5f2645d WatchSource:0}: Error finding container e62937064c0184f1051e506b56037c4757905d324c01e253560468c2c5f2645d: Status 404 returned error can't find the container with id e62937064c0184f1051e506b56037c4757905d324c01e253560468c2c5f2645d
Sep 10 19:10:38.267117 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:38.267093    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c00c3d431cb7c2cd59dc800908e3a9e.slice/crio-051fcedaa544bc17af065ebe803076318bcfa2c4528d0e47449cbc3fd614fe5a WatchSource:0}: Error finding container 051fcedaa544bc17af065ebe803076318bcfa2c4528d0e47449cbc3fd614fe5a: Status 404 returned error can't find the container with id 051fcedaa544bc17af065ebe803076318bcfa2c4528d0e47449cbc3fd614fe5a
Sep 10 19:10:38.270930 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:38.270911    2546 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 19:10:38.278465 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:38.278424    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-222.ec2.internal" event={"ID":"658d2ad482679f9690b7cc4c32ebd70f","Type":"ContainerStarted","Data":"e62937064c0184f1051e506b56037c4757905d324c01e253560468c2c5f2645d"}
Sep 10 19:10:38.279366 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:38.279330    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-222.ec2.internal" event={"ID":"3c00c3d431cb7c2cd59dc800908e3a9e","Type":"ContainerStarted","Data":"051fcedaa544bc17af065ebe803076318bcfa2c4528d0e47449cbc3fd614fe5a"}
Sep 10 19:10:38.312353 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:38.312305    2546 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 19:10:38.334350 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:38.334300    2546 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-222.ec2.internal"
Sep 10 19:10:38.344366 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:38.344334    2546 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 10 19:10:38.345415 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:38.345403    2546 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-222.ec2.internal"
Sep 10 19:10:38.360472 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:38.360453    2546 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 10 19:10:39.116281 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.116244    2546 apiserver.go:52] "Watching apiserver"
Sep 10 19:10:39.124416 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.123967    2546 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 10 19:10:39.125363 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.125263    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-222.ec2.internal","openshift-multus/network-metrics-daemon-7pg6j","openshift-network-diagnostics/network-check-target-tjtbz","openshift-network-operator/iptables-alerter-vxn8r","openshift-ovn-kubernetes/ovnkube-node-5k9nq","kube-system/global-pull-secret-syncer-qhq2q","kube-system/konnectivity-agent-l4kqt","kube-system/kube-apiserver-proxy-ip-10-0-132-222.ec2.internal","openshift-cluster-node-tuning-operator/tuned-76h6j","openshift-dns/node-resolver-x4n86","openshift-image-registry/node-ca-bq2kb","openshift-multus/multus-additional-cni-plugins-nhrc7","openshift-multus/multus-q9894","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nfbcl"]
Sep 10 19:10:39.126960 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.126940    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-l4kqt"
Sep 10 19:10:39.128620 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.128109    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-q9894"
Sep 10 19:10:39.129519 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.129488    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tjtbz"
Sep 10 19:10:39.129601 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:10:39.129583    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tjtbz" podUID="3ddbe88b-0b64-4363-9803-cd0abd242bdc"
Sep 10 19:10:39.129962 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.129946    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-f85vw\""
Sep 10 19:10:39.130145 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.130126    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 10 19:10:39.130145 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.130141    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 10 19:10:39.131959 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.131626    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 10 19:10:39.131959 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.131638    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 10 19:10:39.131959 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.131680    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-44sd9\""
Sep 10 19:10:39.131959 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.131812    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 10 19:10:39.131959 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.131848    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 10 19:10:39.132255 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.132239    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-5k9nq"
Sep 10 19:10:39.132867 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.132630    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vxn8r"
Sep 10 19:10:39.133702 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.133684    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qhq2q"
Sep 10 19:10:39.133804 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:10:39.133749    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qhq2q" podUID="63940132-1151-4ec6-b8fd-a68baa948a23"
Sep 10 19:10:39.134656 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.134636    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 10 19:10:39.135037 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.135018    2546 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 10 19:10:39.135965 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.135551    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 10 19:10:39.135965 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.135704    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-76h6j"
Sep 10 19:10:39.136680 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.136394    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-5zgxn\""
Sep 10 19:10:39.136680 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.136457    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 10 19:10:39.136680 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.136604    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 10 19:10:39.140164 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.137926    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nfbcl"
Sep 10 19:10:39.140164 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.137941    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 10 19:10:39.140164 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.137944    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 10 19:10:39.140164 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.138543    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 10 19:10:39.140164 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.138545    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-kg4jk\""
Sep 10 19:10:39.140164 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.139123    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 10 19:10:39.140164 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.139290    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 10 19:10:39.140164 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.139595    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 10 19:10:39.140164 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.139848    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 10 19:10:39.140164 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.140107    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-gwx4d\""
Sep 10 19:10:39.141511 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.141479    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 10 19:10:39.141638 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.141620    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 10 19:10:39.141696 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.141624    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-rz88g\""
Sep 10 19:10:39.141803 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.140680    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 10 19:10:39.142065 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.142049    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7pg6j"
Sep 10 19:10:39.142180 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:10:39.142129    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7pg6j" podUID="50ce76d4-2a8b-48b6-8939-df2ca2f0a802"
Sep 10 19:10:39.143478 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.143208    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/74e0d981-08c3-4d91-9ff8-a5e32fc627e8-host-run-multus-certs\") pod \"multus-q9894\" (UID: \"74e0d981-08c3-4d91-9ff8-a5e32fc627e8\") " pod="openshift-multus/multus-q9894"
Sep 10 19:10:39.143478 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.143247    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/315b4bf2-f3fd-4819-a012-73aa1ec10865-host-slash\") pod \"ovnkube-node-5k9nq\" (UID: \"315b4bf2-f3fd-4819-a012-73aa1ec10865\") " pod="openshift-ovn-kubernetes/ovnkube-node-5k9nq"
Sep 10 19:10:39.143478 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.143273    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/315b4bf2-f3fd-4819-a012-73aa1ec10865-ovn-node-metrics-cert\") pod \"ovnkube-node-5k9nq\" (UID: \"315b4bf2-f3fd-4819-a012-73aa1ec10865\") " pod="openshift-ovn-kubernetes/ovnkube-node-5k9nq"
Sep 10 19:10:39.143478 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.143314    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/315b4bf2-f3fd-4819-a012-73aa1ec10865-ovnkube-script-lib\") pod \"ovnkube-node-5k9nq\" (UID: \"315b4bf2-f3fd-4819-a012-73aa1ec10865\") " pod="openshift-ovn-kubernetes/ovnkube-node-5k9nq"
Sep 10 19:10:39.143478 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.143471    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqpxw\" (UniqueName: \"kubernetes.io/projected/315b4bf2-f3fd-4819-a012-73aa1ec10865-kube-api-access-dqpxw\") pod \"ovnkube-node-5k9nq\" (UID: \"315b4bf2-f3fd-4819-a012-73aa1ec10865\") " pod="openshift-ovn-kubernetes/ovnkube-node-5k9nq"
Sep 10 19:10:39.143781 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.143497    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/591dc1dc-1751-46ee-afd4-ab5a89f8d8ab-etc-modprobe-d\") pod \"tuned-76h6j\" (UID: \"591dc1dc-1751-46ee-afd4-ab5a89f8d8ab\") " pod="openshift-cluster-node-tuning-operator/tuned-76h6j"
Sep 10 19:10:39.143781 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.143517    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/591dc1dc-1751-46ee-afd4-ab5a89f8d8ab-etc-sysctl-d\") pod \"tuned-76h6j\" (UID: \"591dc1dc-1751-46ee-afd4-ab5a89f8d8ab\") " pod="openshift-cluster-node-tuning-operator/tuned-76h6j"
Sep 10 19:10:39.143781 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.143537    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/74e0d981-08c3-4d91-9ff8-a5e32fc627e8-hostroot\") pod \"multus-q9894\" (UID: \"74e0d981-08c3-4d91-9ff8-a5e32fc627e8\") " pod="openshift-multus/multus-q9894"
Sep 10 19:10:39.143781 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.143557    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/315b4bf2-f3fd-4819-a012-73aa1ec10865-etc-openvswitch\") pod \"ovnkube-node-5k9nq\" (UID: \"315b4bf2-f3fd-4819-a012-73aa1ec10865\") " pod="openshift-ovn-kubernetes/ovnkube-node-5k9nq"
Sep 10 19:10:39.143781 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.143584    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/315b4bf2-f3fd-4819-a012-73aa1ec10865-node-log\") pod \"ovnkube-node-5k9nq\" (UID: \"315b4bf2-f3fd-4819-a012-73aa1ec10865\") " pod="openshift-ovn-kubernetes/ovnkube-node-5k9nq"
Sep 10 19:10:39.143781 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.143604    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/74e0d981-08c3-4d91-9ff8-a5e32fc627e8-etc-kubernetes\") pod \"multus-q9894\" (UID: \"74e0d981-08c3-4d91-9ff8-a5e32fc627e8\") " pod="openshift-multus/multus-q9894"
Sep 10 19:10:39.143781 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.143629    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/315b4bf2-f3fd-4819-a012-73aa1ec10865-host-run-netns\") pod \"ovnkube-node-5k9nq\" (UID: \"315b4bf2-f3fd-4819-a012-73aa1ec10865\") " pod="openshift-ovn-kubernetes/ovnkube-node-5k9nq"
Sep 10 19:10:39.143781 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.143650    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/315b4bf2-f3fd-4819-a012-73aa1ec10865-host-cni-netd\") pod \"ovnkube-node-5k9nq\" (UID: \"315b4bf2-f3fd-4819-a012-73aa1ec10865\") " pod="openshift-ovn-kubernetes/ovnkube-node-5k9nq"
Sep 10 19:10:39.143781 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.143680    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/315b4bf2-f3fd-4819-a012-73aa1ec10865-ovnkube-config\") pod \"ovnkube-node-5k9nq\" (UID: \"315b4bf2-f3fd-4819-a012-73aa1ec10865\") " pod="openshift-ovn-kubernetes/ovnkube-node-5k9nq"
Sep 10 19:10:39.143781 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.143706    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/591dc1dc-1751-46ee-afd4-ab5a89f8d8ab-run\") pod \"tuned-76h6j\" (UID: \"591dc1dc-1751-46ee-afd4-ab5a89f8d8ab\") " pod="openshift-cluster-node-tuning-operator/tuned-76h6j"
Sep 10 19:10:39.143781 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.143728    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/591dc1dc-1751-46ee-afd4-ab5a89f8d8ab-tmp\") pod \"tuned-76h6j\" (UID: \"591dc1dc-1751-46ee-afd4-ab5a89f8d8ab\") " pod="openshift-cluster-node-tuning-operator/tuned-76h6j"
Sep 10 19:10:39.143781 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.143750    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/74e0d981-08c3-4d91-9ff8-a5e32fc627e8-host-var-lib-cni-multus\") pod \"multus-q9894\" (UID: \"74e0d981-08c3-4d91-9ff8-a5e32fc627e8\") " pod="openshift-multus/multus-q9894"
Sep 10 19:10:39.143781 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.143773    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0fd21370-c32c-4778-87cb-93ca2dd49ac4-host-slash\") pod \"iptables-alerter-vxn8r\" (UID: \"0fd21370-c32c-4778-87cb-93ca2dd49ac4\") " pod="openshift-network-operator/iptables-alerter-vxn8r"
Sep 10 19:10:39.144406 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.143804    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/591dc1dc-1751-46ee-afd4-ab5a89f8d8ab-etc-tuned\") pod \"tuned-76h6j\" (UID: \"591dc1dc-1751-46ee-afd4-ab5a89f8d8ab\") " pod="openshift-cluster-node-tuning-operator/tuned-76h6j"
Sep 10 19:10:39.144406 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.143837    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/74e0d981-08c3-4d91-9ff8-a5e32fc627e8-host-run-netns\") pod \"multus-q9894\" (UID: \"74e0d981-08c3-4d91-9ff8-a5e32fc627e8\") " pod="openshift-multus/multus-q9894"
Sep 10 19:10:39.144406 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.143860    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/315b4bf2-f3fd-4819-a012-73aa1ec10865-run-openvswitch\") pod \"ovnkube-node-5k9nq\" (UID: \"315b4bf2-f3fd-4819-a012-73aa1ec10865\") " pod="openshift-ovn-kubernetes/ovnkube-node-5k9nq"
Sep 10 19:10:39.144406 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.143895    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/74e0d981-08c3-4d91-9ff8-a5e32fc627e8-system-cni-dir\") pod \"multus-q9894\" (UID: \"74e0d981-08c3-4d91-9ff8-a5e32fc627e8\") " pod="openshift-multus/multus-q9894"
Sep 10 19:10:39.144406 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.143923    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/591dc1dc-1751-46ee-afd4-ab5a89f8d8ab-lib-modules\") pod \"tuned-76h6j\" (UID: \"591dc1dc-1751-46ee-afd4-ab5a89f8d8ab\") " pod="openshift-cluster-node-tuning-operator/tuned-76h6j"
Sep 10 19:10:39.144406 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.143944    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/591dc1dc-1751-46ee-afd4-ab5a89f8d8ab-host\") pod \"tuned-76h6j\" (UID: \"591dc1dc-1751-46ee-afd4-ab5a89f8d8ab\") " pod="openshift-cluster-node-tuning-operator/tuned-76h6j"
Sep 10 19:10:39.144406 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.143965    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqgf4\" (UniqueName: \"kubernetes.io/projected/591dc1dc-1751-46ee-afd4-ab5a89f8d8ab-kube-api-access-qqgf4\") pod \"tuned-76h6j\" (UID: \"591dc1dc-1751-46ee-afd4-ab5a89f8d8ab\") " pod="openshift-cluster-node-tuning-operator/tuned-76h6j"
Sep 10 19:10:39.144406 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.143987    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/639f9409-c20b-4e73-aa74-9859ab69bdaa-konnectivity-ca\") pod \"konnectivity-agent-l4kqt\" (UID: \"639f9409-c20b-4e73-aa74-9859ab69bdaa\") " pod="kube-system/konnectivity-agent-l4kqt"
Sep 10 19:10:39.144406 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.144009    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/63940132-1151-4ec6-b8fd-a68baa948a23-dbus\") pod \"global-pull-secret-syncer-qhq2q\" (UID: \"63940132-1151-4ec6-b8fd-a68baa948a23\") " pod="kube-system/global-pull-secret-syncer-qhq2q"
Sep 10 19:10:39.144406 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.144031    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/74e0d981-08c3-4d91-9ff8-a5e32fc627e8-cni-binary-copy\") pod \"multus-q9894\" (UID: \"74e0d981-08c3-4d91-9ff8-a5e32fc627e8\") " pod="openshift-multus/multus-q9894"
Sep 10 19:10:39.144406 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.144056    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/315b4bf2-f3fd-4819-a012-73aa1ec10865-host-kubelet\") pod \"ovnkube-node-5k9nq\" (UID: \"315b4bf2-f3fd-4819-a012-73aa1ec10865\") " pod="openshift-ovn-kubernetes/ovnkube-node-5k9nq"
Sep 10 19:10:39.144406 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.144077    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/591dc1dc-1751-46ee-afd4-ab5a89f8d8ab-etc-sysctl-conf\") pod \"tuned-76h6j\" (UID: \"591dc1dc-1751-46ee-afd4-ab5a89f8d8ab\") " pod="openshift-cluster-node-tuning-operator/tuned-76h6j"
Sep 10 19:10:39.144406 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.144114    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/591dc1dc-1751-46ee-afd4-ab5a89f8d8ab-etc-systemd\") pod \"tuned-76h6j\" (UID: \"591dc1dc-1751-46ee-afd4-ab5a89f8d8ab\") " pod="openshift-cluster-node-tuning-operator/tuned-76h6j"
Sep 10 19:10:39.144406 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.144155    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/74e0d981-08c3-4d91-9ff8-a5e32fc627e8-multus-cni-dir\") pod \"multus-q9894\" (UID: \"74e0d981-08c3-4d91-9ff8-a5e32fc627e8\") " pod="openshift-multus/multus-q9894"
Sep 10 19:10:39.144406 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.144182    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/74e0d981-08c3-4d91-9ff8-a5e32fc627e8-host-var-lib-kubelet\") pod \"multus-q9894\" (UID: \"74e0d981-08c3-4d91-9ff8-a5e32fc627e8\") " pod="openshift-multus/multus-q9894"
Sep 10 19:10:39.144406 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.144210    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/63940132-1151-4ec6-b8fd-a68baa948a23-kubelet-config\") pod \"global-pull-secret-syncer-qhq2q\" (UID: \"63940132-1151-4ec6-b8fd-a68baa948a23\") " pod="kube-system/global-pull-secret-syncer-qhq2q"
Sep 10 19:10:39.144406 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.144234    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txt2k\" (UniqueName: \"kubernetes.io/projected/0fd21370-c32c-4778-87cb-93ca2dd49ac4-kube-api-access-txt2k\") pod \"iptables-alerter-vxn8r\" (UID: \"0fd21370-c32c-4778-87cb-93ca2dd49ac4\") " pod="openshift-network-operator/iptables-alerter-vxn8r"
Sep 10 19:10:39.145175 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.144265    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/74e0d981-08c3-4d91-9ff8-a5e32fc627e8-host-run-k8s-cni-cncf-io\") pod \"multus-q9894\" (UID: \"74e0d981-08c3-4d91-9ff8-a5e32fc627e8\") " pod="openshift-multus/multus-q9894"
Sep 10 19:10:39.145175 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.144295    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/315b4bf2-f3fd-4819-a012-73aa1ec10865-run-systemd\") pod \"ovnkube-node-5k9nq\" (UID: \"315b4bf2-f3fd-4819-a012-73aa1ec10865\") " pod="openshift-ovn-kubernetes/ovnkube-node-5k9nq"
Sep 10 19:10:39.145175 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.144318    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/315b4bf2-f3fd-4819-a012-73aa1ec10865-host-run-ovn-kubernetes\") pod \"ovnkube-node-5k9nq\" (UID: \"315b4bf2-f3fd-4819-a012-73aa1ec10865\") " pod="openshift-ovn-kubernetes/ovnkube-node-5k9nq"
Sep 10 19:10:39.145175 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.144355    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/591dc1dc-1751-46ee-afd4-ab5a89f8d8ab-etc-sysconfig\") pod \"tuned-76h6j\" (UID: \"591dc1dc-1751-46ee-afd4-ab5a89f8d8ab\") " pod="openshift-cluster-node-tuning-operator/tuned-76h6j"
Sep 10 19:10:39.145175 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.144381    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/74e0d981-08c3-4d91-9ff8-a5e32fc627e8-multus-socket-dir-parent\") pod \"multus-q9894\" (UID: \"74e0d981-08c3-4d91-9ff8-a5e32fc627e8\") " pod="openshift-multus/multus-q9894"
Sep 10 19:10:39.145175 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.144408    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgrcc\" (UniqueName: \"kubernetes.io/projected/74e0d981-08c3-4d91-9ff8-a5e32fc627e8-kube-api-access-mgrcc\") pod \"multus-q9894\" (UID: \"74e0d981-08c3-4d91-9ff8-a5e32fc627e8\") " pod="openshift-multus/multus-q9894"
Sep 10 19:10:39.145175 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.144440    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/315b4bf2-f3fd-4819-a012-73aa1ec10865-systemd-units\") pod \"ovnkube-node-5k9nq\" (UID: \"315b4bf2-f3fd-4819-a012-73aa1ec10865\") " pod="openshift-ovn-kubernetes/ovnkube-node-5k9nq"
Sep 10 19:10:39.145175 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.144475    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/74e0d981-08c3-4d91-9ff8-a5e32fc627e8-cnibin\") pod \"multus-q9894\" (UID: \"74e0d981-08c3-4d91-9ff8-a5e32fc627e8\") " pod="openshift-multus/multus-q9894"
Sep 10 19:10:39.145175 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.144507    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/74e0d981-08c3-4d91-9ff8-a5e32fc627e8-multus-daemon-config\") pod \"multus-q9894\" (UID: \"74e0d981-08c3-4d91-9ff8-a5e32fc627e8\") " pod="openshift-multus/multus-q9894"
Sep 10 19:10:39.145175 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.144535    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxpc6\" (UniqueName: \"kubernetes.io/projected/3ddbe88b-0b64-4363-9803-cd0abd242bdc-kube-api-access-nxpc6\") pod \"network-check-target-tjtbz\" (UID: \"3ddbe88b-0b64-4363-9803-cd0abd242bdc\") " pod="openshift-network-diagnostics/network-check-target-tjtbz"
Sep 10 19:10:39.145175 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.144565    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/315b4bf2-f3fd-4819-a012-73aa1ec10865-env-overrides\") pod \"ovnkube-node-5k9nq\" (UID: \"315b4bf2-f3fd-4819-a012-73aa1ec10865\") " pod="openshift-ovn-kubernetes/ovnkube-node-5k9nq"
Sep 10 19:10:39.145175 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.144586    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/591dc1dc-1751-46ee-afd4-ab5a89f8d8ab-etc-kubernetes\") pod \"tuned-76h6j\" (UID: \"591dc1dc-1751-46ee-afd4-ab5a89f8d8ab\") " pod="openshift-cluster-node-tuning-operator/tuned-76h6j"
Sep 10 19:10:39.145175 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.144608    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0fd21370-c32c-4778-87cb-93ca2dd49ac4-iptables-alerter-script\") pod \"iptables-alerter-vxn8r\" (UID: \"0fd21370-c32c-4778-87cb-93ca2dd49ac4\") " pod="openshift-network-operator/iptables-alerter-vxn8r"
Sep 10 19:10:39.145175 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.144638    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/315b4bf2-f3fd-4819-a012-73aa1ec10865-host-cni-bin\") pod \"ovnkube-node-5k9nq\" (UID: \"315b4bf2-f3fd-4819-a012-73aa1ec10865\") " pod="openshift-ovn-kubernetes/ovnkube-node-5k9nq"
Sep 10 19:10:39.145175 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.144661    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/315b4bf2-f3fd-4819-a012-73aa1ec10865-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5k9nq\" (UID: \"315b4bf2-f3fd-4819-a012-73aa1ec10865\") " pod="openshift-ovn-kubernetes/ovnkube-node-5k9nq"
Sep 10 19:10:39.145175 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.144684    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/591dc1dc-1751-46ee-afd4-ab5a89f8d8ab-sys\") pod \"tuned-76h6j\" (UID: \"591dc1dc-1751-46ee-afd4-ab5a89f8d8ab\") " pod="openshift-cluster-node-tuning-operator/tuned-76h6j"
Sep 10 19:10:39.146006 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.144705    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/74e0d981-08c3-4d91-9ff8-a5e32fc627e8-host-var-lib-cni-bin\") pod \"multus-q9894\" (UID: \"74e0d981-08c3-4d91-9ff8-a5e32fc627e8\") " pod="openshift-multus/multus-q9894"
Sep 10 19:10:39.146006 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.144727    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/63940132-1151-4ec6-b8fd-a68baa948a23-original-pull-secret\") pod \"global-pull-secret-syncer-qhq2q\" (UID: \"63940132-1151-4ec6-b8fd-a68baa948a23\") " pod="kube-system/global-pull-secret-syncer-qhq2q"
Sep 10 19:10:39.146006 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.144748    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/315b4bf2-f3fd-4819-a012-73aa1ec10865-run-ovn\") pod \"ovnkube-node-5k9nq\" (UID: \"315b4bf2-f3fd-4819-a012-73aa1ec10865\") " pod="openshift-ovn-kubernetes/ovnkube-node-5k9nq"
Sep 10 19:10:39.146006 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.144794    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/315b4bf2-f3fd-4819-a012-73aa1ec10865-log-socket\") pod \"ovnkube-node-5k9nq\" (UID: \"315b4bf2-f3fd-4819-a012-73aa1ec10865\") " pod="openshift-ovn-kubernetes/ovnkube-node-5k9nq"
Sep 10 19:10:39.146006 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.144823    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/591dc1dc-1751-46ee-afd4-ab5a89f8d8ab-var-lib-kubelet\") pod \"tuned-76h6j\" (UID: \"591dc1dc-1751-46ee-afd4-ab5a89f8d8ab\") " pod="openshift-cluster-node-tuning-operator/tuned-76h6j"
Sep 10 19:10:39.146006 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.144863    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/315b4bf2-f3fd-4819-a012-73aa1ec10865-var-lib-openvswitch\") pod \"ovnkube-node-5k9nq\" (UID: \"315b4bf2-f3fd-4819-a012-73aa1ec10865\") " pod="openshift-ovn-kubernetes/ovnkube-node-5k9nq"
Sep 10 19:10:39.146006 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.144901    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/639f9409-c20b-4e73-aa74-9859ab69bdaa-agent-certs\") pod \"konnectivity-agent-l4kqt\" (UID: \"639f9409-c20b-4e73-aa74-9859ab69bdaa\") " pod="kube-system/konnectivity-agent-l4kqt"
Sep 10 19:10:39.146006 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.144927    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/74e0d981-08c3-4d91-9ff8-a5e32fc627e8-multus-conf-dir\") pod \"multus-q9894\" (UID: \"74e0d981-08c3-4d91-9ff8-a5e32fc627e8\") " pod="openshift-multus/multus-q9894"
Sep 10 19:10:39.146006 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.145779    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-x4n86"
Sep 10 19:10:39.148276 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.148256    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 10 19:10:39.148415 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.148394    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 10 19:10:39.148484 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.148453    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-n2w68\""
Sep 10 19:10:39.148484 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.148476    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bq2kb"
Sep 10 19:10:39.148704 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.148680    2546 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 10 19:10:39.149490 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.149460    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-nhrc7"
Sep 10 19:10:39.150411 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.150392    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 10 19:10:39.150499 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.150469    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 10 19:10:39.150818 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.150802    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-c7k4l\""
Sep 10 19:10:39.150902 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.150882    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 10 19:10:39.151848 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.151830    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 10 19:10:39.151929 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.151853    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-ckj79\""
Sep 10 19:10:39.152423 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.152404    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 10 19:10:39.167245 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.166958    2546 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 19:10:39.167245 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.167217    2546 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-sx4l7"
Sep 10 19:10:39.174570 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.174546    2546 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-sx4l7"
Sep 10 19:10:39.236163 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.236123    2546 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 10 19:10:39.245542 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.245504    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/315b4bf2-f3fd-4819-a012-73aa1ec10865-run-systemd\") pod \"ovnkube-node-5k9nq\" (UID: \"315b4bf2-f3fd-4819-a012-73aa1ec10865\") " pod="openshift-ovn-kubernetes/ovnkube-node-5k9nq"
Sep 10 19:10:39.245716 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.245548    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/315b4bf2-f3fd-4819-a012-73aa1ec10865-host-run-ovn-kubernetes\") pod \"ovnkube-node-5k9nq\" (UID: \"315b4bf2-f3fd-4819-a012-73aa1ec10865\") " pod="openshift-ovn-kubernetes/ovnkube-node-5k9nq"
Sep 10 19:10:39.245716 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.245578    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/591dc1dc-1751-46ee-afd4-ab5a89f8d8ab-etc-sysconfig\") pod \"tuned-76h6j\" (UID: \"591dc1dc-1751-46ee-afd4-ab5a89f8d8ab\") " pod="openshift-cluster-node-tuning-operator/tuned-76h6j"
Sep 10 19:10:39.245716 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.245602    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/74e0d981-08c3-4d91-9ff8-a5e32fc627e8-multus-socket-dir-parent\") pod \"multus-q9894\" (UID: \"74e0d981-08c3-4d91-9ff8-a5e32fc627e8\") " pod="openshift-multus/multus-q9894"
Sep 10 19:10:39.245716 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.245615    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/315b4bf2-f3fd-4819-a012-73aa1ec10865-run-systemd\") pod \"ovnkube-node-5k9nq\" (UID: \"315b4bf2-f3fd-4819-a012-73aa1ec10865\") " pod="openshift-ovn-kubernetes/ovnkube-node-5k9nq"
Sep 10 19:10:39.245716 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.245626    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgrcc\" (UniqueName: \"kubernetes.io/projected/74e0d981-08c3-4d91-9ff8-a5e32fc627e8-kube-api-access-mgrcc\") pod \"multus-q9894\" (UID: \"74e0d981-08c3-4d91-9ff8-a5e32fc627e8\") " pod="openshift-multus/multus-q9894"
Sep 10 19:10:39.245716 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.245656    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggmqx\" (UniqueName: \"kubernetes.io/projected/50ce76d4-2a8b-48b6-8939-df2ca2f0a802-kube-api-access-ggmqx\") pod \"network-metrics-daemon-7pg6j\" (UID: \"50ce76d4-2a8b-48b6-8939-df2ca2f0a802\") " pod="openshift-multus/network-metrics-daemon-7pg6j"
Sep 10 19:10:39.245716 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.245681    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/591dc1dc-1751-46ee-afd4-ab5a89f8d8ab-etc-sysconfig\") pod \"tuned-76h6j\" (UID: \"591dc1dc-1751-46ee-afd4-ab5a89f8d8ab\") " pod="openshift-cluster-node-tuning-operator/tuned-76h6j"
Sep 10 19:10:39.245716 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.245691    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/97e5ac31-cbf0-464c-b59c-9708fa487b4d-system-cni-dir\") pod \"multus-additional-cni-plugins-nhrc7\" (UID: \"97e5ac31-cbf0-464c-b59c-9708fa487b4d\") " pod="openshift-multus/multus-additional-cni-plugins-nhrc7"
Sep 10 19:10:39.245716 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.245712    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/315b4bf2-f3fd-4819-a012-73aa1ec10865-host-run-ovn-kubernetes\") pod \"ovnkube-node-5k9nq\" (UID: \"315b4bf2-f3fd-4819-a012-73aa1ec10865\") " pod="openshift-ovn-kubernetes/ovnkube-node-5k9nq"
Sep 10 19:10:39.245716 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.245713    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/97e5ac31-cbf0-464c-b59c-9708fa487b4d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nhrc7\" (UID: \"97e5ac31-cbf0-464c-b59c-9708fa487b4d\") " pod="openshift-multus/multus-additional-cni-plugins-nhrc7"
Sep 10 19:10:39.246211 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.245742    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/315b4bf2-f3fd-4819-a012-73aa1ec10865-systemd-units\") pod \"ovnkube-node-5k9nq\" (UID: \"315b4bf2-f3fd-4819-a012-73aa1ec10865\") " pod="openshift-ovn-kubernetes/ovnkube-node-5k9nq"
Sep 10 19:10:39.246211 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.245766    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/74e0d981-08c3-4d91-9ff8-a5e32fc627e8-cnibin\") pod \"multus-q9894\" (UID: \"74e0d981-08c3-4d91-9ff8-a5e32fc627e8\") " pod="openshift-multus/multus-q9894"
Sep 10 19:10:39.246211 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.245783    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/74e0d981-08c3-4d91-9ff8-a5e32fc627e8-multus-daemon-config\") pod \"multus-q9894\" (UID: \"74e0d981-08c3-4d91-9ff8-a5e32fc627e8\") " pod="openshift-multus/multus-q9894"
Sep 10 19:10:39.246211 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.245799    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxpc6\" (UniqueName: \"kubernetes.io/projected/3ddbe88b-0b64-4363-9803-cd0abd242bdc-kube-api-access-nxpc6\") pod \"network-check-target-tjtbz\" (UID: \"3ddbe88b-0b64-4363-9803-cd0abd242bdc\") " pod="openshift-network-diagnostics/network-check-target-tjtbz"
Sep 10 19:10:39.246211 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.245817    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/315b4bf2-f3fd-4819-a012-73aa1ec10865-env-overrides\") pod \"ovnkube-node-5k9nq\" (UID: \"315b4bf2-f3fd-4819-a012-73aa1ec10865\") " pod="openshift-ovn-kubernetes/ovnkube-node-5k9nq"
Sep 10 19:10:39.246211 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.245832    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/74e0d981-08c3-4d91-9ff8-a5e32fc627e8-multus-socket-dir-parent\") pod \"multus-q9894\" (UID: \"74e0d981-08c3-4d91-9ff8-a5e32fc627e8\") " pod="openshift-multus/multus-q9894"
Sep 10 19:10:39.246211 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.245836    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/591dc1dc-1751-46ee-afd4-ab5a89f8d8ab-etc-kubernetes\") pod \"tuned-76h6j\" (UID: \"591dc1dc-1751-46ee-afd4-ab5a89f8d8ab\") " pod="openshift-cluster-node-tuning-operator/tuned-76h6j"
Sep 10 19:10:39.246211 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.245858    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/97e5ac31-cbf0-464c-b59c-9708fa487b4d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nhrc7\" (UID: \"97e5ac31-cbf0-464c-b59c-9708fa487b4d\") " pod="openshift-multus/multus-additional-cni-plugins-nhrc7"
Sep 10 19:10:39.246211 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.245885    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82gtz\" (UniqueName: \"kubernetes.io/projected/97e5ac31-cbf0-464c-b59c-9708fa487b4d-kube-api-access-82gtz\") pod \"multus-additional-cni-plugins-nhrc7\" (UID: \"97e5ac31-cbf0-464c-b59c-9708fa487b4d\") " pod="openshift-multus/multus-additional-cni-plugins-nhrc7"
Sep 10 19:10:39.246211 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.245902    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rh2j4\" (UniqueName: \"kubernetes.io/projected/808e9bad-e9c5-4316-b4b6-e967cd0e3ef9-kube-api-access-rh2j4\") pod \"node-resolver-x4n86\" (UID: \"808e9bad-e9c5-4316-b4b6-e967cd0e3ef9\") " pod="openshift-dns/node-resolver-x4n86"
Sep 10 19:10:39.246211 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.245926    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0fd21370-c32c-4778-87cb-93ca2dd49ac4-iptables-alerter-script\") pod \"iptables-alerter-vxn8r\" (UID: \"0fd21370-c32c-4778-87cb-93ca2dd49ac4\") " pod="openshift-network-operator/iptables-alerter-vxn8r"
Sep 10 19:10:39.246211 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.245942    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/315b4bf2-f3fd-4819-a012-73aa1ec10865-host-cni-bin\") pod \"ovnkube-node-5k9nq\" (UID: \"315b4bf2-f3fd-4819-a012-73aa1ec10865\") " pod="openshift-ovn-kubernetes/ovnkube-node-5k9nq"
Sep 10 19:10:39.246211 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.245958    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/315b4bf2-f3fd-4819-a012-73aa1ec10865-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5k9nq\" (UID: \"315b4bf2-f3fd-4819-a012-73aa1ec10865\") " pod="openshift-ovn-kubernetes/ovnkube-node-5k9nq"
Sep 10 19:10:39.246211 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.245973    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/591dc1dc-1751-46ee-afd4-ab5a89f8d8ab-sys\") pod \"tuned-76h6j\" (UID: \"591dc1dc-1751-46ee-afd4-ab5a89f8d8ab\") " pod="openshift-cluster-node-tuning-operator/tuned-76h6j"
Sep 10 19:10:39.246211 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.245991    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/74e0d981-08c3-4d91-9ff8-a5e32fc627e8-host-var-lib-cni-bin\") pod \"multus-q9894\" (UID: \"74e0d981-08c3-4d91-9ff8-a5e32fc627e8\") " pod="openshift-multus/multus-q9894"
Sep 10 19:10:39.246211 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.246010    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/808e9bad-e9c5-4316-b4b6-e967cd0e3ef9-tmp-dir\") pod \"node-resolver-x4n86\" (UID: \"808e9bad-e9c5-4316-b4b6-e967cd0e3ef9\") " pod="openshift-dns/node-resolver-x4n86"
Sep 10 19:10:39.246211 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.246025    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/63940132-1151-4ec6-b8fd-a68baa948a23-original-pull-secret\") pod \"global-pull-secret-syncer-qhq2q\" (UID: \"63940132-1151-4ec6-b8fd-a68baa948a23\") " pod="kube-system/global-pull-secret-syncer-qhq2q"
Sep 10 19:10:39.247047 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.246040    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/315b4bf2-f3fd-4819-a012-73aa1ec10865-run-ovn\") pod \"ovnkube-node-5k9nq\" (UID: \"315b4bf2-f3fd-4819-a012-73aa1ec10865\") " pod="openshift-ovn-kubernetes/ovnkube-node-5k9nq"
Sep 10 19:10:39.247047 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.246054    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/315b4bf2-f3fd-4819-a012-73aa1ec10865-log-socket\") pod \"ovnkube-node-5k9nq\" (UID: \"315b4bf2-f3fd-4819-a012-73aa1ec10865\") " pod="openshift-ovn-kubernetes/ovnkube-node-5k9nq"
Sep 10 19:10:39.247047 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.246075    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/591dc1dc-1751-46ee-afd4-ab5a89f8d8ab-var-lib-kubelet\") pod \"tuned-76h6j\" (UID: \"591dc1dc-1751-46ee-afd4-ab5a89f8d8ab\") " pod="openshift-cluster-node-tuning-operator/tuned-76h6j"
Sep 10 19:10:39.247047 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.246099    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bfe72c48-1d12-4ebc-9a54-6032f900cb2e-device-dir\") pod \"aws-ebs-csi-driver-node-nfbcl\" (UID: \"bfe72c48-1d12-4ebc-9a54-6032f900cb2e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nfbcl"
Sep 10 19:10:39.247047 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.246143    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8fdl\" (UniqueName: \"kubernetes.io/projected/bfe72c48-1d12-4ebc-9a54-6032f900cb2e-kube-api-access-r8fdl\") pod \"aws-ebs-csi-driver-node-nfbcl\" (UID: \"bfe72c48-1d12-4ebc-9a54-6032f900cb2e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nfbcl"
Sep 10 19:10:39.247047 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.246159    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/315b4bf2-f3fd-4819-a012-73aa1ec10865-systemd-units\") pod \"ovnkube-node-5k9nq\" (UID: \"315b4bf2-f3fd-4819-a012-73aa1ec10865\") " pod="openshift-ovn-kubernetes/ovnkube-node-5k9nq"
Sep 10 19:10:39.247047 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.246175    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/808e9bad-e9c5-4316-b4b6-e967cd0e3ef9-hosts-file\") pod \"node-resolver-x4n86\" (UID: \"808e9bad-e9c5-4316-b4b6-e967cd0e3ef9\") " pod="openshift-dns/node-resolver-x4n86"
Sep 10 19:10:39.247047 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.246192    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vxrg\" (UniqueName: \"kubernetes.io/projected/20594470-2f86-4aab-88b9-20cbdae08a97-kube-api-access-5vxrg\") pod \"node-ca-bq2kb\" (UID: \"20594470-2f86-4aab-88b9-20cbdae08a97\") " pod="openshift-image-registry/node-ca-bq2kb"
Sep 10 19:10:39.247047 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.246210    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/315b4bf2-f3fd-4819-a012-73aa1ec10865-var-lib-openvswitch\") pod \"ovnkube-node-5k9nq\" (UID: \"315b4bf2-f3fd-4819-a012-73aa1ec10865\") " pod="openshift-ovn-kubernetes/ovnkube-node-5k9nq"
Sep 10 19:10:39.247047 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.246211    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/74e0d981-08c3-4d91-9ff8-a5e32fc627e8-cnibin\") pod \"multus-q9894\" (UID: \"74e0d981-08c3-4d91-9ff8-a5e32fc627e8\") " pod="openshift-multus/multus-q9894"
Sep 10 19:10:39.247047 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.246233    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/639f9409-c20b-4e73-aa74-9859ab69bdaa-agent-certs\") pod \"konnectivity-agent-l4kqt\" (UID: \"639f9409-c20b-4e73-aa74-9859ab69bdaa\") " pod="kube-system/konnectivity-agent-l4kqt"
Sep 10 19:10:39.247047 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.246269    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/74e0d981-08c3-4d91-9ff8-a5e32fc627e8-multus-conf-dir\") pod \"multus-q9894\" (UID: \"74e0d981-08c3-4d91-9ff8-a5e32fc627e8\") " pod="openshift-multus/multus-q9894"
Sep 10 19:10:39.247047 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.246286    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/74e0d981-08c3-4d91-9ff8-a5e32fc627e8-host-run-multus-certs\") pod \"multus-q9894\" (UID: \"74e0d981-08c3-4d91-9ff8-a5e32fc627e8\") " pod="openshift-multus/multus-q9894"
Sep 10 19:10:39.247047 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.246304    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bfe72c48-1d12-4ebc-9a54-6032f900cb2e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nfbcl\" (UID: \"bfe72c48-1d12-4ebc-9a54-6032f900cb2e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nfbcl"
Sep 10 19:10:39.247047 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.246322    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/315b4bf2-f3fd-4819-a012-73aa1ec10865-host-slash\") pod \"ovnkube-node-5k9nq\" (UID: \"315b4bf2-f3fd-4819-a012-73aa1ec10865\") " pod="openshift-ovn-kubernetes/ovnkube-node-5k9nq"
Sep 10 19:10:39.247047 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.246349    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/315b4bf2-f3fd-4819-a012-73aa1ec10865-ovn-node-metrics-cert\") pod \"ovnkube-node-5k9nq\" (UID: \"315b4bf2-f3fd-4819-a012-73aa1ec10865\") " pod="openshift-ovn-kubernetes/ovnkube-node-5k9nq"
Sep 10 19:10:39.247047 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.246367    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/315b4bf2-f3fd-4819-a012-73aa1ec10865-ovnkube-script-lib\") pod \"ovnkube-node-5k9nq\" (UID: \"315b4bf2-f3fd-4819-a012-73aa1ec10865\") " pod="openshift-ovn-kubernetes/ovnkube-node-5k9nq"
Sep 10 19:10:39.247887 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.246611    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqpxw\" (UniqueName: \"kubernetes.io/projected/315b4bf2-f3fd-4819-a012-73aa1ec10865-kube-api-access-dqpxw\") pod \"ovnkube-node-5k9nq\" (UID: \"315b4bf2-f3fd-4819-a012-73aa1ec10865\") " pod="openshift-ovn-kubernetes/ovnkube-node-5k9nq"
Sep 10 19:10:39.247887 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.246644    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/591dc1dc-1751-46ee-afd4-ab5a89f8d8ab-etc-modprobe-d\") pod \"tuned-76h6j\" (UID: \"591dc1dc-1751-46ee-afd4-ab5a89f8d8ab\") " pod="openshift-cluster-node-tuning-operator/tuned-76h6j"
Sep 10 19:10:39.247887 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.246653    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/591dc1dc-1751-46ee-afd4-ab5a89f8d8ab-var-lib-kubelet\") pod \"tuned-76h6j\" (UID: \"591dc1dc-1751-46ee-afd4-ab5a89f8d8ab\") " pod="openshift-cluster-node-tuning-operator/tuned-76h6j"
Sep 10 19:10:39.247887 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.246666    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/315b4bf2-f3fd-4819-a012-73aa1ec10865-log-socket\") pod \"ovnkube-node-5k9nq\" (UID: \"315b4bf2-f3fd-4819-a012-73aa1ec10865\") " pod="openshift-ovn-kubernetes/ovnkube-node-5k9nq"
Sep 10 19:10:39.247887 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.246670    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/591dc1dc-1751-46ee-afd4-ab5a89f8d8ab-etc-sysctl-d\") pod \"tuned-76h6j\" (UID: \"591dc1dc-1751-46ee-afd4-ab5a89f8d8ab\") " pod="openshift-cluster-node-tuning-operator/tuned-76h6j"
Sep 10 19:10:39.247887 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.246699    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/591dc1dc-1751-46ee-afd4-ab5a89f8d8ab-etc-kubernetes\") pod \"tuned-76h6j\" (UID: \"591dc1dc-1751-46ee-afd4-ab5a89f8d8ab\") " pod="openshift-cluster-node-tuning-operator/tuned-76h6j"
Sep 10 19:10:39.247887 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.246710    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/74e0d981-08c3-4d91-9ff8-a5e32fc627e8-hostroot\") pod \"multus-q9894\" (UID: \"74e0d981-08c3-4d91-9ff8-a5e32fc627e8\") " pod="openshift-multus/multus-q9894"
Sep 10 19:10:39.247887 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.246728    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/74e0d981-08c3-4d91-9ff8-a5e32fc627e8-host-var-lib-cni-bin\") pod \"multus-q9894\" (UID: \"74e0d981-08c3-4d91-9ff8-a5e32fc627e8\") " pod="openshift-multus/multus-q9894"
Sep 10 19:10:39.247887 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.246737    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/315b4bf2-f3fd-4819-a012-73aa1ec10865-etc-openvswitch\") pod \"ovnkube-node-5k9nq\" (UID: \"315b4bf2-f3fd-4819-a012-73aa1ec10865\") " pod="openshift-ovn-kubernetes/ovnkube-node-5k9nq"
Sep 10 19:10:39.247887 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.246766    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/315b4bf2-f3fd-4819-a012-73aa1ec10865-node-log\") pod \"ovnkube-node-5k9nq\" (UID: \"315b4bf2-f3fd-4819-a012-73aa1ec10865\") " pod="openshift-ovn-kubernetes/ovnkube-node-5k9nq"
Sep 10 19:10:39.247887 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.246794    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/591dc1dc-1751-46ee-afd4-ab5a89f8d8ab-etc-sysctl-d\") pod \"tuned-76h6j\" (UID: \"591dc1dc-1751-46ee-afd4-ab5a89f8d8ab\") " pod="openshift-cluster-node-tuning-operator/tuned-76h6j"
Sep 10 19:10:39.247887 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.246795    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/74e0d981-08c3-4d91-9ff8-a5e32fc627e8-etc-kubernetes\") pod \"multus-q9894\" (UID: \"74e0d981-08c3-4d91-9ff8-a5e32fc627e8\") " pod="openshift-multus/multus-q9894"
Sep 10 19:10:39.247887 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:10:39.246814    2546 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 19:10:39.247887 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.246832    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bfe72c48-1d12-4ebc-9a54-6032f900cb2e-registration-dir\") pod \"aws-ebs-csi-driver-node-nfbcl\" (UID: \"bfe72c48-1d12-4ebc-9a54-6032f900cb2e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nfbcl"
Sep 10 19:10:39.247887 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.246868    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/315b4bf2-f3fd-4819-a012-73aa1ec10865-var-lib-openvswitch\") pod \"ovnkube-node-5k9nq\" (UID: \"315b4bf2-f3fd-4819-a012-73aa1ec10865\") " pod="openshift-ovn-kubernetes/ovnkube-node-5k9nq"
Sep 10 19:10:39.247887 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:10:39.246884    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63940132-1151-4ec6-b8fd-a68baa948a23-original-pull-secret podName:63940132-1151-4ec6-b8fd-a68baa948a23 nodeName:}" failed. No retries permitted until 2026-09-10 19:10:39.746861614 +0000 UTC m=+2.975911638 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/63940132-1151-4ec6-b8fd-a68baa948a23-original-pull-secret") pod "global-pull-secret-syncer-qhq2q" (UID: "63940132-1151-4ec6-b8fd-a68baa948a23") : object "kube-system"/"original-pull-secret" not registered
Sep 10 19:10:39.247887 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.246832    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/74e0d981-08c3-4d91-9ff8-a5e32fc627e8-etc-kubernetes\") pod \"multus-q9894\" (UID: \"74e0d981-08c3-4d91-9ff8-a5e32fc627e8\") " pod="openshift-multus/multus-q9894"
Sep 10 19:10:39.247887 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.247157    2546 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 10 19:10:39.248781 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.247212    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/315b4bf2-f3fd-4819-a012-73aa1ec10865-ovnkube-script-lib\") pod \"ovnkube-node-5k9nq\" (UID: \"315b4bf2-f3fd-4819-a012-73aa1ec10865\") " pod="openshift-ovn-kubernetes/ovnkube-node-5k9nq"
Sep 10 19:10:39.248781 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.246857    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bfe72c48-1d12-4ebc-9a54-6032f900cb2e-sys-fs\") pod \"aws-ebs-csi-driver-node-nfbcl\" (UID: \"bfe72c48-1d12-4ebc-9a54-6032f900cb2e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nfbcl"
Sep 10 19:10:39.248781 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.247301    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/315b4bf2-f3fd-4819-a012-73aa1ec10865-node-log\") pod \"ovnkube-node-5k9nq\" (UID: \"315b4bf2-f3fd-4819-a012-73aa1ec10865\") " pod="openshift-ovn-kubernetes/ovnkube-node-5k9nq"
Sep 10 19:10:39.248781 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.247332    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/97e5ac31-cbf0-464c-b59c-9708fa487b4d-cni-binary-copy\") pod \"multus-additional-cni-plugins-nhrc7\" (UID: \"97e5ac31-cbf0-464c-b59c-9708fa487b4d\") " pod="openshift-multus/multus-additional-cni-plugins-nhrc7"
Sep 10 19:10:39.248781 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.247360    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/315b4bf2-f3fd-4819-a012-73aa1ec10865-host-slash\") pod \"ovnkube-node-5k9nq\" (UID: \"315b4bf2-f3fd-4819-a012-73aa1ec10865\") " pod="openshift-ovn-kubernetes/ovnkube-node-5k9nq"
Sep 10 19:10:39.248781 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.247368    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/74e0d981-08c3-4d91-9ff8-a5e32fc627e8-multus-conf-dir\") pod \"multus-q9894\" (UID: \"74e0d981-08c3-4d91-9ff8-a5e32fc627e8\") " pod="openshift-multus/multus-q9894"
Sep 10 19:10:39.248781 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.247297    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/74e0d981-08c3-4d91-9ff8-a5e32fc627e8-hostroot\") pod \"multus-q9894\" (UID: \"74e0d981-08c3-4d91-9ff8-a5e32fc627e8\") " pod="openshift-multus/multus-q9894"
Sep 10 19:10:39.248781 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.247388    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/315b4bf2-f3fd-4819-a012-73aa1ec10865-host-run-netns\") pod \"ovnkube-node-5k9nq\" (UID: \"315b4bf2-f3fd-4819-a012-73aa1ec10865\") " pod="openshift-ovn-kubernetes/ovnkube-node-5k9nq"
Sep 10 19:10:39.248781 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.247409    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/315b4bf2-f3fd-4819-a012-73aa1ec10865-etc-openvswitch\") pod \"ovnkube-node-5k9nq\" (UID: \"315b4bf2-f3fd-4819-a012-73aa1ec10865\") " pod="openshift-ovn-kubernetes/ovnkube-node-5k9nq"
Sep 10 19:10:39.248781 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.247438    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/591dc1dc-1751-46ee-afd4-ab5a89f8d8ab-sys\") pod \"tuned-76h6j\" (UID: \"591dc1dc-1751-46ee-afd4-ab5a89f8d8ab\") " pod="openshift-cluster-node-tuning-operator/tuned-76h6j"
Sep 10 19:10:39.248781 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.247317    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/591dc1dc-1751-46ee-afd4-ab5a89f8d8ab-etc-modprobe-d\") pod \"tuned-76h6j\" (UID: \"591dc1dc-1751-46ee-afd4-ab5a89f8d8ab\") " pod="openshift-cluster-node-tuning-operator/tuned-76h6j"
Sep 10 19:10:39.248781 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.247648    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/74e0d981-08c3-4d91-9ff8-a5e32fc627e8-host-run-multus-certs\") pod \"multus-q9894\" (UID: \"74e0d981-08c3-4d91-9ff8-a5e32fc627e8\") " pod="openshift-multus/multus-q9894"
Sep 10 19:10:39.248781 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.247695    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/315b4bf2-f3fd-4819-a012-73aa1ec10865-run-ovn\") pod \"ovnkube-node-5k9nq\" (UID: \"315b4bf2-f3fd-4819-a012-73aa1ec10865\") " pod="openshift-ovn-kubernetes/ovnkube-node-5k9nq"
Sep 10 19:10:39.248781 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.247708    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/315b4bf2-f3fd-4819-a012-73aa1ec10865-host-run-netns\") pod \"ovnkube-node-5k9nq\" (UID: \"315b4bf2-f3fd-4819-a012-73aa1ec10865\") " pod="openshift-ovn-kubernetes/ovnkube-node-5k9nq"
Sep 10 19:10:39.248781 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.247702    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/315b4bf2-f3fd-4819-a012-73aa1ec10865-host-cni-netd\") pod \"ovnkube-node-5k9nq\" (UID: \"315b4bf2-f3fd-4819-a012-73aa1ec10865\") " pod="openshift-ovn-kubernetes/ovnkube-node-5k9nq"
Sep 10 19:10:39.248781 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.247729    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/315b4bf2-f3fd-4819-a012-73aa1ec10865-host-cni-bin\") pod \"ovnkube-node-5k9nq\" (UID: \"315b4bf2-f3fd-4819-a012-73aa1ec10865\") " pod="openshift-ovn-kubernetes/ovnkube-node-5k9nq"
Sep 10 19:10:39.248781 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.247751    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/315b4bf2-f3fd-4819-a012-73aa1ec10865-host-cni-netd\") pod \"ovnkube-node-5k9nq\" (UID: \"315b4bf2-f3fd-4819-a012-73aa1ec10865\") " pod="openshift-ovn-kubernetes/ovnkube-node-5k9nq"
Sep 10 19:10:39.248781 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.247753    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/315b4bf2-f3fd-4819-a012-73aa1ec10865-env-overrides\") pod \"ovnkube-node-5k9nq\" (UID: \"315b4bf2-f3fd-4819-a012-73aa1ec10865\") " pod="openshift-ovn-kubernetes/ovnkube-node-5k9nq"
Sep 10 19:10:39.249434 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.247805    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/315b4bf2-f3fd-4819-a012-73aa1ec10865-ovnkube-config\") pod \"ovnkube-node-5k9nq\" (UID: \"315b4bf2-f3fd-4819-a012-73aa1ec10865\") " pod="openshift-ovn-kubernetes/ovnkube-node-5k9nq"
Sep 10 19:10:39.249434 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.247832    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/591dc1dc-1751-46ee-afd4-ab5a89f8d8ab-run\") pod \"tuned-76h6j\" (UID: \"591dc1dc-1751-46ee-afd4-ab5a89f8d8ab\") " pod="openshift-cluster-node-tuning-operator/tuned-76h6j"
Sep 10 19:10:39.249434 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.247848    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/315b4bf2-f3fd-4819-a012-73aa1ec10865-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5k9nq\" (UID: \"315b4bf2-f3fd-4819-a012-73aa1ec10865\") " pod="openshift-ovn-kubernetes/ovnkube-node-5k9nq"
Sep 10 19:10:39.249434 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.247870    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/591dc1dc-1751-46ee-afd4-ab5a89f8d8ab-tmp\") pod \"tuned-76h6j\" (UID: \"591dc1dc-1751-46ee-afd4-ab5a89f8d8ab\") " pod="openshift-cluster-node-tuning-operator/tuned-76h6j"
Sep 10 19:10:39.249434 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.247931    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/74e0d981-08c3-4d91-9ff8-a5e32fc627e8-host-var-lib-cni-multus\") pod \"multus-q9894\" (UID: \"74e0d981-08c3-4d91-9ff8-a5e32fc627e8\") " pod="openshift-multus/multus-q9894"
Sep 10 19:10:39.249434 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.247959    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/97e5ac31-cbf0-464c-b59c-9708fa487b4d-cnibin\") pod \"multus-additional-cni-plugins-nhrc7\" (UID: \"97e5ac31-cbf0-464c-b59c-9708fa487b4d\") " pod="openshift-multus/multus-additional-cni-plugins-nhrc7"
Sep 10 19:10:39.249434 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.247986    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0fd21370-c32c-4778-87cb-93ca2dd49ac4-host-slash\") pod \"iptables-alerter-vxn8r\" (UID: \"0fd21370-c32c-4778-87cb-93ca2dd49ac4\") " pod="openshift-network-operator/iptables-alerter-vxn8r"
Sep 10 19:10:39.249434 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.248025    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/591dc1dc-1751-46ee-afd4-ab5a89f8d8ab-etc-tuned\") pod \"tuned-76h6j\" (UID: \"591dc1dc-1751-46ee-afd4-ab5a89f8d8ab\") " pod="openshift-cluster-node-tuning-operator/tuned-76h6j"
Sep 10 19:10:39.249434 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.248050    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/74e0d981-08c3-4d91-9ff8-a5e32fc627e8-host-run-netns\") pod \"multus-q9894\" (UID: \"74e0d981-08c3-4d91-9ff8-a5e32fc627e8\") " pod="openshift-multus/multus-q9894"
Sep 10 19:10:39.249434 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.248074    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/315b4bf2-f3fd-4819-a012-73aa1ec10865-run-openvswitch\") pod \"ovnkube-node-5k9nq\" (UID: \"315b4bf2-f3fd-4819-a012-73aa1ec10865\") " pod="openshift-ovn-kubernetes/ovnkube-node-5k9nq"
Sep 10 19:10:39.249434 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.248100    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/74e0d981-08c3-4d91-9ff8-a5e32fc627e8-system-cni-dir\") pod \"multus-q9894\" (UID: \"74e0d981-08c3-4d91-9ff8-a5e32fc627e8\") " pod="openshift-multus/multus-q9894"
Sep 10 19:10:39.249434 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.248102    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/74e0d981-08c3-4d91-9ff8-a5e32fc627e8-multus-daemon-config\") pod \"multus-q9894\" (UID: \"74e0d981-08c3-4d91-9ff8-a5e32fc627e8\") " pod="openshift-multus/multus-q9894"
Sep 10 19:10:39.249434 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.248127    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/591dc1dc-1751-46ee-afd4-ab5a89f8d8ab-lib-modules\") pod \"tuned-76h6j\" (UID: \"591dc1dc-1751-46ee-afd4-ab5a89f8d8ab\") " pod="openshift-cluster-node-tuning-operator/tuned-76h6j"
Sep 10 19:10:39.249434 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.248151    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/591dc1dc-1751-46ee-afd4-ab5a89f8d8ab-host\") pod \"tuned-76h6j\" (UID: \"591dc1dc-1751-46ee-afd4-ab5a89f8d8ab\") " pod="openshift-cluster-node-tuning-operator/tuned-76h6j"
Sep 10 19:10:39.249434 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.248161    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0fd21370-c32c-4778-87cb-93ca2dd49ac4-host-slash\") pod \"iptables-alerter-vxn8r\" (UID: \"0fd21370-c32c-4778-87cb-93ca2dd49ac4\") " pod="openshift-network-operator/iptables-alerter-vxn8r"
Sep 10 19:10:39.249434 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.248177    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqgf4\" (UniqueName: \"kubernetes.io/projected/591dc1dc-1751-46ee-afd4-ab5a89f8d8ab-kube-api-access-qqgf4\") pod \"tuned-76h6j\" (UID: \"591dc1dc-1751-46ee-afd4-ab5a89f8d8ab\") " pod="openshift-cluster-node-tuning-operator/tuned-76h6j"
Sep 10 19:10:39.249434 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.248317    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/315b4bf2-f3fd-4819-a012-73aa1ec10865-ovnkube-config\") pod \"ovnkube-node-5k9nq\" (UID: \"315b4bf2-f3fd-4819-a012-73aa1ec10865\") " pod="openshift-ovn-kubernetes/ovnkube-node-5k9nq"
Sep 10 19:10:39.250106 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.248363    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/639f9409-c20b-4e73-aa74-9859ab69bdaa-konnectivity-ca\") pod \"konnectivity-agent-l4kqt\" (UID: \"639f9409-c20b-4e73-aa74-9859ab69bdaa\") " pod="kube-system/konnectivity-agent-l4kqt"
Sep 10 19:10:39.250106 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.248384    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0fd21370-c32c-4778-87cb-93ca2dd49ac4-iptables-alerter-script\") pod \"iptables-alerter-vxn8r\" (UID: \"0fd21370-c32c-4778-87cb-93ca2dd49ac4\") " pod="openshift-network-operator/iptables-alerter-vxn8r"
Sep 10 19:10:39.250106 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.248392    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/63940132-1151-4ec6-b8fd-a68baa948a23-dbus\") pod \"global-pull-secret-syncer-qhq2q\" (UID: \"63940132-1151-4ec6-b8fd-a68baa948a23\") " pod="kube-system/global-pull-secret-syncer-qhq2q"
Sep 10 19:10:39.250106 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.248415    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/74e0d981-08c3-4d91-9ff8-a5e32fc627e8-system-cni-dir\") pod \"multus-q9894\" (UID: \"74e0d981-08c3-4d91-9ff8-a5e32fc627e8\") " pod="openshift-multus/multus-q9894"
Sep 10 19:10:39.250106 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.248419    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/74e0d981-08c3-4d91-9ff8-a5e32fc627e8-cni-binary-copy\") pod \"multus-q9894\" (UID: \"74e0d981-08c3-4d91-9ff8-a5e32fc627e8\") " pod="openshift-multus/multus-q9894"
Sep 10 19:10:39.250106 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.248459    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bfe72c48-1d12-4ebc-9a54-6032f900cb2e-etc-selinux\") pod \"aws-ebs-csi-driver-node-nfbcl\" (UID: \"bfe72c48-1d12-4ebc-9a54-6032f900cb2e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nfbcl"
Sep 10 19:10:39.250106 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.248484    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/20594470-2f86-4aab-88b9-20cbdae08a97-host\") pod \"node-ca-bq2kb\" (UID: \"20594470-2f86-4aab-88b9-20cbdae08a97\") " pod="openshift-image-registry/node-ca-bq2kb"
Sep 10 19:10:39.250106 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.248512    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/20594470-2f86-4aab-88b9-20cbdae08a97-serviceca\") pod \"node-ca-bq2kb\" (UID: \"20594470-2f86-4aab-88b9-20cbdae08a97\") " pod="openshift-image-registry/node-ca-bq2kb"
Sep 10 19:10:39.250106 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.248537    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/97e5ac31-cbf0-464c-b59c-9708fa487b4d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nhrc7\" (UID: \"97e5ac31-cbf0-464c-b59c-9708fa487b4d\") " pod="openshift-multus/multus-additional-cni-plugins-nhrc7"
Sep 10 19:10:39.250106 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.248583    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/315b4bf2-f3fd-4819-a012-73aa1ec10865-host-kubelet\") pod \"ovnkube-node-5k9nq\" (UID: \"315b4bf2-f3fd-4819-a012-73aa1ec10865\") " pod="openshift-ovn-kubernetes/ovnkube-node-5k9nq"
Sep 10 19:10:39.250106 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.248610    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/591dc1dc-1751-46ee-afd4-ab5a89f8d8ab-etc-sysctl-conf\") pod \"tuned-76h6j\" (UID: \"591dc1dc-1751-46ee-afd4-ab5a89f8d8ab\") " pod="openshift-cluster-node-tuning-operator/tuned-76h6j"
Sep 10 19:10:39.250106 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.248632    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/591dc1dc-1751-46ee-afd4-ab5a89f8d8ab-etc-systemd\") pod \"tuned-76h6j\" (UID: \"591dc1dc-1751-46ee-afd4-ab5a89f8d8ab\") " pod="openshift-cluster-node-tuning-operator/tuned-76h6j"
Sep 10 19:10:39.250106 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.248654    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/74e0d981-08c3-4d91-9ff8-a5e32fc627e8-multus-cni-dir\") pod \"multus-q9894\" (UID: \"74e0d981-08c3-4d91-9ff8-a5e32fc627e8\") " pod="openshift-multus/multus-q9894"
Sep 10 19:10:39.250106 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.248675    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/74e0d981-08c3-4d91-9ff8-a5e32fc627e8-host-var-lib-kubelet\") pod \"multus-q9894\" (UID: \"74e0d981-08c3-4d91-9ff8-a5e32fc627e8\") " pod="openshift-multus/multus-q9894"
Sep 10 19:10:39.250106 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.248697    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bfe72c48-1d12-4ebc-9a54-6032f900cb2e-socket-dir\") pod \"aws-ebs-csi-driver-node-nfbcl\" (UID: \"bfe72c48-1d12-4ebc-9a54-6032f900cb2e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nfbcl"
Sep 10 19:10:39.250106 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.248728    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/63940132-1151-4ec6-b8fd-a68baa948a23-kubelet-config\") pod \"global-pull-secret-syncer-qhq2q\" (UID: \"63940132-1151-4ec6-b8fd-a68baa948a23\") " pod="kube-system/global-pull-secret-syncer-qhq2q"
Sep 10 19:10:39.250106 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.248749    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txt2k\" (UniqueName: \"kubernetes.io/projected/0fd21370-c32c-4778-87cb-93ca2dd49ac4-kube-api-access-txt2k\") pod \"iptables-alerter-vxn8r\" (UID: \"0fd21370-c32c-4778-87cb-93ca2dd49ac4\") " pod="openshift-network-operator/iptables-alerter-vxn8r"
Sep 10 19:10:39.250934 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.248771    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/74e0d981-08c3-4d91-9ff8-a5e32fc627e8-host-run-k8s-cni-cncf-io\") pod \"multus-q9894\" (UID: \"74e0d981-08c3-4d91-9ff8-a5e32fc627e8\") " pod="openshift-multus/multus-q9894"
Sep 10 19:10:39.250934 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.248793    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50ce76d4-2a8b-48b6-8939-df2ca2f0a802-metrics-certs\") pod \"network-metrics-daemon-7pg6j\" (UID: \"50ce76d4-2a8b-48b6-8939-df2ca2f0a802\") " pod="openshift-multus/network-metrics-daemon-7pg6j"
Sep 10 19:10:39.250934 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.248868    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/74e0d981-08c3-4d91-9ff8-a5e32fc627e8-cni-binary-copy\") pod \"multus-q9894\" (UID: \"74e0d981-08c3-4d91-9ff8-a5e32fc627e8\") " pod="openshift-multus/multus-q9894"
Sep 10 19:10:39.250934 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.248878    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/74e0d981-08c3-4d91-9ff8-a5e32fc627e8-host-run-netns\") pod \"multus-q9894\" (UID: \"74e0d981-08c3-4d91-9ff8-a5e32fc627e8\") " pod="openshift-multus/multus-q9894"
Sep 10 19:10:39.250934 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.248920    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/591dc1dc-1751-46ee-afd4-ab5a89f8d8ab-etc-systemd\") pod \"tuned-76h6j\" (UID: \"591dc1dc-1751-46ee-afd4-ab5a89f8d8ab\") " pod="openshift-cluster-node-tuning-operator/tuned-76h6j"
Sep 10 19:10:39.250934 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.248921    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/74e0d981-08c3-4d91-9ff8-a5e32fc627e8-host-var-lib-cni-multus\") pod \"multus-q9894\" (UID: \"74e0d981-08c3-4d91-9ff8-a5e32fc627e8\") " pod="openshift-multus/multus-q9894"
Sep 10 19:10:39.250934 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.248919    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/315b4bf2-f3fd-4819-a012-73aa1ec10865-run-openvswitch\") pod \"ovnkube-node-5k9nq\" (UID: \"315b4bf2-f3fd-4819-a012-73aa1ec10865\") " pod="openshift-ovn-kubernetes/ovnkube-node-5k9nq"
Sep 10 19:10:39.250934 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.248986    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/591dc1dc-1751-46ee-afd4-ab5a89f8d8ab-run\") pod \"tuned-76h6j\" (UID: \"591dc1dc-1751-46ee-afd4-ab5a89f8d8ab\") " pod="openshift-cluster-node-tuning-operator/tuned-76h6j"
Sep 10 19:10:39.250934 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.249023    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/591dc1dc-1751-46ee-afd4-ab5a89f8d8ab-lib-modules\") pod \"tuned-76h6j\" (UID: \"591dc1dc-1751-46ee-afd4-ab5a89f8d8ab\") " pod="openshift-cluster-node-tuning-operator/tuned-76h6j"
Sep 10 19:10:39.250934 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.249054    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/591dc1dc-1751-46ee-afd4-ab5a89f8d8ab-host\") pod \"tuned-76h6j\" (UID: \"591dc1dc-1751-46ee-afd4-ab5a89f8d8ab\") " pod="openshift-cluster-node-tuning-operator/tuned-76h6j"
Sep 10 19:10:39.250934 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.249086    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/74e0d981-08c3-4d91-9ff8-a5e32fc627e8-multus-cni-dir\") pod \"multus-q9894\" (UID: \"74e0d981-08c3-4d91-9ff8-a5e32fc627e8\") " pod="openshift-multus/multus-q9894"
Sep 10 19:10:39.250934 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.249119    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/63940132-1151-4ec6-b8fd-a68baa948a23-dbus\") pod \"global-pull-secret-syncer-qhq2q\" (UID: \"63940132-1151-4ec6-b8fd-a68baa948a23\") " pod="kube-system/global-pull-secret-syncer-qhq2q"
Sep 10 19:10:39.250934 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.249125    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/315b4bf2-f3fd-4819-a012-73aa1ec10865-host-kubelet\") pod \"ovnkube-node-5k9nq\" (UID: \"315b4bf2-f3fd-4819-a012-73aa1ec10865\") " pod="openshift-ovn-kubernetes/ovnkube-node-5k9nq"
Sep 10 19:10:39.250934 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.249157    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/74e0d981-08c3-4d91-9ff8-a5e32fc627e8-host-var-lib-kubelet\") pod \"multus-q9894\" (UID: \"74e0d981-08c3-4d91-9ff8-a5e32fc627e8\") " pod="openshift-multus/multus-q9894"
Sep 10 19:10:39.250934 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.249167    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/63940132-1151-4ec6-b8fd-a68baa948a23-kubelet-config\") pod \"global-pull-secret-syncer-qhq2q\" (UID: \"63940132-1151-4ec6-b8fd-a68baa948a23\") " pod="kube-system/global-pull-secret-syncer-qhq2q"
Sep 10 19:10:39.250934 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.249197    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/74e0d981-08c3-4d91-9ff8-a5e32fc627e8-host-run-k8s-cni-cncf-io\") pod \"multus-q9894\" (UID: \"74e0d981-08c3-4d91-9ff8-a5e32fc627e8\") " pod="openshift-multus/multus-q9894"
Sep 10 19:10:39.250934 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.249522    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/591dc1dc-1751-46ee-afd4-ab5a89f8d8ab-etc-sysctl-conf\") pod \"tuned-76h6j\" (UID: \"591dc1dc-1751-46ee-afd4-ab5a89f8d8ab\") " pod="openshift-cluster-node-tuning-operator/tuned-76h6j"
Sep 10 19:10:39.250934 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.249754    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/639f9409-c20b-4e73-aa74-9859ab69bdaa-konnectivity-ca\") pod \"konnectivity-agent-l4kqt\" (UID: \"639f9409-c20b-4e73-aa74-9859ab69bdaa\") " pod="kube-system/konnectivity-agent-l4kqt"
Sep 10 19:10:39.254462 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.252135    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/639f9409-c20b-4e73-aa74-9859ab69bdaa-agent-certs\") pod \"konnectivity-agent-l4kqt\" (UID: \"639f9409-c20b-4e73-aa74-9859ab69bdaa\") " pod="kube-system/konnectivity-agent-l4kqt"
Sep 10 19:10:39.254808 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.254666    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/315b4bf2-f3fd-4819-a012-73aa1ec10865-ovn-node-metrics-cert\") pod \"ovnkube-node-5k9nq\" (UID: \"315b4bf2-f3fd-4819-a012-73aa1ec10865\") " pod="openshift-ovn-kubernetes/ovnkube-node-5k9nq"
Sep 10 19:10:39.254913 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:10:39.254807    2546 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 19:10:39.254913 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:10:39.254833    2546 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 19:10:39.254913 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:10:39.254844    2546 projected.go:196] Error preparing data for projected volume kube-api-access-nxpc6 for pod openshift-network-diagnostics/network-check-target-tjtbz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 19:10:39.254913 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:10:39.254898    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3ddbe88b-0b64-4363-9803-cd0abd242bdc-kube-api-access-nxpc6 podName:3ddbe88b-0b64-4363-9803-cd0abd242bdc nodeName:}" failed. No retries permitted until 2026-09-10 19:10:39.7548813 +0000 UTC m=+2.983931303 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-nxpc6" (UniqueName: "kubernetes.io/projected/3ddbe88b-0b64-4363-9803-cd0abd242bdc-kube-api-access-nxpc6") pod "network-check-target-tjtbz" (UID: "3ddbe88b-0b64-4363-9803-cd0abd242bdc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 19:10:39.255122 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.254955    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/591dc1dc-1751-46ee-afd4-ab5a89f8d8ab-tmp\") pod \"tuned-76h6j\" (UID: \"591dc1dc-1751-46ee-afd4-ab5a89f8d8ab\") " pod="openshift-cluster-node-tuning-operator/tuned-76h6j"
Sep 10 19:10:39.255122 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.255049    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/591dc1dc-1751-46ee-afd4-ab5a89f8d8ab-etc-tuned\") pod \"tuned-76h6j\" (UID: \"591dc1dc-1751-46ee-afd4-ab5a89f8d8ab\") " pod="openshift-cluster-node-tuning-operator/tuned-76h6j"
Sep 10 19:10:39.257522 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.257477    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgrcc\" (UniqueName: \"kubernetes.io/projected/74e0d981-08c3-4d91-9ff8-a5e32fc627e8-kube-api-access-mgrcc\") pod \"multus-q9894\" (UID: \"74e0d981-08c3-4d91-9ff8-a5e32fc627e8\") " pod="openshift-multus/multus-q9894"
Sep 10 19:10:39.259764 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.259742    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqgf4\" (UniqueName: \"kubernetes.io/projected/591dc1dc-1751-46ee-afd4-ab5a89f8d8ab-kube-api-access-qqgf4\") pod \"tuned-76h6j\" (UID: \"591dc1dc-1751-46ee-afd4-ab5a89f8d8ab\") " pod="openshift-cluster-node-tuning-operator/tuned-76h6j"
Sep 10 19:10:39.266213 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.266184    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqpxw\" (UniqueName: \"kubernetes.io/projected/315b4bf2-f3fd-4819-a012-73aa1ec10865-kube-api-access-dqpxw\") pod \"ovnkube-node-5k9nq\" (UID: \"315b4bf2-f3fd-4819-a012-73aa1ec10865\") " pod="openshift-ovn-kubernetes/ovnkube-node-5k9nq"
Sep 10 19:10:39.268441 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.268416    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-txt2k\" (UniqueName: \"kubernetes.io/projected/0fd21370-c32c-4778-87cb-93ca2dd49ac4-kube-api-access-txt2k\") pod \"iptables-alerter-vxn8r\" (UID: \"0fd21370-c32c-4778-87cb-93ca2dd49ac4\") " pod="openshift-network-operator/iptables-alerter-vxn8r"
Sep 10 19:10:39.349219 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.349186    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/97e5ac31-cbf0-464c-b59c-9708fa487b4d-cnibin\") pod \"multus-additional-cni-plugins-nhrc7\" (UID: \"97e5ac31-cbf0-464c-b59c-9708fa487b4d\") " pod="openshift-multus/multus-additional-cni-plugins-nhrc7"
Sep 10 19:10:39.349219 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.349225    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bfe72c48-1d12-4ebc-9a54-6032f900cb2e-etc-selinux\") pod \"aws-ebs-csi-driver-node-nfbcl\" (UID: \"bfe72c48-1d12-4ebc-9a54-6032f900cb2e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nfbcl"
Sep 10 19:10:39.349468 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.349248    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/20594470-2f86-4aab-88b9-20cbdae08a97-host\") pod \"node-ca-bq2kb\" (UID: \"20594470-2f86-4aab-88b9-20cbdae08a97\") " pod="openshift-image-registry/node-ca-bq2kb"
Sep 10 19:10:39.349468 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.349294    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/20594470-2f86-4aab-88b9-20cbdae08a97-host\") pod \"node-ca-bq2kb\" (UID: \"20594470-2f86-4aab-88b9-20cbdae08a97\") " pod="openshift-image-registry/node-ca-bq2kb"
Sep 10 19:10:39.349468 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.349313    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/97e5ac31-cbf0-464c-b59c-9708fa487b4d-cnibin\") pod \"multus-additional-cni-plugins-nhrc7\" (UID: \"97e5ac31-cbf0-464c-b59c-9708fa487b4d\") " pod="openshift-multus/multus-additional-cni-plugins-nhrc7"
Sep 10 19:10:39.349468 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.349360    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/20594470-2f86-4aab-88b9-20cbdae08a97-serviceca\") pod \"node-ca-bq2kb\" (UID: \"20594470-2f86-4aab-88b9-20cbdae08a97\") " pod="openshift-image-registry/node-ca-bq2kb"
Sep 10 19:10:39.349468 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.349378    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/97e5ac31-cbf0-464c-b59c-9708fa487b4d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nhrc7\" (UID: \"97e5ac31-cbf0-464c-b59c-9708fa487b4d\") " pod="openshift-multus/multus-additional-cni-plugins-nhrc7"
Sep 10 19:10:39.349468 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.349393    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bfe72c48-1d12-4ebc-9a54-6032f900cb2e-etc-selinux\") pod \"aws-ebs-csi-driver-node-nfbcl\" (UID: \"bfe72c48-1d12-4ebc-9a54-6032f900cb2e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nfbcl"
Sep 10 19:10:39.349468 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.349402    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bfe72c48-1d12-4ebc-9a54-6032f900cb2e-socket-dir\") pod \"aws-ebs-csi-driver-node-nfbcl\" (UID: \"bfe72c48-1d12-4ebc-9a54-6032f900cb2e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nfbcl"
Sep 10 19:10:39.349468 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.349448    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50ce76d4-2a8b-48b6-8939-df2ca2f0a802-metrics-certs\") pod \"network-metrics-daemon-7pg6j\" (UID: \"50ce76d4-2a8b-48b6-8939-df2ca2f0a802\") " pod="openshift-multus/network-metrics-daemon-7pg6j"
Sep 10 19:10:39.349848 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.349478    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggmqx\" (UniqueName: \"kubernetes.io/projected/50ce76d4-2a8b-48b6-8939-df2ca2f0a802-kube-api-access-ggmqx\") pod \"network-metrics-daemon-7pg6j\" (UID: \"50ce76d4-2a8b-48b6-8939-df2ca2f0a802\") " pod="openshift-multus/network-metrics-daemon-7pg6j"
Sep 10 19:10:39.349848 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.349505    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/97e5ac31-cbf0-464c-b59c-9708fa487b4d-system-cni-dir\") pod \"multus-additional-cni-plugins-nhrc7\" (UID: \"97e5ac31-cbf0-464c-b59c-9708fa487b4d\") " pod="openshift-multus/multus-additional-cni-plugins-nhrc7"
Sep 10 19:10:39.349848 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.349531    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/97e5ac31-cbf0-464c-b59c-9708fa487b4d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nhrc7\" (UID: \"97e5ac31-cbf0-464c-b59c-9708fa487b4d\") " pod="openshift-multus/multus-additional-cni-plugins-nhrc7"
Sep 10 19:10:39.349848 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.349535    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bfe72c48-1d12-4ebc-9a54-6032f900cb2e-socket-dir\") pod \"aws-ebs-csi-driver-node-nfbcl\" (UID: \"bfe72c48-1d12-4ebc-9a54-6032f900cb2e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nfbcl"
Sep 10 19:10:39.349848 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.349568    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/97e5ac31-cbf0-464c-b59c-9708fa487b4d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nhrc7\" (UID: \"97e5ac31-cbf0-464c-b59c-9708fa487b4d\") " pod="openshift-multus/multus-additional-cni-plugins-nhrc7"
Sep 10 19:10:39.349848 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.349613    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/97e5ac31-cbf0-464c-b59c-9708fa487b4d-system-cni-dir\") pod \"multus-additional-cni-plugins-nhrc7\" (UID: \"97e5ac31-cbf0-464c-b59c-9708fa487b4d\") " pod="openshift-multus/multus-additional-cni-plugins-nhrc7"
Sep 10 19:10:39.349848 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:10:39.349613    2546 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 19:10:39.349848 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.349653    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/97e5ac31-cbf0-464c-b59c-9708fa487b4d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nhrc7\" (UID: \"97e5ac31-cbf0-464c-b59c-9708fa487b4d\") " pod="openshift-multus/multus-additional-cni-plugins-nhrc7"
Sep 10 19:10:39.349848 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:10:39.349670    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50ce76d4-2a8b-48b6-8939-df2ca2f0a802-metrics-certs podName:50ce76d4-2a8b-48b6-8939-df2ca2f0a802 nodeName:}" failed. No retries permitted until 2026-09-10 19:10:39.849651585 +0000 UTC m=+3.078701602 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/50ce76d4-2a8b-48b6-8939-df2ca2f0a802-metrics-certs") pod "network-metrics-daemon-7pg6j" (UID: "50ce76d4-2a8b-48b6-8939-df2ca2f0a802") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 19:10:39.349848 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.349694    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82gtz\" (UniqueName: \"kubernetes.io/projected/97e5ac31-cbf0-464c-b59c-9708fa487b4d-kube-api-access-82gtz\") pod \"multus-additional-cni-plugins-nhrc7\" (UID: \"97e5ac31-cbf0-464c-b59c-9708fa487b4d\") " pod="openshift-multus/multus-additional-cni-plugins-nhrc7"
Sep 10 19:10:39.349848 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.349718    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rh2j4\" (UniqueName: \"kubernetes.io/projected/808e9bad-e9c5-4316-b4b6-e967cd0e3ef9-kube-api-access-rh2j4\") pod \"node-resolver-x4n86\" (UID: \"808e9bad-e9c5-4316-b4b6-e967cd0e3ef9\") " pod="openshift-dns/node-resolver-x4n86"
Sep 10 19:10:39.349848 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.349746    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/808e9bad-e9c5-4316-b4b6-e967cd0e3ef9-tmp-dir\") pod \"node-resolver-x4n86\" (UID: \"808e9bad-e9c5-4316-b4b6-e967cd0e3ef9\") " pod="openshift-dns/node-resolver-x4n86"
Sep 10 19:10:39.349848 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.349781    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bfe72c48-1d12-4ebc-9a54-6032f900cb2e-device-dir\") pod \"aws-ebs-csi-driver-node-nfbcl\" (UID: \"bfe72c48-1d12-4ebc-9a54-6032f900cb2e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nfbcl"
Sep 10 19:10:39.349848 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.349804    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8fdl\" (UniqueName: \"kubernetes.io/projected/bfe72c48-1d12-4ebc-9a54-6032f900cb2e-kube-api-access-r8fdl\") pod \"aws-ebs-csi-driver-node-nfbcl\" (UID: \"bfe72c48-1d12-4ebc-9a54-6032f900cb2e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nfbcl"
Sep 10 19:10:39.349848 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.349828    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/808e9bad-e9c5-4316-b4b6-e967cd0e3ef9-hosts-file\") pod \"node-resolver-x4n86\" (UID: \"808e9bad-e9c5-4316-b4b6-e967cd0e3ef9\") " pod="openshift-dns/node-resolver-x4n86"
Sep 10 19:10:39.349848 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.349833    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/20594470-2f86-4aab-88b9-20cbdae08a97-serviceca\") pod \"node-ca-bq2kb\" (UID: \"20594470-2f86-4aab-88b9-20cbdae08a97\") " pod="openshift-image-registry/node-ca-bq2kb"
Sep 10 19:10:39.350704 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.349849    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vxrg\" (UniqueName: \"kubernetes.io/projected/20594470-2f86-4aab-88b9-20cbdae08a97-kube-api-access-5vxrg\") pod \"node-ca-bq2kb\" (UID: \"20594470-2f86-4aab-88b9-20cbdae08a97\") " pod="openshift-image-registry/node-ca-bq2kb"
Sep 10 19:10:39.350704 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.349877    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bfe72c48-1d12-4ebc-9a54-6032f900cb2e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nfbcl\" (UID: \"bfe72c48-1d12-4ebc-9a54-6032f900cb2e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nfbcl"
Sep 10 19:10:39.350704 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.349947    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bfe72c48-1d12-4ebc-9a54-6032f900cb2e-registration-dir\") pod \"aws-ebs-csi-driver-node-nfbcl\" (UID: \"bfe72c48-1d12-4ebc-9a54-6032f900cb2e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nfbcl"
Sep 10 19:10:39.350704 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.349969    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bfe72c48-1d12-4ebc-9a54-6032f900cb2e-sys-fs\") pod \"aws-ebs-csi-driver-node-nfbcl\" (UID: \"bfe72c48-1d12-4ebc-9a54-6032f900cb2e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nfbcl"
Sep 10 19:10:39.350704 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.349993    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/97e5ac31-cbf0-464c-b59c-9708fa487b4d-cni-binary-copy\") pod \"multus-additional-cni-plugins-nhrc7\" (UID: \"97e5ac31-cbf0-464c-b59c-9708fa487b4d\") " pod="openshift-multus/multus-additional-cni-plugins-nhrc7"
Sep 10 19:10:39.350704 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.350037    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/808e9bad-e9c5-4316-b4b6-e967cd0e3ef9-tmp-dir\") pod \"node-resolver-x4n86\" (UID: \"808e9bad-e9c5-4316-b4b6-e967cd0e3ef9\") " pod="openshift-dns/node-resolver-x4n86"
Sep 10 19:10:39.350704 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.350194    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/97e5ac31-cbf0-464c-b59c-9708fa487b4d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nhrc7\" (UID: \"97e5ac31-cbf0-464c-b59c-9708fa487b4d\") " pod="openshift-multus/multus-additional-cni-plugins-nhrc7"
Sep 10 19:10:39.350704 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.350237    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/97e5ac31-cbf0-464c-b59c-9708fa487b4d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nhrc7\" (UID: \"97e5ac31-cbf0-464c-b59c-9708fa487b4d\") " pod="openshift-multus/multus-additional-cni-plugins-nhrc7"
Sep 10 19:10:39.350704 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.350426    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/808e9bad-e9c5-4316-b4b6-e967cd0e3ef9-hosts-file\") pod \"node-resolver-x4n86\" (UID: \"808e9bad-e9c5-4316-b4b6-e967cd0e3ef9\") " pod="openshift-dns/node-resolver-x4n86"
Sep 10 19:10:39.350704 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.350454    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bfe72c48-1d12-4ebc-9a54-6032f900cb2e-device-dir\") pod \"aws-ebs-csi-driver-node-nfbcl\" (UID: \"bfe72c48-1d12-4ebc-9a54-6032f900cb2e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nfbcl"
Sep 10 19:10:39.350704 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.350468    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/97e5ac31-cbf0-464c-b59c-9708fa487b4d-cni-binary-copy\") pod \"multus-additional-cni-plugins-nhrc7\" (UID: \"97e5ac31-cbf0-464c-b59c-9708fa487b4d\") " pod="openshift-multus/multus-additional-cni-plugins-nhrc7"
Sep 10 19:10:39.350704 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.350533    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bfe72c48-1d12-4ebc-9a54-6032f900cb2e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nfbcl\" (UID: \"bfe72c48-1d12-4ebc-9a54-6032f900cb2e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nfbcl"
Sep 10 19:10:39.350704 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.350532    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bfe72c48-1d12-4ebc-9a54-6032f900cb2e-registration-dir\") pod \"aws-ebs-csi-driver-node-nfbcl\" (UID: \"bfe72c48-1d12-4ebc-9a54-6032f900cb2e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nfbcl"
Sep 10 19:10:39.350704 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.350535    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bfe72c48-1d12-4ebc-9a54-6032f900cb2e-sys-fs\") pod \"aws-ebs-csi-driver-node-nfbcl\" (UID: \"bfe72c48-1d12-4ebc-9a54-6032f900cb2e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nfbcl"
Sep 10 19:10:39.363074 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.363033    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-82gtz\" (UniqueName: \"kubernetes.io/projected/97e5ac31-cbf0-464c-b59c-9708fa487b4d-kube-api-access-82gtz\") pod \"multus-additional-cni-plugins-nhrc7\" (UID: \"97e5ac31-cbf0-464c-b59c-9708fa487b4d\") " pod="openshift-multus/multus-additional-cni-plugins-nhrc7"
Sep 10 19:10:39.363325 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.363298    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vxrg\" (UniqueName: \"kubernetes.io/projected/20594470-2f86-4aab-88b9-20cbdae08a97-kube-api-access-5vxrg\") pod \"node-ca-bq2kb\" (UID: \"20594470-2f86-4aab-88b9-20cbdae08a97\") " pod="openshift-image-registry/node-ca-bq2kb"
Sep 10 19:10:39.364292 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.364274    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggmqx\" (UniqueName: \"kubernetes.io/projected/50ce76d4-2a8b-48b6-8939-df2ca2f0a802-kube-api-access-ggmqx\") pod \"network-metrics-daemon-7pg6j\" (UID: \"50ce76d4-2a8b-48b6-8939-df2ca2f0a802\") " pod="openshift-multus/network-metrics-daemon-7pg6j"
Sep 10 19:10:39.365353 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.365314    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rh2j4\" (UniqueName: \"kubernetes.io/projected/808e9bad-e9c5-4316-b4b6-e967cd0e3ef9-kube-api-access-rh2j4\") pod \"node-resolver-x4n86\" (UID: \"808e9bad-e9c5-4316-b4b6-e967cd0e3ef9\") " pod="openshift-dns/node-resolver-x4n86"
Sep 10 19:10:39.365353 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.365328    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8fdl\" (UniqueName: \"kubernetes.io/projected/bfe72c48-1d12-4ebc-9a54-6032f900cb2e-kube-api-access-r8fdl\") pod \"aws-ebs-csi-driver-node-nfbcl\" (UID: \"bfe72c48-1d12-4ebc-9a54-6032f900cb2e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nfbcl"
Sep 10 19:10:39.441791 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.441707    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-l4kqt"
Sep 10 19:10:39.450590 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.450563    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-q9894"
Sep 10 19:10:39.461210 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.461187    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-5k9nq"
Sep 10 19:10:39.468755 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.468730    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vxn8r"
Sep 10 19:10:39.477196 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.477175    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-76h6j"
Sep 10 19:10:39.486701 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.486678    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nfbcl"
Sep 10 19:10:39.495205 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.495182    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-x4n86"
Sep 10 19:10:39.505871 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.505852    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bq2kb"
Sep 10 19:10:39.510553 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.510536    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-nhrc7"
Sep 10 19:10:39.548636 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.548601    2546 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 19:10:39.752224 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.752134    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/63940132-1151-4ec6-b8fd-a68baa948a23-original-pull-secret\") pod \"global-pull-secret-syncer-qhq2q\" (UID: \"63940132-1151-4ec6-b8fd-a68baa948a23\") " pod="kube-system/global-pull-secret-syncer-qhq2q"
Sep 10 19:10:39.752408 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:10:39.752267    2546 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 19:10:39.752408 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:10:39.752323    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63940132-1151-4ec6-b8fd-a68baa948a23-original-pull-secret podName:63940132-1151-4ec6-b8fd-a68baa948a23 nodeName:}" failed. No retries permitted until 2026-09-10 19:10:40.752306659 +0000 UTC m=+3.981356662 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/63940132-1151-4ec6-b8fd-a68baa948a23-original-pull-secret") pod "global-pull-secret-syncer-qhq2q" (UID: "63940132-1151-4ec6-b8fd-a68baa948a23") : object "kube-system"/"original-pull-secret" not registered
Sep 10 19:10:39.853143 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.853102    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50ce76d4-2a8b-48b6-8939-df2ca2f0a802-metrics-certs\") pod \"network-metrics-daemon-7pg6j\" (UID: \"50ce76d4-2a8b-48b6-8939-df2ca2f0a802\") " pod="openshift-multus/network-metrics-daemon-7pg6j"
Sep 10 19:10:39.853368 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:39.853162    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxpc6\" (UniqueName: \"kubernetes.io/projected/3ddbe88b-0b64-4363-9803-cd0abd242bdc-kube-api-access-nxpc6\") pod \"network-check-target-tjtbz\" (UID: \"3ddbe88b-0b64-4363-9803-cd0abd242bdc\") " pod="openshift-network-diagnostics/network-check-target-tjtbz"
Sep 10 19:10:39.853368 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:10:39.853285    2546 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 19:10:39.853515 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:10:39.853378    2546 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 19:10:39.853515 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:10:39.853399    2546 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 19:10:39.853515 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:10:39.853414    2546 projected.go:196] Error preparing data for projected volume kube-api-access-nxpc6 for pod openshift-network-diagnostics/network-check-target-tjtbz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 19:10:39.853515 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:10:39.853399    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50ce76d4-2a8b-48b6-8939-df2ca2f0a802-metrics-certs podName:50ce76d4-2a8b-48b6-8939-df2ca2f0a802 nodeName:}" failed. No retries permitted until 2026-09-10 19:10:40.853376974 +0000 UTC m=+4.082427000 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/50ce76d4-2a8b-48b6-8939-df2ca2f0a802-metrics-certs") pod "network-metrics-daemon-7pg6j" (UID: "50ce76d4-2a8b-48b6-8939-df2ca2f0a802") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 19:10:39.853515 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:10:39.853505    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3ddbe88b-0b64-4363-9803-cd0abd242bdc-kube-api-access-nxpc6 podName:3ddbe88b-0b64-4363-9803-cd0abd242bdc nodeName:}" failed. No retries permitted until 2026-09-10 19:10:40.853471863 +0000 UTC m=+4.082521879 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-nxpc6" (UniqueName: "kubernetes.io/projected/3ddbe88b-0b64-4363-9803-cd0abd242bdc-kube-api-access-nxpc6") pod "network-check-target-tjtbz" (UID: "3ddbe88b-0b64-4363-9803-cd0abd242bdc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 19:10:40.176555 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:40.176469    2546 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 19:05:39 +0000 UTC" deadline="2028-04-30 18:18:06.135205709 +0000 UTC"
Sep 10 19:10:40.176555 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:40.176499    2546 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14351h7m25.95871069s"
Sep 10 19:10:40.221369 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:40.221329    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97e5ac31_cbf0_464c_b59c_9708fa487b4d.slice/crio-cd83195c7df1f092a112ba5f4c8789e8a07509c6bbeea8e2dc55c51bff7d1821 WatchSource:0}: Error finding container cd83195c7df1f092a112ba5f4c8789e8a07509c6bbeea8e2dc55c51bff7d1821: Status 404 returned error can't find the container with id cd83195c7df1f092a112ba5f4c8789e8a07509c6bbeea8e2dc55c51bff7d1821
Sep 10 19:10:40.222523 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:40.222496    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod639f9409_c20b_4e73_aa74_9859ab69bdaa.slice/crio-2a8fbf40cb0928aa404655bec961b55ffc7064fc67dcf21798cb252e4651d82f WatchSource:0}: Error finding container 2a8fbf40cb0928aa404655bec961b55ffc7064fc67dcf21798cb252e4651d82f: Status 404 returned error can't find the container with id 2a8fbf40cb0928aa404655bec961b55ffc7064fc67dcf21798cb252e4651d82f
Sep 10 19:10:40.223225 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:40.223188    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20594470_2f86_4aab_88b9_20cbdae08a97.slice/crio-42b3bd4ed465b225a6ee4856b2184cffc25a5c128c66f8864a239fbdcb1502bd WatchSource:0}: Error finding container 42b3bd4ed465b225a6ee4856b2184cffc25a5c128c66f8864a239fbdcb1502bd: Status 404 returned error can't find the container with id 42b3bd4ed465b225a6ee4856b2184cffc25a5c128c66f8864a239fbdcb1502bd
Sep 10 19:10:40.223935 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:40.223830    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod808e9bad_e9c5_4316_b4b6_e967cd0e3ef9.slice/crio-a1800dbafe0a7f8f135e095dba5557d04b5eaf1f2a15d05c0eafac2caab571ff WatchSource:0}: Error finding container a1800dbafe0a7f8f135e095dba5557d04b5eaf1f2a15d05c0eafac2caab571ff: Status 404 returned error can't find the container with id a1800dbafe0a7f8f135e095dba5557d04b5eaf1f2a15d05c0eafac2caab571ff
Sep 10 19:10:40.227177 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:40.227128    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod315b4bf2_f3fd_4819_a012_73aa1ec10865.slice/crio-f7cfb095c367282a45a555f7f454b10758d026cfc74b84d6f09d85a59f13dd87 WatchSource:0}: Error finding container f7cfb095c367282a45a555f7f454b10758d026cfc74b84d6f09d85a59f13dd87: Status 404 returned error can't find the container with id f7cfb095c367282a45a555f7f454b10758d026cfc74b84d6f09d85a59f13dd87
Sep 10 19:10:40.227971 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:40.227946    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfe72c48_1d12_4ebc_9a54_6032f900cb2e.slice/crio-8c632e4715429eb938d0c873258d337a8faacea974d0380ea93d2741caeed5dc WatchSource:0}: Error finding container 8c632e4715429eb938d0c873258d337a8faacea974d0380ea93d2741caeed5dc: Status 404 returned error can't find the container with id 8c632e4715429eb938d0c873258d337a8faacea974d0380ea93d2741caeed5dc
Sep 10 19:10:40.229075 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:10:40.228939    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fd21370_c32c_4778_87cb_93ca2dd49ac4.slice/crio-6a15f6e53844d160f3d15751fe53f0345d8bd6b4771fe039b6de83312d197104 WatchSource:0}: Error finding container 6a15f6e53844d160f3d15751fe53f0345d8bd6b4771fe039b6de83312d197104: Status 404 returned error can't find the container with id 6a15f6e53844d160f3d15751fe53f0345d8bd6b4771fe039b6de83312d197104
Sep 10 19:10:40.283911 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:40.283876    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5k9nq" event={"ID":"315b4bf2-f3fd-4819-a012-73aa1ec10865","Type":"ContainerStarted","Data":"f7cfb095c367282a45a555f7f454b10758d026cfc74b84d6f09d85a59f13dd87"}
Sep 10 19:10:40.285127 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:40.285096    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-x4n86" event={"ID":"808e9bad-e9c5-4316-b4b6-e967cd0e3ef9","Type":"ContainerStarted","Data":"a1800dbafe0a7f8f135e095dba5557d04b5eaf1f2a15d05c0eafac2caab571ff"}
Sep 10 19:10:40.286398 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:40.286366    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bq2kb" event={"ID":"20594470-2f86-4aab-88b9-20cbdae08a97","Type":"ContainerStarted","Data":"42b3bd4ed465b225a6ee4856b2184cffc25a5c128c66f8864a239fbdcb1502bd"}
Sep 10 19:10:40.287499 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:40.287468    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-l4kqt" event={"ID":"639f9409-c20b-4e73-aa74-9859ab69bdaa","Type":"ContainerStarted","Data":"2a8fbf40cb0928aa404655bec961b55ffc7064fc67dcf21798cb252e4651d82f"}
Sep 10 19:10:40.288677 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:40.288645    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vxn8r" event={"ID":"0fd21370-c32c-4778-87cb-93ca2dd49ac4","Type":"ContainerStarted","Data":"6a15f6e53844d160f3d15751fe53f0345d8bd6b4771fe039b6de83312d197104"}
Sep 10 19:10:40.289648 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:40.289626    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nhrc7" event={"ID":"97e5ac31-cbf0-464c-b59c-9708fa487b4d","Type":"ContainerStarted","Data":"cd83195c7df1f092a112ba5f4c8789e8a07509c6bbeea8e2dc55c51bff7d1821"}
Sep 10 19:10:40.290686 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:40.290665    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-q9894" event={"ID":"74e0d981-08c3-4d91-9ff8-a5e32fc627e8","Type":"ContainerStarted","Data":"3ed0f3db7978d804d09b94368d07d180a10bedf35c206ddc70e2a1930355360f"}
Sep 10 19:10:40.291781 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:40.291763    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-76h6j" event={"ID":"591dc1dc-1751-46ee-afd4-ab5a89f8d8ab","Type":"ContainerStarted","Data":"98b173bb0bc03d33c8f6b826c4fe55fc686595bf2acda72cdcddb178a628b316"}
Sep 10 19:10:40.292791 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:40.292772    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nfbcl" event={"ID":"bfe72c48-1d12-4ebc-9a54-6032f900cb2e","Type":"ContainerStarted","Data":"8c632e4715429eb938d0c873258d337a8faacea974d0380ea93d2741caeed5dc"}
Sep 10 19:10:40.760552 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:40.759752    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/63940132-1151-4ec6-b8fd-a68baa948a23-original-pull-secret\") pod \"global-pull-secret-syncer-qhq2q\" (UID: \"63940132-1151-4ec6-b8fd-a68baa948a23\") " pod="kube-system/global-pull-secret-syncer-qhq2q"
Sep 10 19:10:40.760552 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:10:40.759918    2546 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 19:10:40.760552 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:10:40.760110    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63940132-1151-4ec6-b8fd-a68baa948a23-original-pull-secret podName:63940132-1151-4ec6-b8fd-a68baa948a23 nodeName:}" failed. No retries permitted until 2026-09-10 19:10:42.760089091 +0000 UTC m=+5.989139115 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/63940132-1151-4ec6-b8fd-a68baa948a23-original-pull-secret") pod "global-pull-secret-syncer-qhq2q" (UID: "63940132-1151-4ec6-b8fd-a68baa948a23") : object "kube-system"/"original-pull-secret" not registered
Sep 10 19:10:40.860481 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:40.860441    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxpc6\" (UniqueName: \"kubernetes.io/projected/3ddbe88b-0b64-4363-9803-cd0abd242bdc-kube-api-access-nxpc6\") pod \"network-check-target-tjtbz\" (UID: \"3ddbe88b-0b64-4363-9803-cd0abd242bdc\") " pod="openshift-network-diagnostics/network-check-target-tjtbz"
Sep 10 19:10:40.860666 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:40.860539    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50ce76d4-2a8b-48b6-8939-df2ca2f0a802-metrics-certs\") pod \"network-metrics-daemon-7pg6j\" (UID: \"50ce76d4-2a8b-48b6-8939-df2ca2f0a802\") " pod="openshift-multus/network-metrics-daemon-7pg6j"
Sep 10 19:10:40.860730 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:10:40.860667    2546 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 19:10:40.860787 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:10:40.860731    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50ce76d4-2a8b-48b6-8939-df2ca2f0a802-metrics-certs podName:50ce76d4-2a8b-48b6-8939-df2ca2f0a802 nodeName:}" failed. No retries permitted until 2026-09-10 19:10:42.860712444 +0000 UTC m=+6.089762450 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/50ce76d4-2a8b-48b6-8939-df2ca2f0a802-metrics-certs") pod "network-metrics-daemon-7pg6j" (UID: "50ce76d4-2a8b-48b6-8939-df2ca2f0a802") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 19:10:40.861149 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:10:40.861130    2546 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 19:10:40.861228 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:10:40.861154    2546 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 19:10:40.861228 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:10:40.861170    2546 projected.go:196] Error preparing data for projected volume kube-api-access-nxpc6 for pod openshift-network-diagnostics/network-check-target-tjtbz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 19:10:40.861228 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:10:40.861213    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3ddbe88b-0b64-4363-9803-cd0abd242bdc-kube-api-access-nxpc6 podName:3ddbe88b-0b64-4363-9803-cd0abd242bdc nodeName:}" failed. No retries permitted until 2026-09-10 19:10:42.861198504 +0000 UTC m=+6.090248513 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-nxpc6" (UniqueName: "kubernetes.io/projected/3ddbe88b-0b64-4363-9803-cd0abd242bdc-kube-api-access-nxpc6") pod "network-check-target-tjtbz" (UID: "3ddbe88b-0b64-4363-9803-cd0abd242bdc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 19:10:41.067211 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:41.067131    2546 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 19:10:41.176773 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:41.176677    2546 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 19:05:39 +0000 UTC" deadline="2028-04-24 23:08:47.513010372 +0000 UTC"
Sep 10 19:10:41.176773 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:41.176772    2546 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14211h58m6.336243216s"
Sep 10 19:10:41.277460 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:41.277420    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tjtbz"
Sep 10 19:10:41.277632 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:10:41.277553    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tjtbz" podUID="3ddbe88b-0b64-4363-9803-cd0abd242bdc"
Sep 10 19:10:41.278063 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:41.278042    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qhq2q"
Sep 10 19:10:41.278204 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:10:41.278147    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qhq2q" podUID="63940132-1151-4ec6-b8fd-a68baa948a23"
Sep 10 19:10:41.278270 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:41.278220    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7pg6j"
Sep 10 19:10:41.278379 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:10:41.278313    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7pg6j" podUID="50ce76d4-2a8b-48b6-8939-df2ca2f0a802"
Sep 10 19:10:41.310533 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:41.310500    2546 generic.go:356] "Generic (PLEG): container finished" podID="3c00c3d431cb7c2cd59dc800908e3a9e" containerID="cd949effff3874130b8c24701a9e468bfd1ff9b14f726658beb9b3576c83ee8a" exitCode=0
Sep 10 19:10:41.310696 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:41.310595    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-222.ec2.internal" event={"ID":"3c00c3d431cb7c2cd59dc800908e3a9e","Type":"ContainerDied","Data":"cd949effff3874130b8c24701a9e468bfd1ff9b14f726658beb9b3576c83ee8a"}
Sep 10 19:10:41.326731 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:41.326702    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-222.ec2.internal" event={"ID":"658d2ad482679f9690b7cc4c32ebd70f","Type":"ContainerStarted","Data":"eb6480a8e307488a1c867077ad601b5c98ab4bc26d35967a0f25c9d24fcfd1d6"}
Sep 10 19:10:42.353817 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:42.353738    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-222.ec2.internal" event={"ID":"3c00c3d431cb7c2cd59dc800908e3a9e","Type":"ContainerStarted","Data":"a3f4d75e6f4fbb886e0229e7feb53694120cafc1b2b83f4a8e003c44b82b83b4"}
Sep 10 19:10:42.370393 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:42.370313    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-222.ec2.internal" podStartSLOduration=4.370294164 podStartE2EDuration="4.370294164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 19:10:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:10:42.370252615 +0000 UTC m=+5.599302640" watchObservedRunningTime="2026-09-10 19:10:42.370294164 +0000 UTC m=+5.599344191"
Sep 10 19:10:42.370912 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:42.370870    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-222.ec2.internal" podStartSLOduration=4.370857531 podStartE2EDuration="4.370857531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 19:10:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:10:41.343675995 +0000 UTC m=+4.572726022" watchObservedRunningTime="2026-09-10 19:10:42.370857531 +0000 UTC m=+5.599907562"
Sep 10 19:10:42.777972 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:42.777886    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/63940132-1151-4ec6-b8fd-a68baa948a23-original-pull-secret\") pod \"global-pull-secret-syncer-qhq2q\" (UID: \"63940132-1151-4ec6-b8fd-a68baa948a23\") " pod="kube-system/global-pull-secret-syncer-qhq2q"
Sep 10 19:10:42.778128 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:10:42.778105    2546 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 19:10:42.778185 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:10:42.778174    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63940132-1151-4ec6-b8fd-a68baa948a23-original-pull-secret podName:63940132-1151-4ec6-b8fd-a68baa948a23 nodeName:}" failed. No retries permitted until 2026-09-10 19:10:46.778155198 +0000 UTC m=+10.007205202 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/63940132-1151-4ec6-b8fd-a68baa948a23-original-pull-secret") pod "global-pull-secret-syncer-qhq2q" (UID: "63940132-1151-4ec6-b8fd-a68baa948a23") : object "kube-system"/"original-pull-secret" not registered
Sep 10 19:10:42.878387 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:42.878322    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50ce76d4-2a8b-48b6-8939-df2ca2f0a802-metrics-certs\") pod \"network-metrics-daemon-7pg6j\" (UID: \"50ce76d4-2a8b-48b6-8939-df2ca2f0a802\") " pod="openshift-multus/network-metrics-daemon-7pg6j"
Sep 10 19:10:42.878572 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:42.878393    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxpc6\" (UniqueName: \"kubernetes.io/projected/3ddbe88b-0b64-4363-9803-cd0abd242bdc-kube-api-access-nxpc6\") pod \"network-check-target-tjtbz\" (UID: \"3ddbe88b-0b64-4363-9803-cd0abd242bdc\") " pod="openshift-network-diagnostics/network-check-target-tjtbz"
Sep 10 19:10:42.878572 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:10:42.878529    2546 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 19:10:42.878572 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:10:42.878546    2546 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 19:10:42.878572 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:10:42.878558    2546 projected.go:196] Error preparing data for projected volume kube-api-access-nxpc6 for pod openshift-network-diagnostics/network-check-target-tjtbz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 19:10:42.878776 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:10:42.878614    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3ddbe88b-0b64-4363-9803-cd0abd242bdc-kube-api-access-nxpc6 podName:3ddbe88b-0b64-4363-9803-cd0abd242bdc nodeName:}" failed. No retries permitted until 2026-09-10 19:10:46.87859606 +0000 UTC m=+10.107646068 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-nxpc6" (UniqueName: "kubernetes.io/projected/3ddbe88b-0b64-4363-9803-cd0abd242bdc-kube-api-access-nxpc6") pod "network-check-target-tjtbz" (UID: "3ddbe88b-0b64-4363-9803-cd0abd242bdc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 19:10:42.878776 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:10:42.878678    2546 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 19:10:42.878776 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:10:42.878709    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50ce76d4-2a8b-48b6-8939-df2ca2f0a802-metrics-certs podName:50ce76d4-2a8b-48b6-8939-df2ca2f0a802 nodeName:}" failed. No retries permitted until 2026-09-10 19:10:46.878698544 +0000 UTC m=+10.107748553 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/50ce76d4-2a8b-48b6-8939-df2ca2f0a802-metrics-certs") pod "network-metrics-daemon-7pg6j" (UID: "50ce76d4-2a8b-48b6-8939-df2ca2f0a802") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 19:10:43.275076 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:43.274639    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qhq2q"
Sep 10 19:10:43.275076 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:10:43.274752    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qhq2q" podUID="63940132-1151-4ec6-b8fd-a68baa948a23"
Sep 10 19:10:43.275076 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:43.274827    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7pg6j"
Sep 10 19:10:43.275076 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:10:43.274938    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7pg6j" podUID="50ce76d4-2a8b-48b6-8939-df2ca2f0a802"
Sep 10 19:10:43.275076 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:43.274640    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tjtbz"
Sep 10 19:10:43.275076 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:10:43.275032    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tjtbz" podUID="3ddbe88b-0b64-4363-9803-cd0abd242bdc"
Sep 10 19:10:45.275990 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:45.275369    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qhq2q"
Sep 10 19:10:45.275990 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:10:45.275498    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qhq2q" podUID="63940132-1151-4ec6-b8fd-a68baa948a23"
Sep 10 19:10:45.275990 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:45.275853    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7pg6j"
Sep 10 19:10:45.275990 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:10:45.275952    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7pg6j" podUID="50ce76d4-2a8b-48b6-8939-df2ca2f0a802"
Sep 10 19:10:45.276680 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:45.276315    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tjtbz"
Sep 10 19:10:45.276680 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:10:45.276466    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tjtbz" podUID="3ddbe88b-0b64-4363-9803-cd0abd242bdc"
Sep 10 19:10:46.817298 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:46.817244    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/63940132-1151-4ec6-b8fd-a68baa948a23-original-pull-secret\") pod \"global-pull-secret-syncer-qhq2q\" (UID: \"63940132-1151-4ec6-b8fd-a68baa948a23\") " pod="kube-system/global-pull-secret-syncer-qhq2q"
Sep 10 19:10:46.817791 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:10:46.817441    2546 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 19:10:46.817791 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:10:46.817506    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63940132-1151-4ec6-b8fd-a68baa948a23-original-pull-secret podName:63940132-1151-4ec6-b8fd-a68baa948a23 nodeName:}" failed. No retries permitted until 2026-09-10 19:10:54.817487314 +0000 UTC m=+18.046537342 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/63940132-1151-4ec6-b8fd-a68baa948a23-original-pull-secret") pod "global-pull-secret-syncer-qhq2q" (UID: "63940132-1151-4ec6-b8fd-a68baa948a23") : object "kube-system"/"original-pull-secret" not registered
Sep 10 19:10:46.918578 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:46.918537    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50ce76d4-2a8b-48b6-8939-df2ca2f0a802-metrics-certs\") pod \"network-metrics-daemon-7pg6j\" (UID: \"50ce76d4-2a8b-48b6-8939-df2ca2f0a802\") " pod="openshift-multus/network-metrics-daemon-7pg6j"
Sep 10 19:10:46.918760 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:46.918604    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxpc6\" (UniqueName: \"kubernetes.io/projected/3ddbe88b-0b64-4363-9803-cd0abd242bdc-kube-api-access-nxpc6\") pod \"network-check-target-tjtbz\" (UID: \"3ddbe88b-0b64-4363-9803-cd0abd242bdc\") " pod="openshift-network-diagnostics/network-check-target-tjtbz"
Sep 10 19:10:46.918820 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:10:46.918805    2546 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 19:10:46.918865 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:10:46.918827    2546 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 19:10:46.918865 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:10:46.918840    2546 projected.go:196] Error preparing data for projected volume kube-api-access-nxpc6 for pod openshift-network-diagnostics/network-check-target-tjtbz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 19:10:46.918949 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:10:46.918898    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3ddbe88b-0b64-4363-9803-cd0abd242bdc-kube-api-access-nxpc6 podName:3ddbe88b-0b64-4363-9803-cd0abd242bdc nodeName:}" failed. No retries permitted until 2026-09-10 19:10:54.91887815 +0000 UTC m=+18.147928159 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-nxpc6" (UniqueName: "kubernetes.io/projected/3ddbe88b-0b64-4363-9803-cd0abd242bdc-kube-api-access-nxpc6") pod "network-check-target-tjtbz" (UID: "3ddbe88b-0b64-4363-9803-cd0abd242bdc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 19:10:46.919108 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:10:46.919083    2546 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 19:10:46.919228 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:10:46.919141    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50ce76d4-2a8b-48b6-8939-df2ca2f0a802-metrics-certs podName:50ce76d4-2a8b-48b6-8939-df2ca2f0a802 nodeName:}" failed. No retries permitted until 2026-09-10 19:10:54.919124982 +0000 UTC m=+18.148174991 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/50ce76d4-2a8b-48b6-8939-df2ca2f0a802-metrics-certs") pod "network-metrics-daemon-7pg6j" (UID: "50ce76d4-2a8b-48b6-8939-df2ca2f0a802") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 19:10:47.276094 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:47.276007    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7pg6j"
Sep 10 19:10:47.276263 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:10:47.276141    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7pg6j" podUID="50ce76d4-2a8b-48b6-8939-df2ca2f0a802"
Sep 10 19:10:47.276374 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:47.276330    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tjtbz"
Sep 10 19:10:47.276487 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:10:47.276463    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tjtbz" podUID="3ddbe88b-0b64-4363-9803-cd0abd242bdc"
Sep 10 19:10:47.276550 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:47.276532    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qhq2q"
Sep 10 19:10:47.276623 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:10:47.276605    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qhq2q" podUID="63940132-1151-4ec6-b8fd-a68baa948a23"
Sep 10 19:10:49.277759 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:49.277726    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7pg6j"
Sep 10 19:10:49.277759 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:49.277757    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tjtbz"
Sep 10 19:10:49.278229 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:10:49.277848    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7pg6j" podUID="50ce76d4-2a8b-48b6-8939-df2ca2f0a802"
Sep 10 19:10:49.278229 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:49.277901    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qhq2q"
Sep 10 19:10:49.278229 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:10:49.277985    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tjtbz" podUID="3ddbe88b-0b64-4363-9803-cd0abd242bdc"
Sep 10 19:10:49.278229 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:10:49.278060    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qhq2q" podUID="63940132-1151-4ec6-b8fd-a68baa948a23"
Sep 10 19:10:51.274538 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:51.274502    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7pg6j"
Sep 10 19:10:51.275043 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:51.274503    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qhq2q"
Sep 10 19:10:51.275043 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:10:51.274696    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7pg6j" podUID="50ce76d4-2a8b-48b6-8939-df2ca2f0a802"
Sep 10 19:10:51.275043 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:51.274703    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tjtbz"
Sep 10 19:10:51.275043 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:10:51.274793    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qhq2q" podUID="63940132-1151-4ec6-b8fd-a68baa948a23"
Sep 10 19:10:51.275043 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:10:51.274863    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tjtbz" podUID="3ddbe88b-0b64-4363-9803-cd0abd242bdc"
Sep 10 19:10:53.278289 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:53.278259    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tjtbz"
Sep 10 19:10:53.278767 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:53.278408    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qhq2q"
Sep 10 19:10:53.278767 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:53.278436    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7pg6j"
Sep 10 19:10:53.278767 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:10:53.278426    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tjtbz" podUID="3ddbe88b-0b64-4363-9803-cd0abd242bdc"
Sep 10 19:10:53.278767 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:10:53.278537    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qhq2q" podUID="63940132-1151-4ec6-b8fd-a68baa948a23"
Sep 10 19:10:53.278767 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:10:53.278619    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7pg6j" podUID="50ce76d4-2a8b-48b6-8939-df2ca2f0a802"
Sep 10 19:10:54.875195 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:54.875151    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/63940132-1151-4ec6-b8fd-a68baa948a23-original-pull-secret\") pod \"global-pull-secret-syncer-qhq2q\" (UID: \"63940132-1151-4ec6-b8fd-a68baa948a23\") " pod="kube-system/global-pull-secret-syncer-qhq2q"
Sep 10 19:10:54.875758 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:10:54.875375    2546 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 19:10:54.875758 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:10:54.875461    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63940132-1151-4ec6-b8fd-a68baa948a23-original-pull-secret podName:63940132-1151-4ec6-b8fd-a68baa948a23 nodeName:}" failed. No retries permitted until 2026-09-10 19:11:10.875439956 +0000 UTC m=+34.104489983 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/63940132-1151-4ec6-b8fd-a68baa948a23-original-pull-secret") pod "global-pull-secret-syncer-qhq2q" (UID: "63940132-1151-4ec6-b8fd-a68baa948a23") : object "kube-system"/"original-pull-secret" not registered
Sep 10 19:10:54.976141 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:54.976099    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50ce76d4-2a8b-48b6-8939-df2ca2f0a802-metrics-certs\") pod \"network-metrics-daemon-7pg6j\" (UID: \"50ce76d4-2a8b-48b6-8939-df2ca2f0a802\") " pod="openshift-multus/network-metrics-daemon-7pg6j"
Sep 10 19:10:54.976141 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:54.976159    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxpc6\" (UniqueName: \"kubernetes.io/projected/3ddbe88b-0b64-4363-9803-cd0abd242bdc-kube-api-access-nxpc6\") pod \"network-check-target-tjtbz\" (UID: \"3ddbe88b-0b64-4363-9803-cd0abd242bdc\") " pod="openshift-network-diagnostics/network-check-target-tjtbz"
Sep 10 19:10:54.976400 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:10:54.976314    2546 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 19:10:54.976460 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:10:54.976404    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50ce76d4-2a8b-48b6-8939-df2ca2f0a802-metrics-certs podName:50ce76d4-2a8b-48b6-8939-df2ca2f0a802 nodeName:}" failed. No retries permitted until 2026-09-10 19:11:10.976383377 +0000 UTC m=+34.205433382 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/50ce76d4-2a8b-48b6-8939-df2ca2f0a802-metrics-certs") pod "network-metrics-daemon-7pg6j" (UID: "50ce76d4-2a8b-48b6-8939-df2ca2f0a802") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 19:10:54.976460 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:10:54.976321    2546 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 19:10:54.976460 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:10:54.976428    2546 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 19:10:54.976460 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:10:54.976440    2546 projected.go:196] Error preparing data for projected volume kube-api-access-nxpc6 for pod openshift-network-diagnostics/network-check-target-tjtbz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 19:10:54.976695 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:10:54.976505    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3ddbe88b-0b64-4363-9803-cd0abd242bdc-kube-api-access-nxpc6 podName:3ddbe88b-0b64-4363-9803-cd0abd242bdc nodeName:}" failed. No retries permitted until 2026-09-10 19:11:10.9764743 +0000 UTC m=+34.205524315 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-nxpc6" (UniqueName: "kubernetes.io/projected/3ddbe88b-0b64-4363-9803-cd0abd242bdc-kube-api-access-nxpc6") pod "network-check-target-tjtbz" (UID: "3ddbe88b-0b64-4363-9803-cd0abd242bdc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 19:10:55.278364 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:55.278269    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7pg6j"
Sep 10 19:10:55.278544 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:55.278402    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tjtbz"
Sep 10 19:10:55.278544 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:10:55.278414    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7pg6j" podUID="50ce76d4-2a8b-48b6-8939-df2ca2f0a802"
Sep 10 19:10:55.278544 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:55.278437    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qhq2q"
Sep 10 19:10:55.278544 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:10:55.278526    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tjtbz" podUID="3ddbe88b-0b64-4363-9803-cd0abd242bdc"
Sep 10 19:10:55.278754 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:10:55.278613    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qhq2q" podUID="63940132-1151-4ec6-b8fd-a68baa948a23"
Sep 10 19:10:57.276280 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:57.276211    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qhq2q"
Sep 10 19:10:57.276752 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:57.276308    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7pg6j"
Sep 10 19:10:57.276752 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:10:57.276374    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qhq2q" podUID="63940132-1151-4ec6-b8fd-a68baa948a23"
Sep 10 19:10:57.276752 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:10:57.276412    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7pg6j" podUID="50ce76d4-2a8b-48b6-8939-df2ca2f0a802"
Sep 10 19:10:57.276752 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:57.276449    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tjtbz"
Sep 10 19:10:57.276752 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:10:57.276510    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tjtbz" podUID="3ddbe88b-0b64-4363-9803-cd0abd242bdc"
Sep 10 19:10:58.387858 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:58.387406    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-q9894" event={"ID":"74e0d981-08c3-4d91-9ff8-a5e32fc627e8","Type":"ContainerStarted","Data":"5773ae3b325d0031e24cd218eb45a346353d843f820b5eff3705bd046b4fa5e5"}
Sep 10 19:10:58.388909 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:58.388875    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-76h6j" event={"ID":"591dc1dc-1751-46ee-afd4-ab5a89f8d8ab","Type":"ContainerStarted","Data":"643179ca5c14e7a9a27da82c49c1f303bd132ec7b1935fa9df2e2ecdaa20b9ad"}
Sep 10 19:10:58.390219 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:58.390192    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nfbcl" event={"ID":"bfe72c48-1d12-4ebc-9a54-6032f900cb2e","Type":"ContainerStarted","Data":"ec65b4fb4b63c1509e68ef6caa31c6f7fe3c689ae7aae8220f491a145d594db7"}
Sep 10 19:10:58.392533 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:58.392512    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5k9nq" event={"ID":"315b4bf2-f3fd-4819-a012-73aa1ec10865","Type":"ContainerStarted","Data":"a7e495a9aadc7fc0f1f5cd133bb4df9eeeaa995e9cd79bb5f54f737c438627b9"}
Sep 10 19:10:58.392602 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:58.392569    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5k9nq" event={"ID":"315b4bf2-f3fd-4819-a012-73aa1ec10865","Type":"ContainerStarted","Data":"8018ae1e37abf833c564c04844248c9003945f9af1bc3bfcc72a1ff023c8e767"}
Sep 10 19:10:58.392602 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:58.392600    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5k9nq" event={"ID":"315b4bf2-f3fd-4819-a012-73aa1ec10865","Type":"ContainerStarted","Data":"4e1d2acde897dea2d2c1b9877f00dfd076e8ae8d3508907ba0e9ce6010ad99db"}
Sep 10 19:10:58.392689 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:58.392639    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5k9nq" event={"ID":"315b4bf2-f3fd-4819-a012-73aa1ec10865","Type":"ContainerStarted","Data":"7e46b9c170e9e10336fbd675d110aeb569c3df1eb8eb30676db875a8f857f221"}
Sep 10 19:10:58.393798 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:58.393778    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-x4n86" event={"ID":"808e9bad-e9c5-4316-b4b6-e967cd0e3ef9","Type":"ContainerStarted","Data":"f390d94ec59291d90b663068d1ce49361f1f3127e895081212ad0a33512e6490"}
Sep 10 19:10:58.395128 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:58.395102    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bq2kb" event={"ID":"20594470-2f86-4aab-88b9-20cbdae08a97","Type":"ContainerStarted","Data":"c4887d33ce3965a50983ce864ec0233ce272c1d93811478d02014740c3429e6d"}
Sep 10 19:10:58.396420 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:58.396392    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-l4kqt" event={"ID":"639f9409-c20b-4e73-aa74-9859ab69bdaa","Type":"ContainerStarted","Data":"36e3d0a5a8306437c7e07d838c98e3c055f14a3d1c3383dd609a1ab250271082"}
Sep 10 19:10:58.397871 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:58.397849    2546 generic.go:356] "Generic (PLEG): container finished" podID="97e5ac31-cbf0-464c-b59c-9708fa487b4d" containerID="a94feae3295d546a02638b7675a3c7c6fbc1b27a586a5615c4156097458a35a3" exitCode=0
Sep 10 19:10:58.397952 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:58.397877    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nhrc7" event={"ID":"97e5ac31-cbf0-464c-b59c-9708fa487b4d","Type":"ContainerDied","Data":"a94feae3295d546a02638b7675a3c7c6fbc1b27a586a5615c4156097458a35a3"}
Sep 10 19:10:58.408252 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:58.408217    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-q9894" podStartSLOduration=3.699494691 podStartE2EDuration="21.40820652s" totalImagesPullingTime="17.708711829s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 19:10:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:10:58.408018825 +0000 UTC m=+21.637068851" watchObservedRunningTime="2026-09-10 19:10:58.40820652 +0000 UTC m=+21.637256545"
Sep 10 19:10:58.424820 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:58.424785    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-l4kqt" podStartSLOduration=3.829356173 podStartE2EDuration="21.424775651s" totalImagesPullingTime="17.595419478s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:10:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:10:58.424696104 +0000 UTC m=+21.653746129" watchObservedRunningTime="2026-09-10 19:10:58.424775651 +0000 UTC m=+21.653825675"
Sep 10 19:10:58.459324 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:58.459284    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-76h6j" podStartSLOduration=3.7868908660000002 podStartE2EDuration="21.459267673s" totalImagesPullingTime="17.672376807s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 19:10:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:10:58.44279739 +0000 UTC m=+21.671847415" watchObservedRunningTime="2026-09-10 19:10:58.459267673 +0000 UTC m=+21.688317696"
Sep 10 19:10:58.459427 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:58.459392    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-x4n86" podStartSLOduration=3.779442615 podStartE2EDuration="21.45938634s" totalImagesPullingTime="17.679943725s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 19:10:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:10:58.458877267 +0000 UTC m=+21.687927293" watchObservedRunningTime="2026-09-10 19:10:58.45938634 +0000 UTC m=+21.688436369"
Sep 10 19:10:58.499984 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:58.499943    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-bq2kb" podStartSLOduration=3.857217914 podStartE2EDuration="21.499928583s" totalImagesPullingTime="17.642710669s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 19:10:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:10:58.475481359 +0000 UTC m=+21.704531386" watchObservedRunningTime="2026-09-10 19:10:58.499928583 +0000 UTC m=+21.728978644"
Sep 10 19:10:59.275113 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:59.275033    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tjtbz"
Sep 10 19:10:59.275113 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:59.275063    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qhq2q"
Sep 10 19:10:59.275324 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:59.275031    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7pg6j"
Sep 10 19:10:59.275324 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:10:59.275146    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tjtbz" podUID="3ddbe88b-0b64-4363-9803-cd0abd242bdc"
Sep 10 19:10:59.275324 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:10:59.275244    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qhq2q" podUID="63940132-1151-4ec6-b8fd-a68baa948a23"
Sep 10 19:10:59.275324 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:10:59.275283    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7pg6j" podUID="50ce76d4-2a8b-48b6-8939-df2ca2f0a802"
Sep 10 19:10:59.401480 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:59.401441    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vxn8r" event={"ID":"0fd21370-c32c-4778-87cb-93ca2dd49ac4","Type":"ContainerStarted","Data":"385924c0f81ad391bd0e7ec27ecb52a839f18e67226db7ea9a987c4ff07147ef"}
Sep 10 19:10:59.404368 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:59.404335    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5k9nq" event={"ID":"315b4bf2-f3fd-4819-a012-73aa1ec10865","Type":"ContainerStarted","Data":"5523f95e82c1804eadf4bd99ddc335bd6b95b659ead437709a65bcce5971a0f2"}
Sep 10 19:10:59.404525 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:59.404411    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5k9nq" event={"ID":"315b4bf2-f3fd-4819-a012-73aa1ec10865","Type":"ContainerStarted","Data":"15ae3a46399cfc0dea67e9c3808cb1b7c093fc236651c3a8ea485e6abcf8b7cf"}
Sep 10 19:10:59.416936 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:59.416887    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-vxn8r" podStartSLOduration=4.7471448160000005 podStartE2EDuration="22.416868859s" totalImagesPullingTime="17.669724043s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 19:10:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:10:59.41619556 +0000 UTC m=+22.645245586" watchObservedRunningTime="2026-09-10 19:10:59.416868859 +0000 UTC m=+22.645918884"
Sep 10 19:10:59.456166 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:10:59.456141    2546 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 10 19:11:00.191228 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:00.191132    2546 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-10T19:10:59.456161801Z","UUID":"edec0c44-4d53-4dd0-b7dd-ab0924ed1c72","Handler":null,"Name":"","Endpoint":""}
Sep 10 19:11:00.192450 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:00.192422    2546 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 10 19:11:00.192450 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:00.192455    2546 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 10 19:11:00.409314 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:00.409275    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nfbcl" event={"ID":"bfe72c48-1d12-4ebc-9a54-6032f900cb2e","Type":"ContainerStarted","Data":"a94d3aaefda90ca81469295c6de57f9a27b2b9f465966e090f1cd102c5561e87"}
Sep 10 19:11:01.274732 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:01.274703    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tjtbz"
Sep 10 19:11:01.274887 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:11:01.274844    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tjtbz" podUID="3ddbe88b-0b64-4363-9803-cd0abd242bdc"
Sep 10 19:11:01.275005 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:01.274970    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qhq2q"
Sep 10 19:11:01.275005 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:01.274999    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7pg6j"
Sep 10 19:11:01.275176 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:11:01.275070    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qhq2q" podUID="63940132-1151-4ec6-b8fd-a68baa948a23"
Sep 10 19:11:01.275235 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:11:01.275196    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7pg6j" podUID="50ce76d4-2a8b-48b6-8939-df2ca2f0a802"
Sep 10 19:11:01.414387 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:01.414280    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nfbcl" event={"ID":"bfe72c48-1d12-4ebc-9a54-6032f900cb2e","Type":"ContainerStarted","Data":"08d7588c9031ba055be5ac9aa2d13abc9eb889568a8543dade9cd9bc47525b4b"}
Sep 10 19:11:01.418308 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:01.418281    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5k9nq" event={"ID":"315b4bf2-f3fd-4819-a012-73aa1ec10865","Type":"ContainerStarted","Data":"e7747d6c022fd4bb7d303f204f29736ce814d8aa37f84fdfc8057f9ff7a2d890"}
Sep 10 19:11:01.435272 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:01.435225    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nfbcl" podStartSLOduration=3.915049603 podStartE2EDuration="24.435209685s" totalImagesPullingTime="20.520160082s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 19:10:37 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:11:01.434851205 +0000 UTC m=+24.663901234" watchObservedRunningTime="2026-09-10 19:11:01.435209685 +0000 UTC m=+24.664259710"
Sep 10 19:11:03.179865 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:03.179719    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-l4kqt"
Sep 10 19:11:03.180473 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:03.180200    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-l4kqt"
Sep 10 19:11:03.275028 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:03.275000    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tjtbz"
Sep 10 19:11:03.275160 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:03.275000    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qhq2q"
Sep 10 19:11:03.275160 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:11:03.275121    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tjtbz" podUID="3ddbe88b-0b64-4363-9803-cd0abd242bdc"
Sep 10 19:11:03.275160 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:03.275129    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7pg6j"
Sep 10 19:11:03.275286 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:11:03.275227    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qhq2q" podUID="63940132-1151-4ec6-b8fd-a68baa948a23"
Sep 10 19:11:03.275366 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:11:03.275327    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7pg6j" podUID="50ce76d4-2a8b-48b6-8939-df2ca2f0a802"
Sep 10 19:11:03.425024 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:03.424774    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nhrc7" event={"ID":"97e5ac31-cbf0-464c-b59c-9708fa487b4d","Type":"ContainerStarted","Data":"b89a81f1b559601eacdfd6dd3d32dc786ee8fad70100d0e51871814976818870"}
Sep 10 19:11:03.428741 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:03.428711    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5k9nq" event={"ID":"315b4bf2-f3fd-4819-a012-73aa1ec10865","Type":"ContainerStarted","Data":"658e03c07d4920d66173d7aac39c0dd703843a29b503b5609c939c681f52d10b"}
Sep 10 19:11:03.428884 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:03.428869    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-l4kqt"
Sep 10 19:11:03.429356 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:03.429327    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-l4kqt"
Sep 10 19:11:03.468619 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:03.468494    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-5k9nq" podStartSLOduration=8.667600668 podStartE2EDuration="26.468478792s" totalImagesPullingTime="17.800878124s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 19:10:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:11:03.467129355 +0000 UTC m=+26.696179671" watchObservedRunningTime="2026-09-10 19:11:03.468478792 +0000 UTC m=+26.697528816"
Sep 10 19:11:04.433873 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:04.433832    2546 generic.go:356] "Generic (PLEG): container finished" podID="97e5ac31-cbf0-464c-b59c-9708fa487b4d" containerID="b89a81f1b559601eacdfd6dd3d32dc786ee8fad70100d0e51871814976818870" exitCode=0
Sep 10 19:11:04.434635 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:04.433949    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nhrc7" event={"ID":"97e5ac31-cbf0-464c-b59c-9708fa487b4d","Type":"ContainerDied","Data":"b89a81f1b559601eacdfd6dd3d32dc786ee8fad70100d0e51871814976818870"}
Sep 10 19:11:04.436435 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:04.435444    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5k9nq"
Sep 10 19:11:04.436435 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:04.435523    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5k9nq"
Sep 10 19:11:04.436435 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:04.435570    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5k9nq"
Sep 10 19:11:04.470793 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:04.470744    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5k9nq"
Sep 10 19:11:04.474974 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:04.474928    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5k9nq"
Sep 10 19:11:05.070018 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:05.069961    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qhq2q"]
Sep 10 19:11:05.070307 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:05.070110    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qhq2q"
Sep 10 19:11:05.070307 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:11:05.070230    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qhq2q" podUID="63940132-1151-4ec6-b8fd-a68baa948a23"
Sep 10 19:11:05.073065 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:05.072916    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7pg6j"]
Sep 10 19:11:05.073065 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:05.073051    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7pg6j"
Sep 10 19:11:05.073271 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:11:05.073159    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7pg6j" podUID="50ce76d4-2a8b-48b6-8939-df2ca2f0a802"
Sep 10 19:11:05.073707 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:05.073682    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tjtbz"]
Sep 10 19:11:05.073898 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:05.073792    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tjtbz"
Sep 10 19:11:05.073898 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:11:05.073881    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tjtbz" podUID="3ddbe88b-0b64-4363-9803-cd0abd242bdc"
Sep 10 19:11:05.437954 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:05.437919    2546 generic.go:356] "Generic (PLEG): container finished" podID="97e5ac31-cbf0-464c-b59c-9708fa487b4d" containerID="14b93b54cc637109444d819226218c43f5fda82d1f899f2444d20f099bddc097" exitCode=0
Sep 10 19:11:05.438355 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:05.437999    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nhrc7" event={"ID":"97e5ac31-cbf0-464c-b59c-9708fa487b4d","Type":"ContainerDied","Data":"14b93b54cc637109444d819226218c43f5fda82d1f899f2444d20f099bddc097"}
Sep 10 19:11:06.442656 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:06.442472    2546 generic.go:356] "Generic (PLEG): container finished" podID="97e5ac31-cbf0-464c-b59c-9708fa487b4d" containerID="c9d7247419137b4c06be5e375d9749ebb9026d7b1b78f2f1b1c88716c7c97a26" exitCode=0
Sep 10 19:11:06.443032 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:06.442539    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nhrc7" event={"ID":"97e5ac31-cbf0-464c-b59c-9708fa487b4d","Type":"ContainerDied","Data":"c9d7247419137b4c06be5e375d9749ebb9026d7b1b78f2f1b1c88716c7c97a26"}
Sep 10 19:11:07.278458 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:07.278378    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qhq2q"
Sep 10 19:11:07.278611 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:07.278483    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7pg6j"
Sep 10 19:11:07.278611 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:11:07.278497    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qhq2q" podUID="63940132-1151-4ec6-b8fd-a68baa948a23"
Sep 10 19:11:07.278611 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:07.278514    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tjtbz"
Sep 10 19:11:07.278807 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:11:07.278612    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7pg6j" podUID="50ce76d4-2a8b-48b6-8939-df2ca2f0a802"
Sep 10 19:11:07.278807 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:11:07.278684    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tjtbz" podUID="3ddbe88b-0b64-4363-9803-cd0abd242bdc"
Sep 10 19:11:09.278449 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:09.278413    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7pg6j"
Sep 10 19:11:09.278449 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:09.278432    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tjtbz"
Sep 10 19:11:09.279115 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:09.278437    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qhq2q"
Sep 10 19:11:09.279115 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:11:09.278525    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7pg6j" podUID="50ce76d4-2a8b-48b6-8939-df2ca2f0a802"
Sep 10 19:11:09.279115 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:11:09.278600    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tjtbz" podUID="3ddbe88b-0b64-4363-9803-cd0abd242bdc"
Sep 10 19:11:09.279115 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:11:09.278660    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qhq2q" podUID="63940132-1151-4ec6-b8fd-a68baa948a23"
Sep 10 19:11:10.492740 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:10.492707    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-222.ec2.internal" event="NodeReady"
Sep 10 19:11:10.493290 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:10.492851    2546 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 10 19:11:10.533821 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:10.533784    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-76cdd6f759-kq486"]
Sep 10 19:11:10.563907 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:10.563879    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-m7z8f"]
Sep 10 19:11:10.564082 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:10.564037    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-76cdd6f759-kq486"
Sep 10 19:11:10.568737 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:10.568712    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-hjfhn"]
Sep 10 19:11:10.568879 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:10.568838    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-m7z8f"
Sep 10 19:11:10.571672 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:10.571649    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 10 19:11:10.572712 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:10.572678    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 10 19:11:10.572822 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:10.572753    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-76cdd6f759-kq486"]
Sep 10 19:11:10.572822 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:10.572786    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-m7z8f"]
Sep 10 19:11:10.572822 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:10.572796    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 10 19:11:10.572822 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:10.572815    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hjfhn"]
Sep 10 19:11:10.573024 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:10.572893    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hjfhn"
Sep 10 19:11:10.573080 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:10.573042    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 10 19:11:10.573080 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:10.573053    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 10 19:11:10.573473 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:10.573454    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-b59cd\""
Sep 10 19:11:10.573568 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:10.573456    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-8474t\""
Sep 10 19:11:10.576376 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:10.576279    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 10 19:11:10.576376 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:10.576350    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 10 19:11:10.576600 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:10.576435    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-zn8jq\""
Sep 10 19:11:10.576736 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:10.576718    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 10 19:11:10.583656 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:10.583634    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 10 19:11:10.680295 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:10.680255    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/809134f0-f6fa-4ce5-befc-58bda6b2c1d4-cert\") pod \"ingress-canary-hjfhn\" (UID: \"809134f0-f6fa-4ce5-befc-58bda6b2c1d4\") " pod="openshift-ingress-canary/ingress-canary-hjfhn"
Sep 10 19:11:10.680485 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:10.680305    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f05a659a-63af-4897-a74b-e8b246a1a59f-config-volume\") pod \"dns-default-m7z8f\" (UID: \"f05a659a-63af-4897-a74b-e8b246a1a59f\") " pod="openshift-dns/dns-default-m7z8f"
Sep 10 19:11:10.680485 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:10.680390    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f5fc406b-f790-4121-98c2-af351ed63401-installation-pull-secrets\") pod \"image-registry-76cdd6f759-kq486\" (UID: \"f5fc406b-f790-4121-98c2-af351ed63401\") " pod="openshift-image-registry/image-registry-76cdd6f759-kq486"
Sep 10 19:11:10.680485 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:10.680479    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f05a659a-63af-4897-a74b-e8b246a1a59f-metrics-tls\") pod \"dns-default-m7z8f\" (UID: \"f05a659a-63af-4897-a74b-e8b246a1a59f\") " pod="openshift-dns/dns-default-m7z8f"
Sep 10 19:11:10.680654 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:10.680506    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnl8f\" (UniqueName: \"kubernetes.io/projected/f05a659a-63af-4897-a74b-e8b246a1a59f-kube-api-access-rnl8f\") pod \"dns-default-m7z8f\" (UID: \"f05a659a-63af-4897-a74b-e8b246a1a59f\") " pod="openshift-dns/dns-default-m7z8f"
Sep 10 19:11:10.680654 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:10.680533    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f5fc406b-f790-4121-98c2-af351ed63401-ca-trust-extracted\") pod \"image-registry-76cdd6f759-kq486\" (UID: \"f5fc406b-f790-4121-98c2-af351ed63401\") " pod="openshift-image-registry/image-registry-76cdd6f759-kq486"
Sep 10 19:11:10.680654 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:10.680564    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f5fc406b-f790-4121-98c2-af351ed63401-image-registry-private-configuration\") pod \"image-registry-76cdd6f759-kq486\" (UID: \"f5fc406b-f790-4121-98c2-af351ed63401\") " pod="openshift-image-registry/image-registry-76cdd6f759-kq486"
Sep 10 19:11:10.680790 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:10.680663    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f5fc406b-f790-4121-98c2-af351ed63401-registry-tls\") pod \"image-registry-76cdd6f759-kq486\" (UID: \"f5fc406b-f790-4121-98c2-af351ed63401\") " pod="openshift-image-registry/image-registry-76cdd6f759-kq486"
Sep 10 19:11:10.680790 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:10.680698    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c55zb\" (UniqueName: \"kubernetes.io/projected/f5fc406b-f790-4121-98c2-af351ed63401-kube-api-access-c55zb\") pod \"image-registry-76cdd6f759-kq486\" (UID: \"f5fc406b-f790-4121-98c2-af351ed63401\") " pod="openshift-image-registry/image-registry-76cdd6f759-kq486"
Sep 10 19:11:10.680790 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:10.680721    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgkjj\" (UniqueName: \"kubernetes.io/projected/809134f0-f6fa-4ce5-befc-58bda6b2c1d4-kube-api-access-zgkjj\") pod \"ingress-canary-hjfhn\" (UID: \"809134f0-f6fa-4ce5-befc-58bda6b2c1d4\") " pod="openshift-ingress-canary/ingress-canary-hjfhn"
Sep 10 19:11:10.680896 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:10.680781    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f5fc406b-f790-4121-98c2-af351ed63401-trusted-ca\") pod \"image-registry-76cdd6f759-kq486\" (UID: \"f5fc406b-f790-4121-98c2-af351ed63401\") " pod="openshift-image-registry/image-registry-76cdd6f759-kq486"
Sep 10 19:11:10.680896 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:10.680831    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f5fc406b-f790-4121-98c2-af351ed63401-registry-certificates\") pod \"image-registry-76cdd6f759-kq486\" (UID: \"f5fc406b-f790-4121-98c2-af351ed63401\") " pod="openshift-image-registry/image-registry-76cdd6f759-kq486"
Sep 10 19:11:10.680896 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:10.680852    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f5fc406b-f790-4121-98c2-af351ed63401-bound-sa-token\") pod \"image-registry-76cdd6f759-kq486\" (UID: \"f5fc406b-f790-4121-98c2-af351ed63401\") " pod="openshift-image-registry/image-registry-76cdd6f759-kq486"
Sep 10 19:11:10.680896 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:10.680866    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f05a659a-63af-4897-a74b-e8b246a1a59f-tmp-dir\") pod \"dns-default-m7z8f\" (UID: \"f05a659a-63af-4897-a74b-e8b246a1a59f\") " pod="openshift-dns/dns-default-m7z8f"
Sep 10 19:11:10.782142 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:10.782054    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f5fc406b-f790-4121-98c2-af351ed63401-image-registry-private-configuration\") pod \"image-registry-76cdd6f759-kq486\" (UID: \"f5fc406b-f790-4121-98c2-af351ed63401\") " pod="openshift-image-registry/image-registry-76cdd6f759-kq486"
Sep 10 19:11:10.782142 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:10.782117    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f5fc406b-f790-4121-98c2-af351ed63401-registry-tls\") pod \"image-registry-76cdd6f759-kq486\" (UID: \"f5fc406b-f790-4121-98c2-af351ed63401\") " pod="openshift-image-registry/image-registry-76cdd6f759-kq486"
Sep 10 19:11:10.782142 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:10.782142    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c55zb\" (UniqueName: \"kubernetes.io/projected/f5fc406b-f790-4121-98c2-af351ed63401-kube-api-access-c55zb\") pod \"image-registry-76cdd6f759-kq486\" (UID: \"f5fc406b-f790-4121-98c2-af351ed63401\") " pod="openshift-image-registry/image-registry-76cdd6f759-kq486"
Sep 10 19:11:10.782448 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:10.782165    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgkjj\" (UniqueName: \"kubernetes.io/projected/809134f0-f6fa-4ce5-befc-58bda6b2c1d4-kube-api-access-zgkjj\") pod \"ingress-canary-hjfhn\" (UID: \"809134f0-f6fa-4ce5-befc-58bda6b2c1d4\") " pod="openshift-ingress-canary/ingress-canary-hjfhn"
Sep 10 19:11:10.782448 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:10.782195    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f5fc406b-f790-4121-98c2-af351ed63401-trusted-ca\") pod \"image-registry-76cdd6f759-kq486\" (UID: \"f5fc406b-f790-4121-98c2-af351ed63401\") " pod="openshift-image-registry/image-registry-76cdd6f759-kq486"
Sep 10 19:11:10.782448 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:10.782221    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f5fc406b-f790-4121-98c2-af351ed63401-registry-certificates\") pod \"image-registry-76cdd6f759-kq486\" (UID: \"f5fc406b-f790-4121-98c2-af351ed63401\") " pod="openshift-image-registry/image-registry-76cdd6f759-kq486"
Sep 10 19:11:10.782448 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:10.782244    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f5fc406b-f790-4121-98c2-af351ed63401-bound-sa-token\") pod \"image-registry-76cdd6f759-kq486\" (UID: \"f5fc406b-f790-4121-98c2-af351ed63401\") " pod="openshift-image-registry/image-registry-76cdd6f759-kq486"
Sep 10 19:11:10.782448 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:10.782265    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f05a659a-63af-4897-a74b-e8b246a1a59f-tmp-dir\") pod \"dns-default-m7z8f\" (UID: \"f05a659a-63af-4897-a74b-e8b246a1a59f\") " pod="openshift-dns/dns-default-m7z8f"
Sep 10 19:11:10.782448 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:10.782287    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/809134f0-f6fa-4ce5-befc-58bda6b2c1d4-cert\") pod \"ingress-canary-hjfhn\" (UID: \"809134f0-f6fa-4ce5-befc-58bda6b2c1d4\") " pod="openshift-ingress-canary/ingress-canary-hjfhn"
Sep 10 19:11:10.782448 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:11:10.782287    2546 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 19:11:10.782448 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:11:10.782313    2546 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76cdd6f759-kq486: secret "image-registry-tls" not found
Sep 10 19:11:10.782448 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:11:10.782403    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f5fc406b-f790-4121-98c2-af351ed63401-registry-tls podName:f5fc406b-f790-4121-98c2-af351ed63401 nodeName:}" failed. No retries permitted until 2026-09-10 19:11:11.282381425 +0000 UTC m=+34.511431443 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f5fc406b-f790-4121-98c2-af351ed63401-registry-tls") pod "image-registry-76cdd6f759-kq486" (UID: "f5fc406b-f790-4121-98c2-af351ed63401") : secret "image-registry-tls" not found
Sep 10 19:11:10.782888 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:10.782314    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f05a659a-63af-4897-a74b-e8b246a1a59f-config-volume\") pod \"dns-default-m7z8f\" (UID: \"f05a659a-63af-4897-a74b-e8b246a1a59f\") " pod="openshift-dns/dns-default-m7z8f"
Sep 10 19:11:10.782888 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:10.782563    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f5fc406b-f790-4121-98c2-af351ed63401-installation-pull-secrets\") pod \"image-registry-76cdd6f759-kq486\" (UID: \"f5fc406b-f790-4121-98c2-af351ed63401\") " pod="openshift-image-registry/image-registry-76cdd6f759-kq486"
Sep 10 19:11:10.782888 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:10.782639    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f05a659a-63af-4897-a74b-e8b246a1a59f-metrics-tls\") pod \"dns-default-m7z8f\" (UID: \"f05a659a-63af-4897-a74b-e8b246a1a59f\") " pod="openshift-dns/dns-default-m7z8f"
Sep 10 19:11:10.782888 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:10.782667    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnl8f\" (UniqueName: \"kubernetes.io/projected/f05a659a-63af-4897-a74b-e8b246a1a59f-kube-api-access-rnl8f\") pod \"dns-default-m7z8f\" (UID: \"f05a659a-63af-4897-a74b-e8b246a1a59f\") " pod="openshift-dns/dns-default-m7z8f"
Sep 10 19:11:10.782888 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:10.782699    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f5fc406b-f790-4121-98c2-af351ed63401-ca-trust-extracted\") pod \"image-registry-76cdd6f759-kq486\" (UID: \"f5fc406b-f790-4121-98c2-af351ed63401\") " pod="openshift-image-registry/image-registry-76cdd6f759-kq486"
Sep 10 19:11:10.783135 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:10.782939    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f05a659a-63af-4897-a74b-e8b246a1a59f-config-volume\") pod \"dns-default-m7z8f\" (UID: \"f05a659a-63af-4897-a74b-e8b246a1a59f\") " pod="openshift-dns/dns-default-m7z8f"
Sep 10 19:11:10.783135 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:10.782946    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f5fc406b-f790-4121-98c2-af351ed63401-registry-certificates\") pod \"image-registry-76cdd6f759-kq486\" (UID: \"f5fc406b-f790-4121-98c2-af351ed63401\") " pod="openshift-image-registry/image-registry-76cdd6f759-kq486"
Sep 10 19:11:10.783135 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:10.782993    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f5fc406b-f790-4121-98c2-af351ed63401-ca-trust-extracted\") pod \"image-registry-76cdd6f759-kq486\" (UID: \"f5fc406b-f790-4121-98c2-af351ed63401\") " pod="openshift-image-registry/image-registry-76cdd6f759-kq486"
Sep 10 19:11:10.783135 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:11:10.783055    2546 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 19:11:10.783285 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:10.783154    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f05a659a-63af-4897-a74b-e8b246a1a59f-tmp-dir\") pod \"dns-default-m7z8f\" (UID: \"f05a659a-63af-4897-a74b-e8b246a1a59f\") " pod="openshift-dns/dns-default-m7z8f"
Sep 10 19:11:10.783285 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:11:10.783193    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f05a659a-63af-4897-a74b-e8b246a1a59f-metrics-tls podName:f05a659a-63af-4897-a74b-e8b246a1a59f nodeName:}" failed. No retries permitted until 2026-09-10 19:11:11.283176352 +0000 UTC m=+34.512226369 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f05a659a-63af-4897-a74b-e8b246a1a59f-metrics-tls") pod "dns-default-m7z8f" (UID: "f05a659a-63af-4897-a74b-e8b246a1a59f") : secret "dns-default-metrics-tls" not found
Sep 10 19:11:10.783397 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:10.783375    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f5fc406b-f790-4121-98c2-af351ed63401-trusted-ca\") pod \"image-registry-76cdd6f759-kq486\" (UID: \"f5fc406b-f790-4121-98c2-af351ed63401\") " pod="openshift-image-registry/image-registry-76cdd6f759-kq486"
Sep 10 19:11:10.786430 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:10.786408    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f5fc406b-f790-4121-98c2-af351ed63401-installation-pull-secrets\") pod \"image-registry-76cdd6f759-kq486\" (UID: \"f5fc406b-f790-4121-98c2-af351ed63401\") " pod="openshift-image-registry/image-registry-76cdd6f759-kq486"
Sep 10 19:11:10.786557 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:10.786540    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/809134f0-f6fa-4ce5-befc-58bda6b2c1d4-cert\") pod \"ingress-canary-hjfhn\" (UID: \"809134f0-f6fa-4ce5-befc-58bda6b2c1d4\") " pod="openshift-ingress-canary/ingress-canary-hjfhn"
Sep 10 19:11:10.786598 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:10.786559    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f5fc406b-f790-4121-98c2-af351ed63401-image-registry-private-configuration\") pod \"image-registry-76cdd6f759-kq486\" (UID: \"f5fc406b-f790-4121-98c2-af351ed63401\") " pod="openshift-image-registry/image-registry-76cdd6f759-kq486"
Sep 10 19:11:10.793396 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:10.793366    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgkjj\" (UniqueName: \"kubernetes.io/projected/809134f0-f6fa-4ce5-befc-58bda6b2c1d4-kube-api-access-zgkjj\") pod \"ingress-canary-hjfhn\" (UID: \"809134f0-f6fa-4ce5-befc-58bda6b2c1d4\") " pod="openshift-ingress-canary/ingress-canary-hjfhn"
Sep 10 19:11:10.794293 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:10.794270    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f5fc406b-f790-4121-98c2-af351ed63401-bound-sa-token\") pod \"image-registry-76cdd6f759-kq486\" (UID: \"f5fc406b-f790-4121-98c2-af351ed63401\") " pod="openshift-image-registry/image-registry-76cdd6f759-kq486"
Sep 10 19:11:10.794528 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:10.794506    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnl8f\" (UniqueName: \"kubernetes.io/projected/f05a659a-63af-4897-a74b-e8b246a1a59f-kube-api-access-rnl8f\") pod \"dns-default-m7z8f\" (UID: \"f05a659a-63af-4897-a74b-e8b246a1a59f\") " pod="openshift-dns/dns-default-m7z8f"
Sep 10 19:11:10.794614 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:10.794600    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c55zb\" (UniqueName: \"kubernetes.io/projected/f5fc406b-f790-4121-98c2-af351ed63401-kube-api-access-c55zb\") pod \"image-registry-76cdd6f759-kq486\" (UID: \"f5fc406b-f790-4121-98c2-af351ed63401\") " pod="openshift-image-registry/image-registry-76cdd6f759-kq486"
Sep 10 19:11:10.883145 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:10.883109    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/63940132-1151-4ec6-b8fd-a68baa948a23-original-pull-secret\") pod \"global-pull-secret-syncer-qhq2q\" (UID: \"63940132-1151-4ec6-b8fd-a68baa948a23\") " pod="kube-system/global-pull-secret-syncer-qhq2q"
Sep 10 19:11:10.883314 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:11:10.883235    2546 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 19:11:10.883314 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:11:10.883304    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63940132-1151-4ec6-b8fd-a68baa948a23-original-pull-secret podName:63940132-1151-4ec6-b8fd-a68baa948a23 nodeName:}" failed. No retries permitted until 2026-09-10 19:11:42.883287685 +0000 UTC m=+66.112337693 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/63940132-1151-4ec6-b8fd-a68baa948a23-original-pull-secret") pod "global-pull-secret-syncer-qhq2q" (UID: "63940132-1151-4ec6-b8fd-a68baa948a23") : object "kube-system"/"original-pull-secret" not registered
Sep 10 19:11:10.889936 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:10.889911    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hjfhn"
Sep 10 19:11:10.983994 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:10.983781    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50ce76d4-2a8b-48b6-8939-df2ca2f0a802-metrics-certs\") pod \"network-metrics-daemon-7pg6j\" (UID: \"50ce76d4-2a8b-48b6-8939-df2ca2f0a802\") " pod="openshift-multus/network-metrics-daemon-7pg6j"
Sep 10 19:11:10.984115 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:10.984032    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxpc6\" (UniqueName: \"kubernetes.io/projected/3ddbe88b-0b64-4363-9803-cd0abd242bdc-kube-api-access-nxpc6\") pod \"network-check-target-tjtbz\" (UID: \"3ddbe88b-0b64-4363-9803-cd0abd242bdc\") " pod="openshift-network-diagnostics/network-check-target-tjtbz"
Sep 10 19:11:10.984115 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:11:10.983930    2546 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 19:11:10.984223 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:11:10.984139    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50ce76d4-2a8b-48b6-8939-df2ca2f0a802-metrics-certs podName:50ce76d4-2a8b-48b6-8939-df2ca2f0a802 nodeName:}" failed. No retries permitted until 2026-09-10 19:11:42.984116063 +0000 UTC m=+66.213166083 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/50ce76d4-2a8b-48b6-8939-df2ca2f0a802-metrics-certs") pod "network-metrics-daemon-7pg6j" (UID: "50ce76d4-2a8b-48b6-8939-df2ca2f0a802") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 19:11:10.984275 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:11:10.984242    2546 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 19:11:10.984275 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:11:10.984264    2546 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 19:11:10.984349 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:11:10.984280    2546 projected.go:196] Error preparing data for projected volume kube-api-access-nxpc6 for pod openshift-network-diagnostics/network-check-target-tjtbz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 19:11:10.984397 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:11:10.984356    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3ddbe88b-0b64-4363-9803-cd0abd242bdc-kube-api-access-nxpc6 podName:3ddbe88b-0b64-4363-9803-cd0abd242bdc nodeName:}" failed. No retries permitted until 2026-09-10 19:11:42.984320542 +0000 UTC m=+66.213370562 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-nxpc6" (UniqueName: "kubernetes.io/projected/3ddbe88b-0b64-4363-9803-cd0abd242bdc-kube-api-access-nxpc6") pod "network-check-target-tjtbz" (UID: "3ddbe88b-0b64-4363-9803-cd0abd242bdc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 19:11:11.200920 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:11.200892    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hjfhn"]
Sep 10 19:11:11.207311 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:11:11.207285    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod809134f0_f6fa_4ce5_befc_58bda6b2c1d4.slice/crio-e386fbd76f194f83858b2f48bb13247240234f8563ba6675162687ef33b2fda5 WatchSource:0}: Error finding container e386fbd76f194f83858b2f48bb13247240234f8563ba6675162687ef33b2fda5: Status 404 returned error can't find the container with id e386fbd76f194f83858b2f48bb13247240234f8563ba6675162687ef33b2fda5
Sep 10 19:11:11.274849 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:11.274818    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qhq2q"
Sep 10 19:11:11.275036 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:11.274818    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7pg6j"
Sep 10 19:11:11.275036 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:11.274817    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tjtbz"
Sep 10 19:11:11.277952 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:11.277745    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 10 19:11:11.277952 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:11.277745    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-zj5gq\""
Sep 10 19:11:11.277952 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:11.277826    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 10 19:11:11.277952 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:11.277834    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-r8gdr\""
Sep 10 19:11:11.278196 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:11.278117    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 10 19:11:11.278196 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:11.278151    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 10 19:11:11.285892 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:11.285871    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f5fc406b-f790-4121-98c2-af351ed63401-registry-tls\") pod \"image-registry-76cdd6f759-kq486\" (UID: \"f5fc406b-f790-4121-98c2-af351ed63401\") " pod="openshift-image-registry/image-registry-76cdd6f759-kq486"
Sep 10 19:11:11.285990 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:11.285957    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f05a659a-63af-4897-a74b-e8b246a1a59f-metrics-tls\") pod \"dns-default-m7z8f\" (UID: \"f05a659a-63af-4897-a74b-e8b246a1a59f\") " pod="openshift-dns/dns-default-m7z8f"
Sep 10 19:11:11.286060 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:11:11.286003    2546 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 19:11:11.286060 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:11:11.286021    2546 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76cdd6f759-kq486: secret "image-registry-tls" not found
Sep 10 19:11:11.286149 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:11:11.286060    2546 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 19:11:11.286149 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:11:11.286072    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f5fc406b-f790-4121-98c2-af351ed63401-registry-tls podName:f5fc406b-f790-4121-98c2-af351ed63401 nodeName:}" failed. No retries permitted until 2026-09-10 19:11:12.286056938 +0000 UTC m=+35.515106943 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f5fc406b-f790-4121-98c2-af351ed63401-registry-tls") pod "image-registry-76cdd6f759-kq486" (UID: "f5fc406b-f790-4121-98c2-af351ed63401") : secret "image-registry-tls" not found
Sep 10 19:11:11.286149 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:11:11.286108    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f05a659a-63af-4897-a74b-e8b246a1a59f-metrics-tls podName:f05a659a-63af-4897-a74b-e8b246a1a59f nodeName:}" failed. No retries permitted until 2026-09-10 19:11:12.286092541 +0000 UTC m=+35.515142547 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f05a659a-63af-4897-a74b-e8b246a1a59f-metrics-tls") pod "dns-default-m7z8f" (UID: "f05a659a-63af-4897-a74b-e8b246a1a59f") : secret "dns-default-metrics-tls" not found
Sep 10 19:11:11.458379 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:11.458267    2546 generic.go:356] "Generic (PLEG): container finished" podID="97e5ac31-cbf0-464c-b59c-9708fa487b4d" containerID="498bb7eda7540c8a04de4bfaaa358311a75b9e317c5ff23f9fae8591ad0a35cb" exitCode=0
Sep 10 19:11:11.458379 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:11.458355    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nhrc7" event={"ID":"97e5ac31-cbf0-464c-b59c-9708fa487b4d","Type":"ContainerDied","Data":"498bb7eda7540c8a04de4bfaaa358311a75b9e317c5ff23f9fae8591ad0a35cb"}
Sep 10 19:11:11.459671 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:11.459649    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hjfhn" event={"ID":"809134f0-f6fa-4ce5-befc-58bda6b2c1d4","Type":"ContainerStarted","Data":"e386fbd76f194f83858b2f48bb13247240234f8563ba6675162687ef33b2fda5"}
Sep 10 19:11:12.294581 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:12.294543    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f05a659a-63af-4897-a74b-e8b246a1a59f-metrics-tls\") pod \"dns-default-m7z8f\" (UID: \"f05a659a-63af-4897-a74b-e8b246a1a59f\") " pod="openshift-dns/dns-default-m7z8f"
Sep 10 19:11:12.295042 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:12.294597    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f5fc406b-f790-4121-98c2-af351ed63401-registry-tls\") pod \"image-registry-76cdd6f759-kq486\" (UID: \"f5fc406b-f790-4121-98c2-af351ed63401\") " pod="openshift-image-registry/image-registry-76cdd6f759-kq486"
Sep 10 19:11:12.295042 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:11:12.294693    2546 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 19:11:12.295042 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:11:12.294728    2546 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 19:11:12.295042 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:11:12.294741    2546 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76cdd6f759-kq486: secret "image-registry-tls" not found
Sep 10 19:11:12.295042 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:11:12.294755    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f05a659a-63af-4897-a74b-e8b246a1a59f-metrics-tls podName:f05a659a-63af-4897-a74b-e8b246a1a59f nodeName:}" failed. No retries permitted until 2026-09-10 19:11:14.294738371 +0000 UTC m=+37.523788373 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f05a659a-63af-4897-a74b-e8b246a1a59f-metrics-tls") pod "dns-default-m7z8f" (UID: "f05a659a-63af-4897-a74b-e8b246a1a59f") : secret "dns-default-metrics-tls" not found
Sep 10 19:11:12.295042 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:11:12.294791    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f5fc406b-f790-4121-98c2-af351ed63401-registry-tls podName:f5fc406b-f790-4121-98c2-af351ed63401 nodeName:}" failed. No retries permitted until 2026-09-10 19:11:14.294767062 +0000 UTC m=+37.523817065 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f5fc406b-f790-4121-98c2-af351ed63401-registry-tls") pod "image-registry-76cdd6f759-kq486" (UID: "f5fc406b-f790-4121-98c2-af351ed63401") : secret "image-registry-tls" not found
Sep 10 19:11:12.465544 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:12.465508    2546 generic.go:356] "Generic (PLEG): container finished" podID="97e5ac31-cbf0-464c-b59c-9708fa487b4d" containerID="0ee82c92a2e568d6dec06e74e7ba481e7acd378f4d43a8f4062a41ef971f9fb1" exitCode=0
Sep 10 19:11:12.465544 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:12.465549    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nhrc7" event={"ID":"97e5ac31-cbf0-464c-b59c-9708fa487b4d","Type":"ContainerDied","Data":"0ee82c92a2e568d6dec06e74e7ba481e7acd378f4d43a8f4062a41ef971f9fb1"}
Sep 10 19:11:13.472400 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:13.472355    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nhrc7" event={"ID":"97e5ac31-cbf0-464c-b59c-9708fa487b4d","Type":"ContainerStarted","Data":"05b9300807d6837eb1558b1263f5692bfcd541e3cdac568d9b282de5cee719e9"}
Sep 10 19:11:13.495613 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:13.495538    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-nhrc7" podStartSLOduration=7.7180696090000005 podStartE2EDuration="36.495517539s" totalImagesPullingTime="28.77744793s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 19:10:37 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:11:13.494519873 +0000 UTC m=+36.723569898" watchObservedRunningTime="2026-09-10 19:11:13.495517539 +0000 UTC m=+36.724567565"
Sep 10 19:11:14.309543 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:14.309279    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f5fc406b-f790-4121-98c2-af351ed63401-registry-tls\") pod \"image-registry-76cdd6f759-kq486\" (UID: \"f5fc406b-f790-4121-98c2-af351ed63401\") " pod="openshift-image-registry/image-registry-76cdd6f759-kq486"
Sep 10 19:11:14.309718 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:14.309564    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f05a659a-63af-4897-a74b-e8b246a1a59f-metrics-tls\") pod \"dns-default-m7z8f\" (UID: \"f05a659a-63af-4897-a74b-e8b246a1a59f\") " pod="openshift-dns/dns-default-m7z8f"
Sep 10 19:11:14.309718 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:11:14.309454    2546 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 19:11:14.309718 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:11:14.309596    2546 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76cdd6f759-kq486: secret "image-registry-tls" not found
Sep 10 19:11:14.309718 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:11:14.309649    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f5fc406b-f790-4121-98c2-af351ed63401-registry-tls podName:f5fc406b-f790-4121-98c2-af351ed63401 nodeName:}" failed. No retries permitted until 2026-09-10 19:11:18.309634406 +0000 UTC m=+41.538684414 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f5fc406b-f790-4121-98c2-af351ed63401-registry-tls") pod "image-registry-76cdd6f759-kq486" (UID: "f5fc406b-f790-4121-98c2-af351ed63401") : secret "image-registry-tls" not found
Sep 10 19:11:14.309718 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:11:14.309650    2546 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 19:11:14.309718 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:11:14.309689    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f05a659a-63af-4897-a74b-e8b246a1a59f-metrics-tls podName:f05a659a-63af-4897-a74b-e8b246a1a59f nodeName:}" failed. No retries permitted until 2026-09-10 19:11:18.309677923 +0000 UTC m=+41.538727926 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f05a659a-63af-4897-a74b-e8b246a1a59f-metrics-tls") pod "dns-default-m7z8f" (UID: "f05a659a-63af-4897-a74b-e8b246a1a59f") : secret "dns-default-metrics-tls" not found
Sep 10 19:11:14.476834 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:14.476799    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hjfhn" event={"ID":"809134f0-f6fa-4ce5-befc-58bda6b2c1d4","Type":"ContainerStarted","Data":"944d52104b643ebf888a8c9376bd49f43cb0b356189389d53cea5d6997c14502"}
Sep 10 19:11:14.496681 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:14.496625    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-hjfhn" podStartSLOduration=1.7909786479999998 podStartE2EDuration="4.496611028s" totalImagesPullingTime="2.70563238s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:11:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:11:14.495142736 +0000 UTC m=+37.724192773" watchObservedRunningTime="2026-09-10 19:11:14.496611028 +0000 UTC m=+37.725661053"
Sep 10 19:11:16.963299 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:16.963265    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-97jbx"]
Sep 10 19:11:17.000520 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:17.000481    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-97jbx"]
Sep 10 19:11:17.000675 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:17.000601    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-97jbx"
Sep 10 19:11:17.003172 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:17.003151    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 10 19:11:17.003172 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:17.003170    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 10 19:11:17.003355 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:17.003181    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-6tdn6\""
Sep 10 19:11:17.133190 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:17.133150    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ac0e549a-a86a-4ac5-8a59-9b0cebc5b7f0-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-97jbx\" (UID: \"ac0e549a-a86a-4ac5-8a59-9b0cebc5b7f0\") " pod="openshift-network-console/networking-console-plugin-79977f9844-97jbx"
Sep 10 19:11:17.133371 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:17.133280    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ac0e549a-a86a-4ac5-8a59-9b0cebc5b7f0-nginx-conf\") pod \"networking-console-plugin-79977f9844-97jbx\" (UID: \"ac0e549a-a86a-4ac5-8a59-9b0cebc5b7f0\") " pod="openshift-network-console/networking-console-plugin-79977f9844-97jbx"
Sep 10 19:11:17.233640 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:17.233551    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ac0e549a-a86a-4ac5-8a59-9b0cebc5b7f0-nginx-conf\") pod \"networking-console-plugin-79977f9844-97jbx\" (UID: \"ac0e549a-a86a-4ac5-8a59-9b0cebc5b7f0\") " pod="openshift-network-console/networking-console-plugin-79977f9844-97jbx"
Sep 10 19:11:17.233799 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:17.233659    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ac0e549a-a86a-4ac5-8a59-9b0cebc5b7f0-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-97jbx\" (UID: \"ac0e549a-a86a-4ac5-8a59-9b0cebc5b7f0\") " pod="openshift-network-console/networking-console-plugin-79977f9844-97jbx"
Sep 10 19:11:17.233799 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:11:17.233765    2546 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 10 19:11:17.233922 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:11:17.233819    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac0e549a-a86a-4ac5-8a59-9b0cebc5b7f0-networking-console-plugin-cert podName:ac0e549a-a86a-4ac5-8a59-9b0cebc5b7f0 nodeName:}" failed. No retries permitted until 2026-09-10 19:11:17.733801527 +0000 UTC m=+40.962851531 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ac0e549a-a86a-4ac5-8a59-9b0cebc5b7f0-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-97jbx" (UID: "ac0e549a-a86a-4ac5-8a59-9b0cebc5b7f0") : secret "networking-console-plugin-cert" not found
Sep 10 19:11:17.234085 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:17.234062    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ac0e549a-a86a-4ac5-8a59-9b0cebc5b7f0-nginx-conf\") pod \"networking-console-plugin-79977f9844-97jbx\" (UID: \"ac0e549a-a86a-4ac5-8a59-9b0cebc5b7f0\") " pod="openshift-network-console/networking-console-plugin-79977f9844-97jbx"
Sep 10 19:11:17.737360 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:17.737304    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ac0e549a-a86a-4ac5-8a59-9b0cebc5b7f0-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-97jbx\" (UID: \"ac0e549a-a86a-4ac5-8a59-9b0cebc5b7f0\") " pod="openshift-network-console/networking-console-plugin-79977f9844-97jbx"
Sep 10 19:11:17.737533 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:11:17.737455    2546 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 10 19:11:17.737533 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:11:17.737516    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac0e549a-a86a-4ac5-8a59-9b0cebc5b7f0-networking-console-plugin-cert podName:ac0e549a-a86a-4ac5-8a59-9b0cebc5b7f0 nodeName:}" failed. No retries permitted until 2026-09-10 19:11:18.73750091 +0000 UTC m=+41.966550913 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ac0e549a-a86a-4ac5-8a59-9b0cebc5b7f0-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-97jbx" (UID: "ac0e549a-a86a-4ac5-8a59-9b0cebc5b7f0") : secret "networking-console-plugin-cert" not found
Sep 10 19:11:18.342841 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:18.342797    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f5fc406b-f790-4121-98c2-af351ed63401-registry-tls\") pod \"image-registry-76cdd6f759-kq486\" (UID: \"f5fc406b-f790-4121-98c2-af351ed63401\") " pod="openshift-image-registry/image-registry-76cdd6f759-kq486"
Sep 10 19:11:18.343270 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:11:18.342956    2546 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 19:11:18.343270 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:11:18.342978    2546 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76cdd6f759-kq486: secret "image-registry-tls" not found
Sep 10 19:11:18.343270 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:18.342994    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f05a659a-63af-4897-a74b-e8b246a1a59f-metrics-tls\") pod \"dns-default-m7z8f\" (UID: \"f05a659a-63af-4897-a74b-e8b246a1a59f\") " pod="openshift-dns/dns-default-m7z8f"
Sep 10 19:11:18.343270 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:11:18.343026    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f5fc406b-f790-4121-98c2-af351ed63401-registry-tls podName:f5fc406b-f790-4121-98c2-af351ed63401 nodeName:}" failed. No retries permitted until 2026-09-10 19:11:26.34301079 +0000 UTC m=+49.572060798 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f5fc406b-f790-4121-98c2-af351ed63401-registry-tls") pod "image-registry-76cdd6f759-kq486" (UID: "f5fc406b-f790-4121-98c2-af351ed63401") : secret "image-registry-tls" not found
Sep 10 19:11:18.343270 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:11:18.343073    2546 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 19:11:18.343270 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:11:18.343111    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f05a659a-63af-4897-a74b-e8b246a1a59f-metrics-tls podName:f05a659a-63af-4897-a74b-e8b246a1a59f nodeName:}" failed. No retries permitted until 2026-09-10 19:11:26.343100757 +0000 UTC m=+49.572150760 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f05a659a-63af-4897-a74b-e8b246a1a59f-metrics-tls") pod "dns-default-m7z8f" (UID: "f05a659a-63af-4897-a74b-e8b246a1a59f") : secret "dns-default-metrics-tls" not found
Sep 10 19:11:18.745770 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:18.745730    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ac0e549a-a86a-4ac5-8a59-9b0cebc5b7f0-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-97jbx\" (UID: \"ac0e549a-a86a-4ac5-8a59-9b0cebc5b7f0\") " pod="openshift-network-console/networking-console-plugin-79977f9844-97jbx"
Sep 10 19:11:18.745949 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:11:18.745893    2546 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 10 19:11:18.745990 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:11:18.745956    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac0e549a-a86a-4ac5-8a59-9b0cebc5b7f0-networking-console-plugin-cert podName:ac0e549a-a86a-4ac5-8a59-9b0cebc5b7f0 nodeName:}" failed. No retries permitted until 2026-09-10 19:11:20.745940957 +0000 UTC m=+43.974990959 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ac0e549a-a86a-4ac5-8a59-9b0cebc5b7f0-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-97jbx" (UID: "ac0e549a-a86a-4ac5-8a59-9b0cebc5b7f0") : secret "networking-console-plugin-cert" not found
Sep 10 19:11:20.160584 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:20.160550    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-qhr8m"]
Sep 10 19:11:20.182727 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:20.182695    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-qhr8m"]
Sep 10 19:11:20.182872 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:20.182808    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-qhr8m"
Sep 10 19:11:20.185968 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:20.185944    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 10 19:11:20.186067 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:20.186045    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 10 19:11:20.186621 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:20.186589    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 10 19:11:20.186757 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:20.186737    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 10 19:11:20.190264 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:20.187950    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-zl4hx\""
Sep 10 19:11:20.355408 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:20.355367    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b579572a-1c79-4730-a112-193b3887bbce-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qhr8m\" (UID: \"b579572a-1c79-4730-a112-193b3887bbce\") " pod="openshift-insights/insights-runtime-extractor-qhr8m"
Sep 10 19:11:20.355596 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:20.355502    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b579572a-1c79-4730-a112-193b3887bbce-data-volume\") pod \"insights-runtime-extractor-qhr8m\" (UID: \"b579572a-1c79-4730-a112-193b3887bbce\") " pod="openshift-insights/insights-runtime-extractor-qhr8m"
Sep 10 19:11:20.355596 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:20.355543    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b579572a-1c79-4730-a112-193b3887bbce-crio-socket\") pod \"insights-runtime-extractor-qhr8m\" (UID: \"b579572a-1c79-4730-a112-193b3887bbce\") " pod="openshift-insights/insights-runtime-extractor-qhr8m"
Sep 10 19:11:20.355691 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:20.355663    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b579572a-1c79-4730-a112-193b3887bbce-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qhr8m\" (UID: \"b579572a-1c79-4730-a112-193b3887bbce\") " pod="openshift-insights/insights-runtime-extractor-qhr8m"
Sep 10 19:11:20.355741 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:20.355688    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nq2q8\" (UniqueName: \"kubernetes.io/projected/b579572a-1c79-4730-a112-193b3887bbce-kube-api-access-nq2q8\") pod \"insights-runtime-extractor-qhr8m\" (UID: \"b579572a-1c79-4730-a112-193b3887bbce\") " pod="openshift-insights/insights-runtime-extractor-qhr8m"
Sep 10 19:11:20.456760 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:20.456673    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b579572a-1c79-4730-a112-193b3887bbce-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qhr8m\" (UID: \"b579572a-1c79-4730-a112-193b3887bbce\") " pod="openshift-insights/insights-runtime-extractor-qhr8m"
Sep 10 19:11:20.456888 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:20.456779    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b579572a-1c79-4730-a112-193b3887bbce-data-volume\") pod \"insights-runtime-extractor-qhr8m\" (UID: \"b579572a-1c79-4730-a112-193b3887bbce\") " pod="openshift-insights/insights-runtime-extractor-qhr8m"
Sep 10 19:11:20.456888 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:11:20.456798    2546 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 19:11:20.456888 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:20.456810    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b579572a-1c79-4730-a112-193b3887bbce-crio-socket\") pod \"insights-runtime-extractor-qhr8m\" (UID: \"b579572a-1c79-4730-a112-193b3887bbce\") " pod="openshift-insights/insights-runtime-extractor-qhr8m"
Sep 10 19:11:20.456888 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:11:20.456874    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b579572a-1c79-4730-a112-193b3887bbce-kube-rbac-proxy-cm podName:b579572a-1c79-4730-a112-193b3887bbce nodeName:}" failed. No retries permitted until 2026-09-10 19:11:20.956858919 +0000 UTC m=+44.185908922 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b579572a-1c79-4730-a112-193b3887bbce-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qhr8m" (UID: "b579572a-1c79-4730-a112-193b3887bbce") : configmap "kube-rbac-proxy" not found
Sep 10 19:11:20.457068 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:20.456906    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b579572a-1c79-4730-a112-193b3887bbce-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qhr8m\" (UID: \"b579572a-1c79-4730-a112-193b3887bbce\") " pod="openshift-insights/insights-runtime-extractor-qhr8m"
Sep 10 19:11:20.457068 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:20.456925    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nq2q8\" (UniqueName: \"kubernetes.io/projected/b579572a-1c79-4730-a112-193b3887bbce-kube-api-access-nq2q8\") pod \"insights-runtime-extractor-qhr8m\" (UID: \"b579572a-1c79-4730-a112-193b3887bbce\") " pod="openshift-insights/insights-runtime-extractor-qhr8m"
Sep 10 19:11:20.457068 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:11:20.457028    2546 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 19:11:20.457068 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:20.457036    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b579572a-1c79-4730-a112-193b3887bbce-crio-socket\") pod \"insights-runtime-extractor-qhr8m\" (UID: \"b579572a-1c79-4730-a112-193b3887bbce\") " pod="openshift-insights/insights-runtime-extractor-qhr8m"
Sep 10 19:11:20.457203 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:20.457093    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b579572a-1c79-4730-a112-193b3887bbce-data-volume\") pod \"insights-runtime-extractor-qhr8m\" (UID: \"b579572a-1c79-4730-a112-193b3887bbce\") " pod="openshift-insights/insights-runtime-extractor-qhr8m"
Sep 10 19:11:20.457203 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:11:20.457098    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b579572a-1c79-4730-a112-193b3887bbce-insights-runtime-extractor-tls podName:b579572a-1c79-4730-a112-193b3887bbce nodeName:}" failed. No retries permitted until 2026-09-10 19:11:20.957081072 +0000 UTC m=+44.186131081 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b579572a-1c79-4730-a112-193b3887bbce-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qhr8m" (UID: "b579572a-1c79-4730-a112-193b3887bbce") : secret "insights-runtime-extractor-tls" not found
Sep 10 19:11:20.466653 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:20.466631    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nq2q8\" (UniqueName: \"kubernetes.io/projected/b579572a-1c79-4730-a112-193b3887bbce-kube-api-access-nq2q8\") pod \"insights-runtime-extractor-qhr8m\" (UID: \"b579572a-1c79-4730-a112-193b3887bbce\") " pod="openshift-insights/insights-runtime-extractor-qhr8m"
Sep 10 19:11:20.759058 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:20.758961    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ac0e549a-a86a-4ac5-8a59-9b0cebc5b7f0-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-97jbx\" (UID: \"ac0e549a-a86a-4ac5-8a59-9b0cebc5b7f0\") " pod="openshift-network-console/networking-console-plugin-79977f9844-97jbx"
Sep 10 19:11:20.759232 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:11:20.759130    2546 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 10 19:11:20.759232 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:11:20.759216    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac0e549a-a86a-4ac5-8a59-9b0cebc5b7f0-networking-console-plugin-cert podName:ac0e549a-a86a-4ac5-8a59-9b0cebc5b7f0 nodeName:}" failed. No retries permitted until 2026-09-10 19:11:24.759196585 +0000 UTC m=+47.988246604 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ac0e549a-a86a-4ac5-8a59-9b0cebc5b7f0-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-97jbx" (UID: "ac0e549a-a86a-4ac5-8a59-9b0cebc5b7f0") : secret "networking-console-plugin-cert" not found
Sep 10 19:11:20.880944 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:20.880906    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-rt6b9"]
Sep 10 19:11:20.885663 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:20.885643    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-rt6b9"
Sep 10 19:11:20.888605 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:20.888584    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 10 19:11:20.888699 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:20.888642    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 10 19:11:20.889495 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:20.889480    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-9xc6d\""
Sep 10 19:11:20.892708 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:20.892683    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-rt6b9"]
Sep 10 19:11:20.960754 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:20.960716    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b579572a-1c79-4730-a112-193b3887bbce-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qhr8m\" (UID: \"b579572a-1c79-4730-a112-193b3887bbce\") " pod="openshift-insights/insights-runtime-extractor-qhr8m"
Sep 10 19:11:20.960946 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:20.960761    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b579572a-1c79-4730-a112-193b3887bbce-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qhr8m\" (UID: \"b579572a-1c79-4730-a112-193b3887bbce\") " pod="openshift-insights/insights-runtime-extractor-qhr8m"
Sep 10 19:11:20.960946 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:11:20.960871    2546 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 19:11:20.960946 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:11:20.960924    2546 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 19:11:20.961081 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:11:20.960932    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b579572a-1c79-4730-a112-193b3887bbce-insights-runtime-extractor-tls podName:b579572a-1c79-4730-a112-193b3887bbce nodeName:}" failed. No retries permitted until 2026-09-10 19:11:21.960917287 +0000 UTC m=+45.189967293 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b579572a-1c79-4730-a112-193b3887bbce-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qhr8m" (UID: "b579572a-1c79-4730-a112-193b3887bbce") : secret "insights-runtime-extractor-tls" not found
Sep 10 19:11:20.961081 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:11:20.960997    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b579572a-1c79-4730-a112-193b3887bbce-kube-rbac-proxy-cm podName:b579572a-1c79-4730-a112-193b3887bbce nodeName:}" failed. No retries permitted until 2026-09-10 19:11:21.960978345 +0000 UTC m=+45.190028349 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b579572a-1c79-4730-a112-193b3887bbce-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qhr8m" (UID: "b579572a-1c79-4730-a112-193b3887bbce") : configmap "kube-rbac-proxy" not found
Sep 10 19:11:21.062161 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:21.062067    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chh48\" (UniqueName: \"kubernetes.io/projected/03f3a2ce-0fe6-4460-a42e-279026aba630-kube-api-access-chh48\") pod \"migrator-fdbb8fdff-rt6b9\" (UID: \"03f3a2ce-0fe6-4460-a42e-279026aba630\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-rt6b9"
Sep 10 19:11:21.162934 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:21.162896    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chh48\" (UniqueName: \"kubernetes.io/projected/03f3a2ce-0fe6-4460-a42e-279026aba630-kube-api-access-chh48\") pod \"migrator-fdbb8fdff-rt6b9\" (UID: \"03f3a2ce-0fe6-4460-a42e-279026aba630\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-rt6b9"
Sep 10 19:11:21.171479 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:21.171454    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-chh48\" (UniqueName: \"kubernetes.io/projected/03f3a2ce-0fe6-4460-a42e-279026aba630-kube-api-access-chh48\") pod \"migrator-fdbb8fdff-rt6b9\" (UID: \"03f3a2ce-0fe6-4460-a42e-279026aba630\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-rt6b9"
Sep 10 19:11:21.194650 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:21.194617    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-rt6b9"
Sep 10 19:11:21.475551 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:21.475518    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-rt6b9"]
Sep 10 19:11:21.478463 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:11:21.478438    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03f3a2ce_0fe6_4460_a42e_279026aba630.slice/crio-83027cf1e0531b8d52de94d23f272a975bf47d4d2aa35f8e1c5991f2f86c2efc WatchSource:0}: Error finding container 83027cf1e0531b8d52de94d23f272a975bf47d4d2aa35f8e1c5991f2f86c2efc: Status 404 returned error can't find the container with id 83027cf1e0531b8d52de94d23f272a975bf47d4d2aa35f8e1c5991f2f86c2efc
Sep 10 19:11:21.494592 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:21.494566    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-rt6b9" event={"ID":"03f3a2ce-0fe6-4460-a42e-279026aba630","Type":"ContainerStarted","Data":"83027cf1e0531b8d52de94d23f272a975bf47d4d2aa35f8e1c5991f2f86c2efc"}
Sep 10 19:11:21.916762 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:21.916734    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-x4n86_808e9bad-e9c5-4316-b4b6-e967cd0e3ef9/dns-node-resolver/0.log"
Sep 10 19:11:21.969096 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:21.969058    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b579572a-1c79-4730-a112-193b3887bbce-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qhr8m\" (UID: \"b579572a-1c79-4730-a112-193b3887bbce\") " pod="openshift-insights/insights-runtime-extractor-qhr8m"
Sep 10 19:11:21.969253 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:11:21.969187    2546 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 19:11:21.969253 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:21.969210    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b579572a-1c79-4730-a112-193b3887bbce-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qhr8m\" (UID: \"b579572a-1c79-4730-a112-193b3887bbce\") " pod="openshift-insights/insights-runtime-extractor-qhr8m"
Sep 10 19:11:21.969253 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:11:21.969247    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b579572a-1c79-4730-a112-193b3887bbce-kube-rbac-proxy-cm podName:b579572a-1c79-4730-a112-193b3887bbce nodeName:}" failed. No retries permitted until 2026-09-10 19:11:23.969230511 +0000 UTC m=+47.198280514 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b579572a-1c79-4730-a112-193b3887bbce-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qhr8m" (UID: "b579572a-1c79-4730-a112-193b3887bbce") : configmap "kube-rbac-proxy" not found
Sep 10 19:11:21.969454 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:11:21.969335    2546 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 19:11:21.969454 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:11:21.969420    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b579572a-1c79-4730-a112-193b3887bbce-insights-runtime-extractor-tls podName:b579572a-1c79-4730-a112-193b3887bbce nodeName:}" failed. No retries permitted until 2026-09-10 19:11:23.969403274 +0000 UTC m=+47.198453291 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b579572a-1c79-4730-a112-193b3887bbce-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qhr8m" (UID: "b579572a-1c79-4730-a112-193b3887bbce") : secret "insights-runtime-extractor-tls" not found
Sep 10 19:11:22.717529 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:22.717501    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bq2kb_20594470-2f86-4aab-88b9-20cbdae08a97/node-ca/0.log"
Sep 10 19:11:23.098061 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:23.098029    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-75d99c484-4dh9p"]
Sep 10 19:11:23.119653 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:23.119626    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-4dh9p"]
Sep 10 19:11:23.119790 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:23.119716    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-4dh9p"
Sep 10 19:11:23.122321 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:23.122287    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 10 19:11:23.122321 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:23.122301    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 10 19:11:23.122494 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:23.122297    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 10 19:11:23.123216 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:23.123195    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-jctfz\""
Sep 10 19:11:23.123332 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:23.123195    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 10 19:11:23.280891 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:23.280839    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c1122eb6-1973-4c52-8ba4-76dd06f30eab-signing-key\") pod \"service-ca-75d99c484-4dh9p\" (UID: \"c1122eb6-1973-4c52-8ba4-76dd06f30eab\") " pod="openshift-service-ca/service-ca-75d99c484-4dh9p"
Sep 10 19:11:23.280891 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:23.280866    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9w2rj\" (UniqueName: \"kubernetes.io/projected/c1122eb6-1973-4c52-8ba4-76dd06f30eab-kube-api-access-9w2rj\") pod \"service-ca-75d99c484-4dh9p\" (UID: \"c1122eb6-1973-4c52-8ba4-76dd06f30eab\") " pod="openshift-service-ca/service-ca-75d99c484-4dh9p"
Sep 10 19:11:23.281025 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:23.280972    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c1122eb6-1973-4c52-8ba4-76dd06f30eab-signing-cabundle\") pod \"service-ca-75d99c484-4dh9p\" (UID: \"c1122eb6-1973-4c52-8ba4-76dd06f30eab\") " pod="openshift-service-ca/service-ca-75d99c484-4dh9p"
Sep 10 19:11:23.318183 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:23.318157    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hjfhn_809134f0-f6fa-4ce5-befc-58bda6b2c1d4/serve-healthcheck-canary/0.log"
Sep 10 19:11:23.381477 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:23.381443    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c1122eb6-1973-4c52-8ba4-76dd06f30eab-signing-cabundle\") pod \"service-ca-75d99c484-4dh9p\" (UID: \"c1122eb6-1973-4c52-8ba4-76dd06f30eab\") " pod="openshift-service-ca/service-ca-75d99c484-4dh9p"
Sep 10 19:11:23.381624 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:23.381504    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c1122eb6-1973-4c52-8ba4-76dd06f30eab-signing-key\") pod \"service-ca-75d99c484-4dh9p\" (UID: \"c1122eb6-1973-4c52-8ba4-76dd06f30eab\") " pod="openshift-service-ca/service-ca-75d99c484-4dh9p"
Sep 10 19:11:23.381624 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:23.381521    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9w2rj\" (UniqueName: \"kubernetes.io/projected/c1122eb6-1973-4c52-8ba4-76dd06f30eab-kube-api-access-9w2rj\") pod \"service-ca-75d99c484-4dh9p\" (UID: \"c1122eb6-1973-4c52-8ba4-76dd06f30eab\") " pod="openshift-service-ca/service-ca-75d99c484-4dh9p"
Sep 10 19:11:23.382093 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:23.382071    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c1122eb6-1973-4c52-8ba4-76dd06f30eab-signing-cabundle\") pod \"service-ca-75d99c484-4dh9p\" (UID: \"c1122eb6-1973-4c52-8ba4-76dd06f30eab\") " pod="openshift-service-ca/service-ca-75d99c484-4dh9p"
Sep 10 19:11:23.384028 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:23.384001    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c1122eb6-1973-4c52-8ba4-76dd06f30eab-signing-key\") pod \"service-ca-75d99c484-4dh9p\" (UID: \"c1122eb6-1973-4c52-8ba4-76dd06f30eab\") " pod="openshift-service-ca/service-ca-75d99c484-4dh9p"
Sep 10 19:11:23.394186 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:23.394160    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9w2rj\" (UniqueName: \"kubernetes.io/projected/c1122eb6-1973-4c52-8ba4-76dd06f30eab-kube-api-access-9w2rj\") pod \"service-ca-75d99c484-4dh9p\" (UID: \"c1122eb6-1973-4c52-8ba4-76dd06f30eab\") " pod="openshift-service-ca/service-ca-75d99c484-4dh9p"
Sep 10 19:11:23.429992 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:23.429958    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-4dh9p"
Sep 10 19:11:23.502841 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:23.502809    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-rt6b9" event={"ID":"03f3a2ce-0fe6-4460-a42e-279026aba630","Type":"ContainerStarted","Data":"7f86e692814f4a78fe18a96ab329a4bc9981176f33d48894e027ff72f099d3f9"}
Sep 10 19:11:23.502955 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:23.502874    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-rt6b9" event={"ID":"03f3a2ce-0fe6-4460-a42e-279026aba630","Type":"ContainerStarted","Data":"8909ce7f9b55cbc86c3ab929640055d0731a35c2c424f3492ea36ade3498bfc7"}
Sep 10 19:11:23.520750 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:23.520709    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-rt6b9" podStartSLOduration=1.900547566 podStartE2EDuration="3.520693733s" totalImagesPullingTime="1.620146167s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:11:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:11:23.52049871 +0000 UTC m=+46.749548736" watchObservedRunningTime="2026-09-10 19:11:23.520693733 +0000 UTC m=+46.749743751"
Sep 10 19:11:23.711726 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:23.711692    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-4dh9p"]
Sep 10 19:11:23.714378 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:11:23.714331    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1122eb6_1973_4c52_8ba4_76dd06f30eab.slice/crio-9d7d653b9b4180acb000538869a3a8d71ea465ba8a19dc615b7272510455426c WatchSource:0}: Error finding container 9d7d653b9b4180acb000538869a3a8d71ea465ba8a19dc615b7272510455426c: Status 404 returned error can't find the container with id 9d7d653b9b4180acb000538869a3a8d71ea465ba8a19dc615b7272510455426c
Sep 10 19:11:23.987890 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:23.987803    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b579572a-1c79-4730-a112-193b3887bbce-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qhr8m\" (UID: \"b579572a-1c79-4730-a112-193b3887bbce\") " pod="openshift-insights/insights-runtime-extractor-qhr8m"
Sep 10 19:11:23.988245 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:23.987917    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b579572a-1c79-4730-a112-193b3887bbce-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qhr8m\" (UID: \"b579572a-1c79-4730-a112-193b3887bbce\") " pod="openshift-insights/insights-runtime-extractor-qhr8m"
Sep 10 19:11:23.988245 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:11:23.987933    2546 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 19:11:23.988245 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:11:23.987998    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b579572a-1c79-4730-a112-193b3887bbce-kube-rbac-proxy-cm podName:b579572a-1c79-4730-a112-193b3887bbce nodeName:}" failed. No retries permitted until 2026-09-10 19:11:27.987981638 +0000 UTC m=+51.217031641 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b579572a-1c79-4730-a112-193b3887bbce-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qhr8m" (UID: "b579572a-1c79-4730-a112-193b3887bbce") : configmap "kube-rbac-proxy" not found
Sep 10 19:11:23.988245 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:11:23.988009    2546 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 19:11:23.988245 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:11:23.988048    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b579572a-1c79-4730-a112-193b3887bbce-insights-runtime-extractor-tls podName:b579572a-1c79-4730-a112-193b3887bbce nodeName:}" failed. No retries permitted until 2026-09-10 19:11:27.988036224 +0000 UTC m=+51.217086263 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b579572a-1c79-4730-a112-193b3887bbce-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qhr8m" (UID: "b579572a-1c79-4730-a112-193b3887bbce") : secret "insights-runtime-extractor-tls" not found
Sep 10 19:11:24.506478 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:24.506444    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-4dh9p" event={"ID":"c1122eb6-1973-4c52-8ba4-76dd06f30eab","Type":"ContainerStarted","Data":"9d7d653b9b4180acb000538869a3a8d71ea465ba8a19dc615b7272510455426c"}
Sep 10 19:11:24.795261 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:24.795174    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ac0e549a-a86a-4ac5-8a59-9b0cebc5b7f0-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-97jbx\" (UID: \"ac0e549a-a86a-4ac5-8a59-9b0cebc5b7f0\") " pod="openshift-network-console/networking-console-plugin-79977f9844-97jbx"
Sep 10 19:11:24.795435 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:11:24.795334    2546 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 10 19:11:24.795498 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:11:24.795437    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac0e549a-a86a-4ac5-8a59-9b0cebc5b7f0-networking-console-plugin-cert podName:ac0e549a-a86a-4ac5-8a59-9b0cebc5b7f0 nodeName:}" failed. No retries permitted until 2026-09-10 19:11:32.795416881 +0000 UTC m=+56.024466887 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ac0e549a-a86a-4ac5-8a59-9b0cebc5b7f0-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-97jbx" (UID: "ac0e549a-a86a-4ac5-8a59-9b0cebc5b7f0") : secret "networking-console-plugin-cert" not found
Sep 10 19:11:26.408443 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:26.408402    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f05a659a-63af-4897-a74b-e8b246a1a59f-metrics-tls\") pod \"dns-default-m7z8f\" (UID: \"f05a659a-63af-4897-a74b-e8b246a1a59f\") " pod="openshift-dns/dns-default-m7z8f"
Sep 10 19:11:26.408858 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:26.408462    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f5fc406b-f790-4121-98c2-af351ed63401-registry-tls\") pod \"image-registry-76cdd6f759-kq486\" (UID: \"f5fc406b-f790-4121-98c2-af351ed63401\") " pod="openshift-image-registry/image-registry-76cdd6f759-kq486"
Sep 10 19:11:26.408858 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:11:26.408556    2546 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 19:11:26.408858 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:11:26.408578    2546 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 19:11:26.408858 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:11:26.408591    2546 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76cdd6f759-kq486: secret "image-registry-tls" not found
Sep 10 19:11:26.408858 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:11:26.408631    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f05a659a-63af-4897-a74b-e8b246a1a59f-metrics-tls podName:f05a659a-63af-4897-a74b-e8b246a1a59f nodeName:}" failed. No retries permitted until 2026-09-10 19:11:42.408614941 +0000 UTC m=+65.637664957 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f05a659a-63af-4897-a74b-e8b246a1a59f-metrics-tls") pod "dns-default-m7z8f" (UID: "f05a659a-63af-4897-a74b-e8b246a1a59f") : secret "dns-default-metrics-tls" not found
Sep 10 19:11:26.408858 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:11:26.408646    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f5fc406b-f790-4121-98c2-af351ed63401-registry-tls podName:f5fc406b-f790-4121-98c2-af351ed63401 nodeName:}" failed. No retries permitted until 2026-09-10 19:11:42.408640138 +0000 UTC m=+65.637690141 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f5fc406b-f790-4121-98c2-af351ed63401-registry-tls") pod "image-registry-76cdd6f759-kq486" (UID: "f5fc406b-f790-4121-98c2-af351ed63401") : secret "image-registry-tls" not found
Sep 10 19:11:27.515292 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:27.515251    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-4dh9p" event={"ID":"c1122eb6-1973-4c52-8ba4-76dd06f30eab","Type":"ContainerStarted","Data":"6553a3e04c35ca74597446f5a316dc3662b6fdc3d0247bf2a67ce8adc4cb1dd3"}
Sep 10 19:11:27.534183 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:27.534134    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-75d99c484-4dh9p" podStartSLOduration=1.745379072 podStartE2EDuration="4.534118119s" totalImagesPullingTime="2.788739047s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:11:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:11:27.53344471 +0000 UTC m=+50.762494736" watchObservedRunningTime="2026-09-10 19:11:27.534118119 +0000 UTC m=+50.763168143"
Sep 10 19:11:28.021911 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:28.021868    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b579572a-1c79-4730-a112-193b3887bbce-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qhr8m\" (UID: \"b579572a-1c79-4730-a112-193b3887bbce\") " pod="openshift-insights/insights-runtime-extractor-qhr8m"
Sep 10 19:11:28.022112 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:28.021940    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b579572a-1c79-4730-a112-193b3887bbce-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qhr8m\" (UID: \"b579572a-1c79-4730-a112-193b3887bbce\") " pod="openshift-insights/insights-runtime-extractor-qhr8m"
Sep 10 19:11:28.022777 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:11:28.022286    2546 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 19:11:28.022777 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:11:28.022402    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b579572a-1c79-4730-a112-193b3887bbce-insights-runtime-extractor-tls podName:b579572a-1c79-4730-a112-193b3887bbce nodeName:}" failed. No retries permitted until 2026-09-10 19:11:36.022379547 +0000 UTC m=+59.251429556 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b579572a-1c79-4730-a112-193b3887bbce-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qhr8m" (UID: "b579572a-1c79-4730-a112-193b3887bbce") : secret "insights-runtime-extractor-tls" not found
Sep 10 19:11:28.022777 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:28.022578    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b579572a-1c79-4730-a112-193b3887bbce-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qhr8m\" (UID: \"b579572a-1c79-4730-a112-193b3887bbce\") " pod="openshift-insights/insights-runtime-extractor-qhr8m"
Sep 10 19:11:32.861720 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:32.861682    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ac0e549a-a86a-4ac5-8a59-9b0cebc5b7f0-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-97jbx\" (UID: \"ac0e549a-a86a-4ac5-8a59-9b0cebc5b7f0\") " pod="openshift-network-console/networking-console-plugin-79977f9844-97jbx"
Sep 10 19:11:32.862204 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:11:32.861799    2546 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 10 19:11:32.862204 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:11:32.861851    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac0e549a-a86a-4ac5-8a59-9b0cebc5b7f0-networking-console-plugin-cert podName:ac0e549a-a86a-4ac5-8a59-9b0cebc5b7f0 nodeName:}" failed. No retries permitted until 2026-09-10 19:11:48.861836772 +0000 UTC m=+72.090886780 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ac0e549a-a86a-4ac5-8a59-9b0cebc5b7f0-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-97jbx" (UID: "ac0e549a-a86a-4ac5-8a59-9b0cebc5b7f0") : secret "networking-console-plugin-cert" not found
Sep 10 19:11:36.087402 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:36.087364    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b579572a-1c79-4730-a112-193b3887bbce-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qhr8m\" (UID: \"b579572a-1c79-4730-a112-193b3887bbce\") " pod="openshift-insights/insights-runtime-extractor-qhr8m"
Sep 10 19:11:36.089693 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:36.089666    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b579572a-1c79-4730-a112-193b3887bbce-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qhr8m\" (UID: \"b579572a-1c79-4730-a112-193b3887bbce\") " pod="openshift-insights/insights-runtime-extractor-qhr8m"
Sep 10 19:11:36.093549 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:36.093530    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-qhr8m"
Sep 10 19:11:36.363765 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:36.363731    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-qhr8m"]
Sep 10 19:11:36.367714 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:11:36.367682    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb579572a_1c79_4730_a112_193b3887bbce.slice/crio-25071c27ab95489ad23bf6a98fdea4b0dd1b87fcdfec1677f195dc7978327f10 WatchSource:0}: Error finding container 25071c27ab95489ad23bf6a98fdea4b0dd1b87fcdfec1677f195dc7978327f10: Status 404 returned error can't find the container with id 25071c27ab95489ad23bf6a98fdea4b0dd1b87fcdfec1677f195dc7978327f10
Sep 10 19:11:36.454785 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:36.454758    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5k9nq"
Sep 10 19:11:36.540726 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:36.540684    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qhr8m" event={"ID":"b579572a-1c79-4730-a112-193b3887bbce","Type":"ContainerStarted","Data":"e5943a260d9e26f7dfb15d9dde72bf7925a3a454368a8139cee3c573d356de5f"}
Sep 10 19:11:36.540892 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:36.540760    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qhr8m" event={"ID":"b579572a-1c79-4730-a112-193b3887bbce","Type":"ContainerStarted","Data":"25071c27ab95489ad23bf6a98fdea4b0dd1b87fcdfec1677f195dc7978327f10"}
Sep 10 19:11:37.546098 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:37.546059    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qhr8m" event={"ID":"b579572a-1c79-4730-a112-193b3887bbce","Type":"ContainerStarted","Data":"967a6a022c7a4fc32c971675efcf4dc7f18f46808e44e9f02326a3417fb1af48"}
Sep 10 19:11:39.555672 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:39.555631    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qhr8m" event={"ID":"b579572a-1c79-4730-a112-193b3887bbce","Type":"ContainerStarted","Data":"68f7dbce022e459311d5e078fdc814c128ec3c5a2577aeaaec2170280cb96d55"}
Sep 10 19:11:39.574811 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:39.574762    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-qhr8m" podStartSLOduration=17.064498604 podStartE2EDuration="19.5747465s" totalImagesPullingTime="2.510247896s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 19:11:20 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:11:39.573269149 +0000 UTC m=+62.802319173" watchObservedRunningTime="2026-09-10 19:11:39.5747465 +0000 UTC m=+62.803796526"
Sep 10 19:11:42.436879 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:42.436838    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f05a659a-63af-4897-a74b-e8b246a1a59f-metrics-tls\") pod \"dns-default-m7z8f\" (UID: \"f05a659a-63af-4897-a74b-e8b246a1a59f\") " pod="openshift-dns/dns-default-m7z8f"
Sep 10 19:11:42.437366 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:42.436900    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f5fc406b-f790-4121-98c2-af351ed63401-registry-tls\") pod \"image-registry-76cdd6f759-kq486\" (UID: \"f5fc406b-f790-4121-98c2-af351ed63401\") " pod="openshift-image-registry/image-registry-76cdd6f759-kq486"
Sep 10 19:11:42.439314 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:42.439290    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f5fc406b-f790-4121-98c2-af351ed63401-registry-tls\") pod \"image-registry-76cdd6f759-kq486\" (UID: \"f5fc406b-f790-4121-98c2-af351ed63401\") " pod="openshift-image-registry/image-registry-76cdd6f759-kq486"
Sep 10 19:11:42.439388 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:42.439298    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f05a659a-63af-4897-a74b-e8b246a1a59f-metrics-tls\") pod \"dns-default-m7z8f\" (UID: \"f05a659a-63af-4897-a74b-e8b246a1a59f\") " pod="openshift-dns/dns-default-m7z8f"
Sep 10 19:11:42.679045 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:42.679012    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-8474t\""
Sep 10 19:11:42.686725 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:42.686706    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-76cdd6f759-kq486"
Sep 10 19:11:42.687779 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:42.687739    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-b59cd\""
Sep 10 19:11:42.695336 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:42.695318    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-m7z8f"
Sep 10 19:11:42.940912 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:42.940818    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/63940132-1151-4ec6-b8fd-a68baa948a23-original-pull-secret\") pod \"global-pull-secret-syncer-qhq2q\" (UID: \"63940132-1151-4ec6-b8fd-a68baa948a23\") " pod="kube-system/global-pull-secret-syncer-qhq2q"
Sep 10 19:11:42.945779 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:42.945758    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 10 19:11:42.954040 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:42.954012    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/63940132-1151-4ec6-b8fd-a68baa948a23-original-pull-secret\") pod \"global-pull-secret-syncer-qhq2q\" (UID: \"63940132-1151-4ec6-b8fd-a68baa948a23\") " pod="kube-system/global-pull-secret-syncer-qhq2q"
Sep 10 19:11:42.982051 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:42.982015    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-76cdd6f759-kq486"]
Sep 10 19:11:42.985597 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:11:42.985570    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5fc406b_f790_4121_98c2_af351ed63401.slice/crio-8bdad1039c745fa016b1fee599cc1655ab2b4456dfeffb89c108514d8662fce6 WatchSource:0}: Error finding container 8bdad1039c745fa016b1fee599cc1655ab2b4456dfeffb89c108514d8662fce6: Status 404 returned error can't find the container with id 8bdad1039c745fa016b1fee599cc1655ab2b4456dfeffb89c108514d8662fce6
Sep 10 19:11:43.001770 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:43.001744    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-m7z8f"]
Sep 10 19:11:43.004947 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:11:43.004921    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf05a659a_63af_4897_a74b_e8b246a1a59f.slice/crio-efa8ed19d59e8b8d1f2278bad5d6f7c0a1ee85fddfaf82114fa939238003cafe WatchSource:0}: Error finding container efa8ed19d59e8b8d1f2278bad5d6f7c0a1ee85fddfaf82114fa939238003cafe: Status 404 returned error can't find the container with id efa8ed19d59e8b8d1f2278bad5d6f7c0a1ee85fddfaf82114fa939238003cafe
Sep 10 19:11:43.042259 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:43.042235    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50ce76d4-2a8b-48b6-8939-df2ca2f0a802-metrics-certs\") pod \"network-metrics-daemon-7pg6j\" (UID: \"50ce76d4-2a8b-48b6-8939-df2ca2f0a802\") " pod="openshift-multus/network-metrics-daemon-7pg6j"
Sep 10 19:11:43.042401 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:43.042278    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxpc6\" (UniqueName: \"kubernetes.io/projected/3ddbe88b-0b64-4363-9803-cd0abd242bdc-kube-api-access-nxpc6\") pod \"network-check-target-tjtbz\" (UID: \"3ddbe88b-0b64-4363-9803-cd0abd242bdc\") " pod="openshift-network-diagnostics/network-check-target-tjtbz"
Sep 10 19:11:43.046927 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:43.046906    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 10 19:11:43.047044 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:43.046942    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 10 19:11:43.054735 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:43.054716    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50ce76d4-2a8b-48b6-8939-df2ca2f0a802-metrics-certs\") pod \"network-metrics-daemon-7pg6j\" (UID: \"50ce76d4-2a8b-48b6-8939-df2ca2f0a802\") " pod="openshift-multus/network-metrics-daemon-7pg6j"
Sep 10 19:11:43.055439 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:43.055422    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 10 19:11:43.065049 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:43.065025    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxpc6\" (UniqueName: \"kubernetes.io/projected/3ddbe88b-0b64-4363-9803-cd0abd242bdc-kube-api-access-nxpc6\") pod \"network-check-target-tjtbz\" (UID: \"3ddbe88b-0b64-4363-9803-cd0abd242bdc\") " pod="openshift-network-diagnostics/network-check-target-tjtbz"
Sep 10 19:11:43.086057 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:43.086028    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qhq2q"
Sep 10 19:11:43.094034 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:43.094008    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-r8gdr\""
Sep 10 19:11:43.098500 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:43.098481    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-zj5gq\""
Sep 10 19:11:43.101475 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:43.101457    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7pg6j"
Sep 10 19:11:43.106797 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:43.106783    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tjtbz"
Sep 10 19:11:43.373573 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:43.373543    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qhq2q"]
Sep 10 19:11:43.375988 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:11:43.375953    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63940132_1151_4ec6_b8fd_a68baa948a23.slice/crio-3ed2882eb90ea28ca1d20a3f796a934d7687e8cbe42e67bc096d09beffce8410 WatchSource:0}: Error finding container 3ed2882eb90ea28ca1d20a3f796a934d7687e8cbe42e67bc096d09beffce8410: Status 404 returned error can't find the container with id 3ed2882eb90ea28ca1d20a3f796a934d7687e8cbe42e67bc096d09beffce8410
Sep 10 19:11:43.396736 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:43.396711    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7pg6j"]
Sep 10 19:11:43.400441 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:11:43.400407    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50ce76d4_2a8b_48b6_8939_df2ca2f0a802.slice/crio-8708e8ec63a4ca657d2acd0bc0d9b3f4b9f0022e2fcbafacdf6faacee711bcde WatchSource:0}: Error finding container 8708e8ec63a4ca657d2acd0bc0d9b3f4b9f0022e2fcbafacdf6faacee711bcde: Status 404 returned error can't find the container with id 8708e8ec63a4ca657d2acd0bc0d9b3f4b9f0022e2fcbafacdf6faacee711bcde
Sep 10 19:11:43.416997 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:43.416972    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tjtbz"]
Sep 10 19:11:43.419680 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:11:43.419647    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ddbe88b_0b64_4363_9803_cd0abd242bdc.slice/crio-b0f4895547f3d700d949b059db37234181ac9001df12b1432be5fbdaf79bfdb3 WatchSource:0}: Error finding container b0f4895547f3d700d949b059db37234181ac9001df12b1432be5fbdaf79bfdb3: Status 404 returned error can't find the container with id b0f4895547f3d700d949b059db37234181ac9001df12b1432be5fbdaf79bfdb3
Sep 10 19:11:43.569428 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:43.569323    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7pg6j" event={"ID":"50ce76d4-2a8b-48b6-8939-df2ca2f0a802","Type":"ContainerStarted","Data":"8708e8ec63a4ca657d2acd0bc0d9b3f4b9f0022e2fcbafacdf6faacee711bcde"}
Sep 10 19:11:43.570850 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:43.570822    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-76cdd6f759-kq486" event={"ID":"f5fc406b-f790-4121-98c2-af351ed63401","Type":"ContainerStarted","Data":"3ddc2f69231a108c390a73de3f21e7ab97d2f85590c4e2e19aab63d9627554b2"}
Sep 10 19:11:43.570952 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:43.570869    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-76cdd6f759-kq486" event={"ID":"f5fc406b-f790-4121-98c2-af351ed63401","Type":"ContainerStarted","Data":"8bdad1039c745fa016b1fee599cc1655ab2b4456dfeffb89c108514d8662fce6"}
Sep 10 19:11:43.570952 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:43.570896    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-76cdd6f759-kq486"
Sep 10 19:11:43.572029 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:43.572003    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tjtbz" event={"ID":"3ddbe88b-0b64-4363-9803-cd0abd242bdc","Type":"ContainerStarted","Data":"b0f4895547f3d700d949b059db37234181ac9001df12b1432be5fbdaf79bfdb3"}
Sep 10 19:11:43.573227 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:43.573193    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-qhq2q" event={"ID":"63940132-1151-4ec6-b8fd-a68baa948a23","Type":"ContainerStarted","Data":"3ed2882eb90ea28ca1d20a3f796a934d7687e8cbe42e67bc096d09beffce8410"}
Sep 10 19:11:43.574268 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:43.574247    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-m7z8f" event={"ID":"f05a659a-63af-4897-a74b-e8b246a1a59f","Type":"ContainerStarted","Data":"efa8ed19d59e8b8d1f2278bad5d6f7c0a1ee85fddfaf82114fa939238003cafe"}
Sep 10 19:11:43.591924 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:43.591881    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-76cdd6f759-kq486" podStartSLOduration=46.591868661 podStartE2EDuration="46.591868661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:10:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:11:43.590610084 +0000 UTC m=+66.819660118" watchObservedRunningTime="2026-09-10 19:11:43.591868661 +0000 UTC m=+66.820918701"
Sep 10 19:11:45.584798 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:45.584761    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-m7z8f" event={"ID":"f05a659a-63af-4897-a74b-e8b246a1a59f","Type":"ContainerStarted","Data":"9c080b0f133c1f23c1591da7af4e6c5daa83fb4c8f76e18f0d7f212f7a18aa14"}
Sep 10 19:11:45.585217 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:45.584822    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-m7z8f" event={"ID":"f05a659a-63af-4897-a74b-e8b246a1a59f","Type":"ContainerStarted","Data":"5d0b6cf9b4c43b304e9781916ed5ae3c798fd27a3841c694dbda76f930869069"}
Sep 10 19:11:45.585217 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:45.584917    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-m7z8f"
Sep 10 19:11:45.602922 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:45.602854    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-m7z8f" podStartSLOduration=33.651938616 podStartE2EDuration="35.602834224s" totalImagesPullingTime="1.950895608s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:11:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:11:45.601886068 +0000 UTC m=+68.830936094" watchObservedRunningTime="2026-09-10 19:11:45.602834224 +0000 UTC m=+68.831884249"
Sep 10 19:11:46.590361 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:46.590301    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7pg6j" event={"ID":"50ce76d4-2a8b-48b6-8939-df2ca2f0a802","Type":"ContainerStarted","Data":"0d76f3a749106a0d9f86d24834401925abe2a33e483d24e53c6a7708231c9760"}
Sep 10 19:11:46.710185 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:46.710146    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-777d9c8655-s5kf5"]
Sep 10 19:11:46.712415 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:46.712384    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-777d9c8655-s5kf5"
Sep 10 19:11:46.715592 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:46.715563    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 10 19:11:46.716362 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:46.716111    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 10 19:11:46.716362 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:46.716289    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 10 19:11:46.735234 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:46.735210    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 10 19:11:46.740353 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:46.740313    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 10 19:11:46.742081 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:46.742012    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7854b89bd6-95l6h"]
Sep 10 19:11:46.744085 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:46.744056    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-777d9c8655-s5kf5"]
Sep 10 19:11:46.744184 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:46.744168    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7854b89bd6-95l6h"
Sep 10 19:11:46.754434 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:46.754409    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 10 19:11:46.755550 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:46.755508    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 10 19:11:46.755814 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:46.755795    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 10 19:11:46.756195 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:46.756175    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-6hf7t\""
Sep 10 19:11:46.758823 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:46.758802    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-76cdd6f759-kq486"]
Sep 10 19:11:46.775796 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:46.775765    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/014f5b9b-77d3-4dd1-b6aa-0759b6434ff0-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-777d9c8655-s5kf5\" (UID: \"014f5b9b-77d3-4dd1-b6aa-0759b6434ff0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-777d9c8655-s5kf5"
Sep 10 19:11:46.775932 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:46.775831    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/014f5b9b-77d3-4dd1-b6aa-0759b6434ff0-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-777d9c8655-s5kf5\" (UID: \"014f5b9b-77d3-4dd1-b6aa-0759b6434ff0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-777d9c8655-s5kf5"
Sep 10 19:11:46.775932 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:46.775866    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/014f5b9b-77d3-4dd1-b6aa-0759b6434ff0-hub\") pod \"cluster-proxy-proxy-agent-777d9c8655-s5kf5\" (UID: \"014f5b9b-77d3-4dd1-b6aa-0759b6434ff0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-777d9c8655-s5kf5"
Sep 10 19:11:46.775932 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:46.775903    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kn4b9\" (UniqueName: \"kubernetes.io/projected/6e48c5d5-ac68-46fa-bbb5-1307b0979b16-kube-api-access-kn4b9\") pod \"managed-serviceaccount-addon-agent-7854b89bd6-95l6h\" (UID: \"6e48c5d5-ac68-46fa-bbb5-1307b0979b16\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7854b89bd6-95l6h"
Sep 10 19:11:46.775932 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:46.775928    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/014f5b9b-77d3-4dd1-b6aa-0759b6434ff0-ca\") pod \"cluster-proxy-proxy-agent-777d9c8655-s5kf5\" (UID: \"014f5b9b-77d3-4dd1-b6aa-0759b6434ff0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-777d9c8655-s5kf5"
Sep 10 19:11:46.776116 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:46.776006    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6e48c5d5-ac68-46fa-bbb5-1307b0979b16-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7854b89bd6-95l6h\" (UID: \"6e48c5d5-ac68-46fa-bbb5-1307b0979b16\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7854b89bd6-95l6h"
Sep 10 19:11:46.776116 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:46.776073    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7km84\" (UniqueName: \"kubernetes.io/projected/014f5b9b-77d3-4dd1-b6aa-0759b6434ff0-kube-api-access-7km84\") pod \"cluster-proxy-proxy-agent-777d9c8655-s5kf5\" (UID: \"014f5b9b-77d3-4dd1-b6aa-0759b6434ff0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-777d9c8655-s5kf5"
Sep 10 19:11:46.776199 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:46.776111    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/014f5b9b-77d3-4dd1-b6aa-0759b6434ff0-additional-service-ca\") pod \"cluster-proxy-proxy-agent-777d9c8655-s5kf5\" (UID: \"014f5b9b-77d3-4dd1-b6aa-0759b6434ff0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-777d9c8655-s5kf5"
Sep 10 19:11:46.780391 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:46.780365    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7854b89bd6-95l6h"]
Sep 10 19:11:46.867137 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:46.867103    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-b4fdbd8dd-5xwv4"]
Sep 10 19:11:46.869321 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:46.869287    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b4fdbd8dd-5xwv4"
Sep 10 19:11:46.877072 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:46.877048    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7km84\" (UniqueName: \"kubernetes.io/projected/014f5b9b-77d3-4dd1-b6aa-0759b6434ff0-kube-api-access-7km84\") pod \"cluster-proxy-proxy-agent-777d9c8655-s5kf5\" (UID: \"014f5b9b-77d3-4dd1-b6aa-0759b6434ff0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-777d9c8655-s5kf5"
Sep 10 19:11:46.877185 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:46.877085    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/014f5b9b-77d3-4dd1-b6aa-0759b6434ff0-additional-service-ca\") pod \"cluster-proxy-proxy-agent-777d9c8655-s5kf5\" (UID: \"014f5b9b-77d3-4dd1-b6aa-0759b6434ff0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-777d9c8655-s5kf5"
Sep 10 19:11:46.877185 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:46.877120    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/014f5b9b-77d3-4dd1-b6aa-0759b6434ff0-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-777d9c8655-s5kf5\" (UID: \"014f5b9b-77d3-4dd1-b6aa-0759b6434ff0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-777d9c8655-s5kf5"
Sep 10 19:11:46.877185 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:46.877171    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/014f5b9b-77d3-4dd1-b6aa-0759b6434ff0-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-777d9c8655-s5kf5\" (UID: \"014f5b9b-77d3-4dd1-b6aa-0759b6434ff0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-777d9c8655-s5kf5"
Sep 10 19:11:46.877403 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:46.877380    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/014f5b9b-77d3-4dd1-b6aa-0759b6434ff0-hub\") pod \"cluster-proxy-proxy-agent-777d9c8655-s5kf5\" (UID: \"014f5b9b-77d3-4dd1-b6aa-0759b6434ff0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-777d9c8655-s5kf5"
Sep 10 19:11:46.877473 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:46.877435    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kn4b9\" (UniqueName: \"kubernetes.io/projected/6e48c5d5-ac68-46fa-bbb5-1307b0979b16-kube-api-access-kn4b9\") pod \"managed-serviceaccount-addon-agent-7854b89bd6-95l6h\" (UID: \"6e48c5d5-ac68-46fa-bbb5-1307b0979b16\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7854b89bd6-95l6h"
Sep 10 19:11:46.877473 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:46.877465    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/014f5b9b-77d3-4dd1-b6aa-0759b6434ff0-ca\") pod \"cluster-proxy-proxy-agent-777d9c8655-s5kf5\" (UID: \"014f5b9b-77d3-4dd1-b6aa-0759b6434ff0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-777d9c8655-s5kf5"
Sep 10 19:11:46.877574 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:46.877504    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6e48c5d5-ac68-46fa-bbb5-1307b0979b16-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7854b89bd6-95l6h\" (UID: \"6e48c5d5-ac68-46fa-bbb5-1307b0979b16\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7854b89bd6-95l6h"
Sep 10 19:11:46.877910 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:46.877863    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/014f5b9b-77d3-4dd1-b6aa-0759b6434ff0-additional-service-ca\") pod \"cluster-proxy-proxy-agent-777d9c8655-s5kf5\" (UID: \"014f5b9b-77d3-4dd1-b6aa-0759b6434ff0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-777d9c8655-s5kf5"
Sep 10 19:11:46.880454 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:46.880414    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/014f5b9b-77d3-4dd1-b6aa-0759b6434ff0-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-777d9c8655-s5kf5\" (UID: \"014f5b9b-77d3-4dd1-b6aa-0759b6434ff0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-777d9c8655-s5kf5"
Sep 10 19:11:46.880587 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:46.880422    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/014f5b9b-77d3-4dd1-b6aa-0759b6434ff0-hub\") pod \"cluster-proxy-proxy-agent-777d9c8655-s5kf5\" (UID: \"014f5b9b-77d3-4dd1-b6aa-0759b6434ff0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-777d9c8655-s5kf5"
Sep 10 19:11:46.880587 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:46.880483    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6e48c5d5-ac68-46fa-bbb5-1307b0979b16-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7854b89bd6-95l6h\" (UID: \"6e48c5d5-ac68-46fa-bbb5-1307b0979b16\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7854b89bd6-95l6h"
Sep 10 19:11:46.880853 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:46.880828    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/014f5b9b-77d3-4dd1-b6aa-0759b6434ff0-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-777d9c8655-s5kf5\" (UID: \"014f5b9b-77d3-4dd1-b6aa-0759b6434ff0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-777d9c8655-s5kf5"
Sep 10 19:11:46.880948 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:46.880932    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/014f5b9b-77d3-4dd1-b6aa-0759b6434ff0-ca\") pod \"cluster-proxy-proxy-agent-777d9c8655-s5kf5\" (UID: \"014f5b9b-77d3-4dd1-b6aa-0759b6434ff0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-777d9c8655-s5kf5"
Sep 10 19:11:46.882624 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:46.882602    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-b4fdbd8dd-5xwv4"]
Sep 10 19:11:46.895572 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:46.895522    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7km84\" (UniqueName: \"kubernetes.io/projected/014f5b9b-77d3-4dd1-b6aa-0759b6434ff0-kube-api-access-7km84\") pod \"cluster-proxy-proxy-agent-777d9c8655-s5kf5\" (UID: \"014f5b9b-77d3-4dd1-b6aa-0759b6434ff0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-777d9c8655-s5kf5"
Sep 10 19:11:46.899968 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:46.899916    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kn4b9\" (UniqueName: \"kubernetes.io/projected/6e48c5d5-ac68-46fa-bbb5-1307b0979b16-kube-api-access-kn4b9\") pod \"managed-serviceaccount-addon-agent-7854b89bd6-95l6h\" (UID: \"6e48c5d5-ac68-46fa-bbb5-1307b0979b16\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7854b89bd6-95l6h"
Sep 10 19:11:46.978165 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:46.978130    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ls8b\" (UniqueName: \"kubernetes.io/projected/403c4bb0-0592-40f6-9c80-e1a680b6f480-kube-api-access-6ls8b\") pod \"image-registry-b4fdbd8dd-5xwv4\" (UID: \"403c4bb0-0592-40f6-9c80-e1a680b6f480\") " pod="openshift-image-registry/image-registry-b4fdbd8dd-5xwv4"
Sep 10 19:11:46.978314 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:46.978176    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/403c4bb0-0592-40f6-9c80-e1a680b6f480-trusted-ca\") pod \"image-registry-b4fdbd8dd-5xwv4\" (UID: \"403c4bb0-0592-40f6-9c80-e1a680b6f480\") " pod="openshift-image-registry/image-registry-b4fdbd8dd-5xwv4"
Sep 10 19:11:46.978314 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:46.978236    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/403c4bb0-0592-40f6-9c80-e1a680b6f480-ca-trust-extracted\") pod \"image-registry-b4fdbd8dd-5xwv4\" (UID: \"403c4bb0-0592-40f6-9c80-e1a680b6f480\") " pod="openshift-image-registry/image-registry-b4fdbd8dd-5xwv4"
Sep 10 19:11:46.978314 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:46.978271    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/403c4bb0-0592-40f6-9c80-e1a680b6f480-installation-pull-secrets\") pod \"image-registry-b4fdbd8dd-5xwv4\" (UID: \"403c4bb0-0592-40f6-9c80-e1a680b6f480\") " pod="openshift-image-registry/image-registry-b4fdbd8dd-5xwv4"
Sep 10 19:11:46.978461 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:46.978329    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/403c4bb0-0592-40f6-9c80-e1a680b6f480-image-registry-private-configuration\") pod \"image-registry-b4fdbd8dd-5xwv4\" (UID: \"403c4bb0-0592-40f6-9c80-e1a680b6f480\") " pod="openshift-image-registry/image-registry-b4fdbd8dd-5xwv4"
Sep 10 19:11:46.978461 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:46.978379    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/403c4bb0-0592-40f6-9c80-e1a680b6f480-registry-tls\") pod \"image-registry-b4fdbd8dd-5xwv4\" (UID: \"403c4bb0-0592-40f6-9c80-e1a680b6f480\") " pod="openshift-image-registry/image-registry-b4fdbd8dd-5xwv4"
Sep 10 19:11:46.978461 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:46.978431    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/403c4bb0-0592-40f6-9c80-e1a680b6f480-bound-sa-token\") pod \"image-registry-b4fdbd8dd-5xwv4\" (UID: \"403c4bb0-0592-40f6-9c80-e1a680b6f480\") " pod="openshift-image-registry/image-registry-b4fdbd8dd-5xwv4"
Sep 10 19:11:46.978461 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:46.978456    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/403c4bb0-0592-40f6-9c80-e1a680b6f480-registry-certificates\") pod \"image-registry-b4fdbd8dd-5xwv4\" (UID: \"403c4bb0-0592-40f6-9c80-e1a680b6f480\") " pod="openshift-image-registry/image-registry-b4fdbd8dd-5xwv4"
Sep 10 19:11:47.038648 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:47.038614    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-777d9c8655-s5kf5"
Sep 10 19:11:47.054750 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:47.054720    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7854b89bd6-95l6h"
Sep 10 19:11:47.079312 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:47.079276    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/403c4bb0-0592-40f6-9c80-e1a680b6f480-installation-pull-secrets\") pod \"image-registry-b4fdbd8dd-5xwv4\" (UID: \"403c4bb0-0592-40f6-9c80-e1a680b6f480\") " pod="openshift-image-registry/image-registry-b4fdbd8dd-5xwv4"
Sep 10 19:11:47.079456 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:47.079361    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/403c4bb0-0592-40f6-9c80-e1a680b6f480-image-registry-private-configuration\") pod \"image-registry-b4fdbd8dd-5xwv4\" (UID: \"403c4bb0-0592-40f6-9c80-e1a680b6f480\") " pod="openshift-image-registry/image-registry-b4fdbd8dd-5xwv4"
Sep 10 19:11:47.079456 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:47.079404    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/403c4bb0-0592-40f6-9c80-e1a680b6f480-registry-tls\") pod \"image-registry-b4fdbd8dd-5xwv4\" (UID: \"403c4bb0-0592-40f6-9c80-e1a680b6f480\") " pod="openshift-image-registry/image-registry-b4fdbd8dd-5xwv4"
Sep 10 19:11:47.079456 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:47.079452    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/403c4bb0-0592-40f6-9c80-e1a680b6f480-bound-sa-token\") pod \"image-registry-b4fdbd8dd-5xwv4\" (UID: \"403c4bb0-0592-40f6-9c80-e1a680b6f480\") " pod="openshift-image-registry/image-registry-b4fdbd8dd-5xwv4"
Sep 10 19:11:47.079638 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:47.079485    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/403c4bb0-0592-40f6-9c80-e1a680b6f480-registry-certificates\") pod \"image-registry-b4fdbd8dd-5xwv4\" (UID: \"403c4bb0-0592-40f6-9c80-e1a680b6f480\") " pod="openshift-image-registry/image-registry-b4fdbd8dd-5xwv4"
Sep 10 19:11:47.079638 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:47.079530    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ls8b\" (UniqueName: \"kubernetes.io/projected/403c4bb0-0592-40f6-9c80-e1a680b6f480-kube-api-access-6ls8b\") pod \"image-registry-b4fdbd8dd-5xwv4\" (UID: \"403c4bb0-0592-40f6-9c80-e1a680b6f480\") " pod="openshift-image-registry/image-registry-b4fdbd8dd-5xwv4"
Sep 10 19:11:47.079638 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:47.079552    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/403c4bb0-0592-40f6-9c80-e1a680b6f480-trusted-ca\") pod \"image-registry-b4fdbd8dd-5xwv4\" (UID: \"403c4bb0-0592-40f6-9c80-e1a680b6f480\") " pod="openshift-image-registry/image-registry-b4fdbd8dd-5xwv4"
Sep 10 19:11:47.079638 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:47.079579    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/403c4bb0-0592-40f6-9c80-e1a680b6f480-ca-trust-extracted\") pod \"image-registry-b4fdbd8dd-5xwv4\" (UID: \"403c4bb0-0592-40f6-9c80-e1a680b6f480\") " pod="openshift-image-registry/image-registry-b4fdbd8dd-5xwv4"
Sep 10 19:11:47.081172 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:47.079971    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/403c4bb0-0592-40f6-9c80-e1a680b6f480-ca-trust-extracted\") pod \"image-registry-b4fdbd8dd-5xwv4\" (UID: \"403c4bb0-0592-40f6-9c80-e1a680b6f480\") " pod="openshift-image-registry/image-registry-b4fdbd8dd-5xwv4"
Sep 10 19:11:47.081172 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:47.080967    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/403c4bb0-0592-40f6-9c80-e1a680b6f480-trusted-ca\") pod \"image-registry-b4fdbd8dd-5xwv4\" (UID: \"403c4bb0-0592-40f6-9c80-e1a680b6f480\") " pod="openshift-image-registry/image-registry-b4fdbd8dd-5xwv4"
Sep 10 19:11:47.081172 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:47.080998    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/403c4bb0-0592-40f6-9c80-e1a680b6f480-registry-certificates\") pod \"image-registry-b4fdbd8dd-5xwv4\" (UID: \"403c4bb0-0592-40f6-9c80-e1a680b6f480\") " pod="openshift-image-registry/image-registry-b4fdbd8dd-5xwv4"
Sep 10 19:11:47.082321 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:47.082297    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/403c4bb0-0592-40f6-9c80-e1a680b6f480-installation-pull-secrets\") pod \"image-registry-b4fdbd8dd-5xwv4\" (UID: \"403c4bb0-0592-40f6-9c80-e1a680b6f480\") " pod="openshift-image-registry/image-registry-b4fdbd8dd-5xwv4"
Sep 10 19:11:47.082438 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:47.082411    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/403c4bb0-0592-40f6-9c80-e1a680b6f480-image-registry-private-configuration\") pod \"image-registry-b4fdbd8dd-5xwv4\" (UID: \"403c4bb0-0592-40f6-9c80-e1a680b6f480\") " pod="openshift-image-registry/image-registry-b4fdbd8dd-5xwv4"
Sep 10 19:11:47.082514 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:47.082485    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/403c4bb0-0592-40f6-9c80-e1a680b6f480-registry-tls\") pod \"image-registry-b4fdbd8dd-5xwv4\" (UID: \"403c4bb0-0592-40f6-9c80-e1a680b6f480\") " pod="openshift-image-registry/image-registry-b4fdbd8dd-5xwv4"
Sep 10 19:11:47.087850 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:47.087825    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/403c4bb0-0592-40f6-9c80-e1a680b6f480-bound-sa-token\") pod \"image-registry-b4fdbd8dd-5xwv4\" (UID: \"403c4bb0-0592-40f6-9c80-e1a680b6f480\") " pod="openshift-image-registry/image-registry-b4fdbd8dd-5xwv4"
Sep 10 19:11:47.092580 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:47.092557    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ls8b\" (UniqueName: \"kubernetes.io/projected/403c4bb0-0592-40f6-9c80-e1a680b6f480-kube-api-access-6ls8b\") pod \"image-registry-b4fdbd8dd-5xwv4\" (UID: \"403c4bb0-0592-40f6-9c80-e1a680b6f480\") " pod="openshift-image-registry/image-registry-b4fdbd8dd-5xwv4"
Sep 10 19:11:47.212129 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:47.212040    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b4fdbd8dd-5xwv4"
Sep 10 19:11:48.600073 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:48.599969    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7pg6j" event={"ID":"50ce76d4-2a8b-48b6-8939-df2ca2f0a802","Type":"ContainerStarted","Data":"485b6fff21ac1013ad49c434a42137deb71a35958234984bcaa713ed9042d35b"}
Sep 10 19:11:48.601638 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:48.601603    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tjtbz" event={"ID":"3ddbe88b-0b64-4363-9803-cd0abd242bdc","Type":"ContainerStarted","Data":"9a3839893108822d6cafdd99fcb3bc03447a9d20ad7aa8efc027decea19e9b81"}
Sep 10 19:11:48.601778 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:48.601680    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-tjtbz"
Sep 10 19:11:48.603078 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:48.603057    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-qhq2q" event={"ID":"63940132-1151-4ec6-b8fd-a68baa948a23","Type":"ContainerStarted","Data":"f129c12018bc3200910361fd711d63e1cebc386959219650d45f97745fed318a"}
Sep 10 19:11:48.621441 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:48.621392    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-7pg6j" podStartSLOduration=69.234424094 podStartE2EDuration="1m11.621375455s" totalImagesPullingTime="2.386951361s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:10:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:11:48.619887155 +0000 UTC m=+71.848937183" watchObservedRunningTime="2026-09-10 19:11:48.621375455 +0000 UTC m=+71.850425481"
Sep 10 19:11:48.641466 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:48.641165    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-tjtbz" podStartSLOduration=66.699034446 podStartE2EDuration="1m11.641145302s" totalImagesPullingTime="4.942110856s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:10:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:11:48.640355583 +0000 UTC m=+71.869405608" watchObservedRunningTime="2026-09-10 19:11:48.641145302 +0000 UTC m=+71.870195328"
Sep 10 19:11:48.651361 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:48.651310    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7854b89bd6-95l6h"]
Sep 10 19:11:48.654743 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:11:48.654715    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6e48c5d5_ac68_46fa_bbb5_1307b0979b16.slice/crio-ce5f31a0656df01be7578492b3e8d73754733037d2354d23d76fcabc0965009c WatchSource:0}: Error finding container ce5f31a0656df01be7578492b3e8d73754733037d2354d23d76fcabc0965009c: Status 404 returned error can't find the container with id ce5f31a0656df01be7578492b3e8d73754733037d2354d23d76fcabc0965009c
Sep 10 19:11:48.658247 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:48.658226    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-777d9c8655-s5kf5"]
Sep 10 19:11:48.661388 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:11:48.661293    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod014f5b9b_77d3_4dd1_b6aa_0759b6434ff0.slice/crio-25b2bf444c7775ff3080f8432b455ea96379d85816509247e803a48b7a78874e WatchSource:0}: Error finding container 25b2bf444c7775ff3080f8432b455ea96379d85816509247e803a48b7a78874e: Status 404 returned error can't find the container with id 25b2bf444c7775ff3080f8432b455ea96379d85816509247e803a48b7a78874e
Sep 10 19:11:48.711638 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:48.710003    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-qhq2q" podStartSLOduration=66.702852673 podStartE2EDuration="1m11.709979255s" totalImagesPullingTime="5.007126582s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 19:10:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:11:48.661867239 +0000 UTC m=+71.890917265" watchObservedRunningTime="2026-09-10 19:11:48.709979255 +0000 UTC m=+71.939029282"
Sep 10 19:11:48.711812 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:48.711661    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-b4fdbd8dd-5xwv4"]
Sep 10 19:11:48.716075 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:11:48.716049    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod403c4bb0_0592_40f6_9c80_e1a680b6f480.slice/crio-57d0457b37ac1cbc34628765b9c5f4c47fb3f07cfa9824a57b4b5a90552a84e3 WatchSource:0}: Error finding container 57d0457b37ac1cbc34628765b9c5f4c47fb3f07cfa9824a57b4b5a90552a84e3: Status 404 returned error can't find the container with id 57d0457b37ac1cbc34628765b9c5f4c47fb3f07cfa9824a57b4b5a90552a84e3
Sep 10 19:11:48.896462 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:48.896424    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ac0e549a-a86a-4ac5-8a59-9b0cebc5b7f0-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-97jbx\" (UID: \"ac0e549a-a86a-4ac5-8a59-9b0cebc5b7f0\") " pod="openshift-network-console/networking-console-plugin-79977f9844-97jbx"
Sep 10 19:11:48.898833 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:48.898799    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ac0e549a-a86a-4ac5-8a59-9b0cebc5b7f0-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-97jbx\" (UID: \"ac0e549a-a86a-4ac5-8a59-9b0cebc5b7f0\") " pod="openshift-network-console/networking-console-plugin-79977f9844-97jbx"
Sep 10 19:11:49.112584 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:49.112556    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-6tdn6\""
Sep 10 19:11:49.119765 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:49.119741    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-97jbx"
Sep 10 19:11:49.390554 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:49.390463    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-97jbx"]
Sep 10 19:11:49.394480 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:11:49.394432    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac0e549a_a86a_4ac5_8a59_9b0cebc5b7f0.slice/crio-16fd4bf62f5d7c4dc4d38d9426cfebca5e51d564b56c23ce19df705fb6006627 WatchSource:0}: Error finding container 16fd4bf62f5d7c4dc4d38d9426cfebca5e51d564b56c23ce19df705fb6006627: Status 404 returned error can't find the container with id 16fd4bf62f5d7c4dc4d38d9426cfebca5e51d564b56c23ce19df705fb6006627
Sep 10 19:11:49.616731 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:49.616668    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-b4fdbd8dd-5xwv4" event={"ID":"403c4bb0-0592-40f6-9c80-e1a680b6f480","Type":"ContainerStarted","Data":"0a70f059c56e0e759e183131143d01fb6e94b0e802d23e5fb0aaaedc0184b7bb"}
Sep 10 19:11:49.617186 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:49.616749    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-b4fdbd8dd-5xwv4" event={"ID":"403c4bb0-0592-40f6-9c80-e1a680b6f480","Type":"ContainerStarted","Data":"57d0457b37ac1cbc34628765b9c5f4c47fb3f07cfa9824a57b4b5a90552a84e3"}
Sep 10 19:11:49.617186 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:49.616793    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-b4fdbd8dd-5xwv4"
Sep 10 19:11:49.620097 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:49.620066    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7854b89bd6-95l6h" event={"ID":"6e48c5d5-ac68-46fa-bbb5-1307b0979b16","Type":"ContainerStarted","Data":"ce5f31a0656df01be7578492b3e8d73754733037d2354d23d76fcabc0965009c"}
Sep 10 19:11:49.622786 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:49.622758    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-97jbx" event={"ID":"ac0e549a-a86a-4ac5-8a59-9b0cebc5b7f0","Type":"ContainerStarted","Data":"16fd4bf62f5d7c4dc4d38d9426cfebca5e51d564b56c23ce19df705fb6006627"}
Sep 10 19:11:49.625044 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:49.625022    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-777d9c8655-s5kf5" event={"ID":"014f5b9b-77d3-4dd1-b6aa-0759b6434ff0","Type":"ContainerStarted","Data":"25b2bf444c7775ff3080f8432b455ea96379d85816509247e803a48b7a78874e"}
Sep 10 19:11:49.642455 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:49.641415    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-b4fdbd8dd-5xwv4" podStartSLOduration=3.641393956 podStartE2EDuration="3.641393956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:11:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:11:49.640797911 +0000 UTC m=+72.869847948" watchObservedRunningTime="2026-09-10 19:11:49.641393956 +0000 UTC m=+72.870443979"
Sep 10 19:11:53.644399 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:53.644362    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-97jbx" event={"ID":"ac0e549a-a86a-4ac5-8a59-9b0cebc5b7f0","Type":"ContainerStarted","Data":"b01b6cea46386820256f337d5f091e8fb976bffb425ff90453ad1acc2535add9"}
Sep 10 19:11:53.646485 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:53.646458    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-777d9c8655-s5kf5" event={"ID":"014f5b9b-77d3-4dd1-b6aa-0759b6434ff0","Type":"ContainerStarted","Data":"7211a4bec4c1111c75ee1e518f128c022ddf228eed8519788fc28815b42186d3"}
Sep 10 19:11:53.646620 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:53.646498    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-777d9c8655-s5kf5" event={"ID":"014f5b9b-77d3-4dd1-b6aa-0759b6434ff0","Type":"ContainerStarted","Data":"0297a5815be243a21933b46b022779f2fcfd4369f2da10a2634a44d725536e32"}
Sep 10 19:11:53.646620 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:53.646511    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-777d9c8655-s5kf5" event={"ID":"014f5b9b-77d3-4dd1-b6aa-0759b6434ff0","Type":"ContainerStarted","Data":"986239883ef350811772b8cf004c8801ef8fedfe152e85699ff91dbf80403155"}
Sep 10 19:11:53.647866 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:53.647841    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7854b89bd6-95l6h" event={"ID":"6e48c5d5-ac68-46fa-bbb5-1307b0979b16","Type":"ContainerStarted","Data":"35b9db10d32783d2b57ee55608d3cc48fe2b18aa85eb48913bb4cc2c2d5113fd"}
Sep 10 19:11:53.676008 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:53.675948    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-79977f9844-97jbx" podStartSLOduration=34.239290803 podStartE2EDuration="37.67593376s" totalImagesPullingTime="3.436642957s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:11:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:11:53.675589764 +0000 UTC m=+76.904639789" watchObservedRunningTime="2026-09-10 19:11:53.67593376 +0000 UTC m=+76.904983784"
Sep 10 19:11:53.728161 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:53.728097    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7854b89bd6-95l6h" podStartSLOduration=3.542721342 podStartE2EDuration="7.728076505s" totalImagesPullingTime="4.185355163s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:11:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:11:53.726531948 +0000 UTC m=+76.955581973" watchObservedRunningTime="2026-09-10 19:11:53.728076505 +0000 UTC m=+76.957126533"
Sep 10 19:11:53.750543 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:53.750494    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-777d9c8655-s5kf5" podStartSLOduration=3.5593366189999998 podStartE2EDuration="7.750482298s" totalImagesPullingTime="4.191145679s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:11:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:11:53.749563968 +0000 UTC m=+76.978613994" watchObservedRunningTime="2026-09-10 19:11:53.750482298 +0000 UTC m=+76.979532323"
Sep 10 19:11:55.593054 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:11:55.593020    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-m7z8f"
Sep 10 19:12:00.228827 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:12:00.228663    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-wd2gl"]
Sep 10 19:12:00.234691 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:12:00.234669    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-wd2gl"
Sep 10 19:12:00.237628 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:12:00.237599    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 10 19:12:00.237890 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:12:00.237853    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 10 19:12:00.237977 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:12:00.237938    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 10 19:12:00.238037 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:12:00.237876    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 10 19:12:00.238784 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:12:00.238657    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-4fnns\""
Sep 10 19:12:00.238900 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:12:00.238871    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 10 19:12:00.239083 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:12:00.239068    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 10 19:12:00.378415 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:12:00.378354    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/81bdadb5-2940-4fbd-ae51-5a04b73f89f4-node-exporter-tls\") pod \"node-exporter-wd2gl\" (UID: \"81bdadb5-2940-4fbd-ae51-5a04b73f89f4\") " pod="openshift-monitoring/node-exporter-wd2gl"
Sep 10 19:12:00.378625 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:12:00.378432    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/81bdadb5-2940-4fbd-ae51-5a04b73f89f4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wd2gl\" (UID: \"81bdadb5-2940-4fbd-ae51-5a04b73f89f4\") " pod="openshift-monitoring/node-exporter-wd2gl"
Sep 10 19:12:00.378625 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:12:00.378494    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhccj\" (UniqueName: \"kubernetes.io/projected/81bdadb5-2940-4fbd-ae51-5a04b73f89f4-kube-api-access-rhccj\") pod \"node-exporter-wd2gl\" (UID: \"81bdadb5-2940-4fbd-ae51-5a04b73f89f4\") " pod="openshift-monitoring/node-exporter-wd2gl"
Sep 10 19:12:00.378625 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:12:00.378545    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/81bdadb5-2940-4fbd-ae51-5a04b73f89f4-node-exporter-textfile\") pod \"node-exporter-wd2gl\" (UID: \"81bdadb5-2940-4fbd-ae51-5a04b73f89f4\") " pod="openshift-monitoring/node-exporter-wd2gl"
Sep 10 19:12:00.378625 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:12:00.378580    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/81bdadb5-2940-4fbd-ae51-5a04b73f89f4-root\") pod \"node-exporter-wd2gl\" (UID: \"81bdadb5-2940-4fbd-ae51-5a04b73f89f4\") " pod="openshift-monitoring/node-exporter-wd2gl"
Sep 10 19:12:00.378825 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:12:00.378641    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/81bdadb5-2940-4fbd-ae51-5a04b73f89f4-node-exporter-accelerators-collector-config\") pod \"node-exporter-wd2gl\" (UID: \"81bdadb5-2940-4fbd-ae51-5a04b73f89f4\") " pod="openshift-monitoring/node-exporter-wd2gl"
Sep 10 19:12:00.378825 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:12:00.378699    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/81bdadb5-2940-4fbd-ae51-5a04b73f89f4-sys\") pod \"node-exporter-wd2gl\" (UID: \"81bdadb5-2940-4fbd-ae51-5a04b73f89f4\") " pod="openshift-monitoring/node-exporter-wd2gl"
Sep 10 19:12:00.378825 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:12:00.378748    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/81bdadb5-2940-4fbd-ae51-5a04b73f89f4-node-exporter-wtmp\") pod \"node-exporter-wd2gl\" (UID: \"81bdadb5-2940-4fbd-ae51-5a04b73f89f4\") " pod="openshift-monitoring/node-exporter-wd2gl"
Sep 10 19:12:00.378825 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:12:00.378784    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/81bdadb5-2940-4fbd-ae51-5a04b73f89f4-metrics-client-ca\") pod \"node-exporter-wd2gl\" (UID: \"81bdadb5-2940-4fbd-ae51-5a04b73f89f4\") " pod="openshift-monitoring/node-exporter-wd2gl"
Sep 10 19:12:00.480009 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:12:00.479918    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/81bdadb5-2940-4fbd-ae51-5a04b73f89f4-sys\") pod \"node-exporter-wd2gl\" (UID: \"81bdadb5-2940-4fbd-ae51-5a04b73f89f4\") " pod="openshift-monitoring/node-exporter-wd2gl"
Sep 10 19:12:00.480009 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:12:00.479971    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/81bdadb5-2940-4fbd-ae51-5a04b73f89f4-node-exporter-wtmp\") pod \"node-exporter-wd2gl\" (UID: \"81bdadb5-2940-4fbd-ae51-5a04b73f89f4\") " pod="openshift-monitoring/node-exporter-wd2gl"
Sep 10 19:12:00.480219 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:12:00.480011    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/81bdadb5-2940-4fbd-ae51-5a04b73f89f4-metrics-client-ca\") pod \"node-exporter-wd2gl\" (UID: \"81bdadb5-2940-4fbd-ae51-5a04b73f89f4\") " pod="openshift-monitoring/node-exporter-wd2gl"
Sep 10 19:12:00.480219 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:12:00.480038    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/81bdadb5-2940-4fbd-ae51-5a04b73f89f4-node-exporter-tls\") pod \"node-exporter-wd2gl\" (UID: \"81bdadb5-2940-4fbd-ae51-5a04b73f89f4\") " pod="openshift-monitoring/node-exporter-wd2gl"
Sep 10 19:12:00.480219 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:12:00.480044    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/81bdadb5-2940-4fbd-ae51-5a04b73f89f4-sys\") pod \"node-exporter-wd2gl\" (UID: \"81bdadb5-2940-4fbd-ae51-5a04b73f89f4\") " pod="openshift-monitoring/node-exporter-wd2gl"
Sep 10 19:12:00.480219 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:12:00.480076    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/81bdadb5-2940-4fbd-ae51-5a04b73f89f4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wd2gl\" (UID: \"81bdadb5-2940-4fbd-ae51-5a04b73f89f4\") " pod="openshift-monitoring/node-exporter-wd2gl"
Sep 10 19:12:00.480219 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:12:00.480106    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhccj\" (UniqueName: \"kubernetes.io/projected/81bdadb5-2940-4fbd-ae51-5a04b73f89f4-kube-api-access-rhccj\") pod \"node-exporter-wd2gl\" (UID: \"81bdadb5-2940-4fbd-ae51-5a04b73f89f4\") " pod="openshift-monitoring/node-exporter-wd2gl"
Sep 10 19:12:00.480219 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:12:00.480133    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/81bdadb5-2940-4fbd-ae51-5a04b73f89f4-node-exporter-textfile\") pod \"node-exporter-wd2gl\" (UID: \"81bdadb5-2940-4fbd-ae51-5a04b73f89f4\") " pod="openshift-monitoring/node-exporter-wd2gl"
Sep 10 19:12:00.480219 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:12:00.480157    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/81bdadb5-2940-4fbd-ae51-5a04b73f89f4-root\") pod \"node-exporter-wd2gl\" (UID: \"81bdadb5-2940-4fbd-ae51-5a04b73f89f4\") " pod="openshift-monitoring/node-exporter-wd2gl"
Sep 10 19:12:00.480219 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:12:00.480165    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/81bdadb5-2940-4fbd-ae51-5a04b73f89f4-node-exporter-wtmp\") pod \"node-exporter-wd2gl\" (UID: \"81bdadb5-2940-4fbd-ae51-5a04b73f89f4\") " pod="openshift-monitoring/node-exporter-wd2gl"
Sep 10 19:12:00.480219 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:12:00.480192    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/81bdadb5-2940-4fbd-ae51-5a04b73f89f4-node-exporter-accelerators-collector-config\") pod \"node-exporter-wd2gl\" (UID: \"81bdadb5-2940-4fbd-ae51-5a04b73f89f4\") " pod="openshift-monitoring/node-exporter-wd2gl"
Sep 10 19:12:00.480820 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:12:00.480796    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/81bdadb5-2940-4fbd-ae51-5a04b73f89f4-node-exporter-accelerators-collector-config\") pod \"node-exporter-wd2gl\" (UID: \"81bdadb5-2940-4fbd-ae51-5a04b73f89f4\") " pod="openshift-monitoring/node-exporter-wd2gl"
Sep 10 19:12:00.480913 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:12:00.480803    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/81bdadb5-2940-4fbd-ae51-5a04b73f89f4-metrics-client-ca\") pod \"node-exporter-wd2gl\" (UID: \"81bdadb5-2940-4fbd-ae51-5a04b73f89f4\") " pod="openshift-monitoring/node-exporter-wd2gl"
Sep 10 19:12:00.480913 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:12:00.480879    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/81bdadb5-2940-4fbd-ae51-5a04b73f89f4-root\") pod \"node-exporter-wd2gl\" (UID: \"81bdadb5-2940-4fbd-ae51-5a04b73f89f4\") " pod="openshift-monitoring/node-exporter-wd2gl"
Sep 10 19:12:00.481237 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:12:00.481212    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/81bdadb5-2940-4fbd-ae51-5a04b73f89f4-node-exporter-textfile\") pod \"node-exporter-wd2gl\" (UID: \"81bdadb5-2940-4fbd-ae51-5a04b73f89f4\") " pod="openshift-monitoring/node-exporter-wd2gl"
Sep 10 19:12:00.483206 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:12:00.483157    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/81bdadb5-2940-4fbd-ae51-5a04b73f89f4-node-exporter-tls\") pod \"node-exporter-wd2gl\" (UID: \"81bdadb5-2940-4fbd-ae51-5a04b73f89f4\") " pod="openshift-monitoring/node-exporter-wd2gl"
Sep 10 19:12:00.483466 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:12:00.483448    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/81bdadb5-2940-4fbd-ae51-5a04b73f89f4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wd2gl\" (UID: \"81bdadb5-2940-4fbd-ae51-5a04b73f89f4\") " pod="openshift-monitoring/node-exporter-wd2gl"
Sep 10 19:12:00.488899 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:12:00.488880    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhccj\" (UniqueName: \"kubernetes.io/projected/81bdadb5-2940-4fbd-ae51-5a04b73f89f4-kube-api-access-rhccj\") pod \"node-exporter-wd2gl\" (UID: \"81bdadb5-2940-4fbd-ae51-5a04b73f89f4\") " pod="openshift-monitoring/node-exporter-wd2gl"
Sep 10 19:12:00.554990 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:12:00.554954    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-wd2gl"
Sep 10 19:12:00.567808 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:12:00.567780    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81bdadb5_2940_4fbd_ae51_5a04b73f89f4.slice/crio-ac5351c38af035c1f579aa21469281f72d9fb6a57c25104637096961e342b159 WatchSource:0}: Error finding container ac5351c38af035c1f579aa21469281f72d9fb6a57c25104637096961e342b159: Status 404 returned error can't find the container with id ac5351c38af035c1f579aa21469281f72d9fb6a57c25104637096961e342b159
Sep 10 19:12:00.673093 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:12:00.673060    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wd2gl" event={"ID":"81bdadb5-2940-4fbd-ae51-5a04b73f89f4","Type":"ContainerStarted","Data":"ac5351c38af035c1f579aa21469281f72d9fb6a57c25104637096961e342b159"}
Sep 10 19:12:02.681565 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:12:02.681533    2546 generic.go:356] "Generic (PLEG): container finished" podID="81bdadb5-2940-4fbd-ae51-5a04b73f89f4" containerID="f1b16cd29b0910b06638f4811250f5c75b56c53fa678bea7bf79b4a93415bcec" exitCode=0
Sep 10 19:12:02.681935 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:12:02.681604    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wd2gl" event={"ID":"81bdadb5-2940-4fbd-ae51-5a04b73f89f4","Type":"ContainerDied","Data":"f1b16cd29b0910b06638f4811250f5c75b56c53fa678bea7bf79b4a93415bcec"}
Sep 10 19:12:03.687101 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:12:03.687061    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wd2gl" event={"ID":"81bdadb5-2940-4fbd-ae51-5a04b73f89f4","Type":"ContainerStarted","Data":"255f89a1310413db8a60899f5034d13683940b81f8d3a44e0a32de0844739063"}
Sep 10 19:12:03.687615 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:12:03.687128    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wd2gl" event={"ID":"81bdadb5-2940-4fbd-ae51-5a04b73f89f4","Type":"ContainerStarted","Data":"13099c30400a44a9f1fb639ebe27d7e4e1f71b902b827355365c1ff67ac9fb81"}
Sep 10 19:12:03.707945 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:12:03.707894    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-wd2gl" podStartSLOduration=2.4112007650000002 podStartE2EDuration="3.707878438s" totalImagesPullingTime="1.235616076s" totalInitContainerRuntime="61.061597ms" isStatefulPod=true podCreationTimestamp="2026-09-10 19:12:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:12:03.705945326 +0000 UTC m=+86.934995350" watchObservedRunningTime="2026-09-10 19:12:03.707878438 +0000 UTC m=+86.936928462"
Sep 10 19:12:06.764286 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:12:06.764256    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-76cdd6f759-kq486"
Sep 10 19:12:10.634388 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:12:10.634355    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-b4fdbd8dd-5xwv4"
Sep 10 19:12:11.780421 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:12:11.780357    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-76cdd6f759-kq486" podUID="f5fc406b-f790-4121-98c2-af351ed63401" containerName="registry" containerID="cri-o://3ddc2f69231a108c390a73de3f21e7ab97d2f85590c4e2e19aab63d9627554b2" gracePeriod=30
Sep 10 19:12:12.008524 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:12:12.008496    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-76cdd6f759-kq486"
Sep 10 19:12:12.171183 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:12:12.171142    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f5fc406b-f790-4121-98c2-af351ed63401-trusted-ca\") pod \"f5fc406b-f790-4121-98c2-af351ed63401\" (UID: \"f5fc406b-f790-4121-98c2-af351ed63401\") "
Sep 10 19:12:12.171396 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:12:12.171206    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f5fc406b-f790-4121-98c2-af351ed63401-installation-pull-secrets\") pod \"f5fc406b-f790-4121-98c2-af351ed63401\" (UID: \"f5fc406b-f790-4121-98c2-af351ed63401\") "
Sep 10 19:12:12.171396 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:12:12.171241    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f5fc406b-f790-4121-98c2-af351ed63401-registry-certificates\") pod \"f5fc406b-f790-4121-98c2-af351ed63401\" (UID: \"f5fc406b-f790-4121-98c2-af351ed63401\") "
Sep 10 19:12:12.171396 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:12:12.171264    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f5fc406b-f790-4121-98c2-af351ed63401-registry-tls\") pod \"f5fc406b-f790-4121-98c2-af351ed63401\" (UID: \"f5fc406b-f790-4121-98c2-af351ed63401\") "
Sep 10 19:12:12.171396 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:12:12.171290    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f5fc406b-f790-4121-98c2-af351ed63401-ca-trust-extracted\") pod \"f5fc406b-f790-4121-98c2-af351ed63401\" (UID: \"f5fc406b-f790-4121-98c2-af351ed63401\") "
Sep 10 19:12:12.171396 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:12:12.171322    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f5fc406b-f790-4121-98c2-af351ed63401-bound-sa-token\") pod \"f5fc406b-f790-4121-98c2-af351ed63401\" (UID: \"f5fc406b-f790-4121-98c2-af351ed63401\") "
Sep 10 19:12:12.171396 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:12:12.171373    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c55zb\" (UniqueName: \"kubernetes.io/projected/f5fc406b-f790-4121-98c2-af351ed63401-kube-api-access-c55zb\") pod \"f5fc406b-f790-4121-98c2-af351ed63401\" (UID: \"f5fc406b-f790-4121-98c2-af351ed63401\") "
Sep 10 19:12:12.171690 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:12:12.171397    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f5fc406b-f790-4121-98c2-af351ed63401-image-registry-private-configuration\") pod \"f5fc406b-f790-4121-98c2-af351ed63401\" (UID: \"f5fc406b-f790-4121-98c2-af351ed63401\") "
Sep 10 19:12:12.171761 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:12:12.171727    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5fc406b-f790-4121-98c2-af351ed63401-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "f5fc406b-f790-4121-98c2-af351ed63401" (UID: "f5fc406b-f790-4121-98c2-af351ed63401"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 19:12:12.171959 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:12:12.171930    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5fc406b-f790-4121-98c2-af351ed63401-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "f5fc406b-f790-4121-98c2-af351ed63401" (UID: "f5fc406b-f790-4121-98c2-af351ed63401"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 19:12:12.173970 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:12:12.173939    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5fc406b-f790-4121-98c2-af351ed63401-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "f5fc406b-f790-4121-98c2-af351ed63401" (UID: "f5fc406b-f790-4121-98c2-af351ed63401"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 19:12:12.174098 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:12:12.173962    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5fc406b-f790-4121-98c2-af351ed63401-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "f5fc406b-f790-4121-98c2-af351ed63401" (UID: "f5fc406b-f790-4121-98c2-af351ed63401"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 19:12:12.174098 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:12:12.174030    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5fc406b-f790-4121-98c2-af351ed63401-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "f5fc406b-f790-4121-98c2-af351ed63401" (UID: "f5fc406b-f790-4121-98c2-af351ed63401"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 19:12:12.174098 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:12:12.174053    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5fc406b-f790-4121-98c2-af351ed63401-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "f5fc406b-f790-4121-98c2-af351ed63401" (UID: "f5fc406b-f790-4121-98c2-af351ed63401"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 19:12:12.174098 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:12:12.174092    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5fc406b-f790-4121-98c2-af351ed63401-kube-api-access-c55zb" (OuterVolumeSpecName: "kube-api-access-c55zb") pod "f5fc406b-f790-4121-98c2-af351ed63401" (UID: "f5fc406b-f790-4121-98c2-af351ed63401"). InnerVolumeSpecName "kube-api-access-c55zb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 19:12:12.180102 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:12:12.180079    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5fc406b-f790-4121-98c2-af351ed63401-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "f5fc406b-f790-4121-98c2-af351ed63401" (UID: "f5fc406b-f790-4121-98c2-af351ed63401"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:12:12.271986 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:12:12.271952    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c55zb\" (UniqueName: \"kubernetes.io/projected/f5fc406b-f790-4121-98c2-af351ed63401-kube-api-access-c55zb\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:12:12.271986 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:12:12.271980    2546 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f5fc406b-f790-4121-98c2-af351ed63401-image-registry-private-configuration\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:12:12.271986 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:12:12.271992    2546 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f5fc406b-f790-4121-98c2-af351ed63401-trusted-ca\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:12:12.272201 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:12:12.272006    2546 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f5fc406b-f790-4121-98c2-af351ed63401-installation-pull-secrets\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:12:12.272201 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:12:12.272015    2546 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f5fc406b-f790-4121-98c2-af351ed63401-registry-certificates\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:12:12.272201 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:12:12.272024    2546 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f5fc406b-f790-4121-98c2-af351ed63401-registry-tls\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:12:12.272201 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:12:12.272033    2546 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f5fc406b-f790-4121-98c2-af351ed63401-ca-trust-extracted\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:12:12.272201 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:12:12.272040    2546 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f5fc406b-f790-4121-98c2-af351ed63401-bound-sa-token\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:12:12.720576 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:12:12.720536    2546 generic.go:356] "Generic (PLEG): container finished" podID="f5fc406b-f790-4121-98c2-af351ed63401" containerID="3ddc2f69231a108c390a73de3f21e7ab97d2f85590c4e2e19aab63d9627554b2" exitCode=0
Sep 10 19:12:12.720576 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:12:12.720579    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-76cdd6f759-kq486"
Sep 10 19:12:12.720799 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:12:12.720629    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-76cdd6f759-kq486" event={"ID":"f5fc406b-f790-4121-98c2-af351ed63401","Type":"ContainerDied","Data":"3ddc2f69231a108c390a73de3f21e7ab97d2f85590c4e2e19aab63d9627554b2"}
Sep 10 19:12:12.720799 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:12:12.720669    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-76cdd6f759-kq486" event={"ID":"f5fc406b-f790-4121-98c2-af351ed63401","Type":"ContainerDied","Data":"8bdad1039c745fa016b1fee599cc1655ab2b4456dfeffb89c108514d8662fce6"}
Sep 10 19:12:12.720799 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:12:12.720690    2546 scope.go:117] "RemoveContainer" containerID="3ddc2f69231a108c390a73de3f21e7ab97d2f85590c4e2e19aab63d9627554b2"
Sep 10 19:12:12.728672 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:12:12.728653    2546 scope.go:117] "RemoveContainer" containerID="3ddc2f69231a108c390a73de3f21e7ab97d2f85590c4e2e19aab63d9627554b2"
Sep 10 19:12:12.728919 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:12:12.728902    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ddc2f69231a108c390a73de3f21e7ab97d2f85590c4e2e19aab63d9627554b2\": container with ID starting with 3ddc2f69231a108c390a73de3f21e7ab97d2f85590c4e2e19aab63d9627554b2 not found: ID does not exist" containerID="3ddc2f69231a108c390a73de3f21e7ab97d2f85590c4e2e19aab63d9627554b2"
Sep 10 19:12:12.728967 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:12:12.728927    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ddc2f69231a108c390a73de3f21e7ab97d2f85590c4e2e19aab63d9627554b2"} err="failed to get container status \"3ddc2f69231a108c390a73de3f21e7ab97d2f85590c4e2e19aab63d9627554b2\": rpc error: code = NotFound desc = could not find container \"3ddc2f69231a108c390a73de3f21e7ab97d2f85590c4e2e19aab63d9627554b2\": container with ID starting with 3ddc2f69231a108c390a73de3f21e7ab97d2f85590c4e2e19aab63d9627554b2 not found: ID does not exist"
Sep 10 19:12:12.741786 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:12:12.741762    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-76cdd6f759-kq486"]
Sep 10 19:12:12.745360 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:12:12.745323    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-76cdd6f759-kq486"]
Sep 10 19:12:13.278902 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:12:13.278871    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5fc406b-f790-4121-98c2-af351ed63401" path="/var/lib/kubelet/pods/f5fc406b-f790-4121-98c2-af351ed63401/volumes"
Sep 10 19:12:19.627487 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:12:19.627454    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-tjtbz"
Sep 10 19:15:10.302024 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:10.301942    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-d54bdf466-f2gzj"]
Sep 10 19:15:10.302500 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:10.302178    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5fc406b-f790-4121-98c2-af351ed63401" containerName="registry"
Sep 10 19:15:10.302500 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:10.302189    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5fc406b-f790-4121-98c2-af351ed63401" containerName="registry"
Sep 10 19:15:10.302500 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:10.302239    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5fc406b-f790-4121-98c2-af351ed63401" containerName="registry"
Sep 10 19:15:10.304919 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:10.304893    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-d54bdf466-f2gzj"
Sep 10 19:15:10.305143 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:10.305113    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f0396ab1-e6a3-4b4e-8ebc-2f0031109a0f-tmp\") pod \"opendatahub-operator-controller-manager-d54bdf466-f2gzj\" (UID: \"f0396ab1-e6a3-4b4e-8ebc-2f0031109a0f\") " pod="opendatahub/opendatahub-operator-controller-manager-d54bdf466-f2gzj"
Sep 10 19:15:10.305143 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:10.305149    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f0396ab1-e6a3-4b4e-8ebc-2f0031109a0f-manifests\") pod \"opendatahub-operator-controller-manager-d54bdf466-f2gzj\" (UID: \"f0396ab1-e6a3-4b4e-8ebc-2f0031109a0f\") " pod="opendatahub/opendatahub-operator-controller-manager-d54bdf466-f2gzj"
Sep 10 19:15:10.305419 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:10.305187    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f0396ab1-e6a3-4b4e-8ebc-2f0031109a0f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-d54bdf466-f2gzj\" (UID: \"f0396ab1-e6a3-4b4e-8ebc-2f0031109a0f\") " pod="opendatahub/opendatahub-operator-controller-manager-d54bdf466-f2gzj"
Sep 10 19:15:10.305419 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:10.305207    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xz27\" (UniqueName: \"kubernetes.io/projected/f0396ab1-e6a3-4b4e-8ebc-2f0031109a0f-kube-api-access-2xz27\") pod \"opendatahub-operator-controller-manager-d54bdf466-f2gzj\" (UID: \"f0396ab1-e6a3-4b4e-8ebc-2f0031109a0f\") " pod="opendatahub/opendatahub-operator-controller-manager-d54bdf466-f2gzj"
Sep 10 19:15:10.305419 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:10.305317    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f0396ab1-e6a3-4b4e-8ebc-2f0031109a0f-webhook-cert\") pod \"opendatahub-operator-controller-manager-d54bdf466-f2gzj\" (UID: \"f0396ab1-e6a3-4b4e-8ebc-2f0031109a0f\") " pod="opendatahub/opendatahub-operator-controller-manager-d54bdf466-f2gzj"
Sep 10 19:15:10.311144 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:10.311122    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 10 19:15:10.311261 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:10.311165    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 10 19:15:10.311708 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:10.311421    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-574n2\""
Sep 10 19:15:10.311708 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:10.311469    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 10 19:15:10.311803 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:10.311720    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 10 19:15:10.332383 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:10.332334    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-d54bdf466-f2gzj"]
Sep 10 19:15:10.405670 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:10.405641    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f0396ab1-e6a3-4b4e-8ebc-2f0031109a0f-webhook-cert\") pod \"opendatahub-operator-controller-manager-d54bdf466-f2gzj\" (UID: \"f0396ab1-e6a3-4b4e-8ebc-2f0031109a0f\") " pod="opendatahub/opendatahub-operator-controller-manager-d54bdf466-f2gzj"
Sep 10 19:15:10.405670 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:10.405673    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f0396ab1-e6a3-4b4e-8ebc-2f0031109a0f-tmp\") pod \"opendatahub-operator-controller-manager-d54bdf466-f2gzj\" (UID: \"f0396ab1-e6a3-4b4e-8ebc-2f0031109a0f\") " pod="opendatahub/opendatahub-operator-controller-manager-d54bdf466-f2gzj"
Sep 10 19:15:10.405883 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:10.405691    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f0396ab1-e6a3-4b4e-8ebc-2f0031109a0f-manifests\") pod \"opendatahub-operator-controller-manager-d54bdf466-f2gzj\" (UID: \"f0396ab1-e6a3-4b4e-8ebc-2f0031109a0f\") " pod="opendatahub/opendatahub-operator-controller-manager-d54bdf466-f2gzj"
Sep 10 19:15:10.405883 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:10.405715    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f0396ab1-e6a3-4b4e-8ebc-2f0031109a0f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-d54bdf466-f2gzj\" (UID: \"f0396ab1-e6a3-4b4e-8ebc-2f0031109a0f\") " pod="opendatahub/opendatahub-operator-controller-manager-d54bdf466-f2gzj"
Sep 10 19:15:10.405883 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:10.405740    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xz27\" (UniqueName: \"kubernetes.io/projected/f0396ab1-e6a3-4b4e-8ebc-2f0031109a0f-kube-api-access-2xz27\") pod \"opendatahub-operator-controller-manager-d54bdf466-f2gzj\" (UID: \"f0396ab1-e6a3-4b4e-8ebc-2f0031109a0f\") " pod="opendatahub/opendatahub-operator-controller-manager-d54bdf466-f2gzj"
Sep 10 19:15:10.406166 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:10.406134    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f0396ab1-e6a3-4b4e-8ebc-2f0031109a0f-manifests\") pod \"opendatahub-operator-controller-manager-d54bdf466-f2gzj\" (UID: \"f0396ab1-e6a3-4b4e-8ebc-2f0031109a0f\") " pod="opendatahub/opendatahub-operator-controller-manager-d54bdf466-f2gzj"
Sep 10 19:15:10.407967 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:10.407943    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f0396ab1-e6a3-4b4e-8ebc-2f0031109a0f-tmp\") pod \"opendatahub-operator-controller-manager-d54bdf466-f2gzj\" (UID: \"f0396ab1-e6a3-4b4e-8ebc-2f0031109a0f\") " pod="opendatahub/opendatahub-operator-controller-manager-d54bdf466-f2gzj"
Sep 10 19:15:10.408184 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:10.408165    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f0396ab1-e6a3-4b4e-8ebc-2f0031109a0f-webhook-cert\") pod \"opendatahub-operator-controller-manager-d54bdf466-f2gzj\" (UID: \"f0396ab1-e6a3-4b4e-8ebc-2f0031109a0f\") " pod="opendatahub/opendatahub-operator-controller-manager-d54bdf466-f2gzj"
Sep 10 19:15:10.408263 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:10.408243    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f0396ab1-e6a3-4b4e-8ebc-2f0031109a0f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-d54bdf466-f2gzj\" (UID: \"f0396ab1-e6a3-4b4e-8ebc-2f0031109a0f\") " pod="opendatahub/opendatahub-operator-controller-manager-d54bdf466-f2gzj"
Sep 10 19:15:10.428633 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:10.428597    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xz27\" (UniqueName: \"kubernetes.io/projected/f0396ab1-e6a3-4b4e-8ebc-2f0031109a0f-kube-api-access-2xz27\") pod \"opendatahub-operator-controller-manager-d54bdf466-f2gzj\" (UID: \"f0396ab1-e6a3-4b4e-8ebc-2f0031109a0f\") " pod="opendatahub/opendatahub-operator-controller-manager-d54bdf466-f2gzj"
Sep 10 19:15:10.615424 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:10.615394    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-d54bdf466-f2gzj"
Sep 10 19:15:10.906321 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:10.906278    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-d54bdf466-f2gzj"]
Sep 10 19:15:10.908952 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:15:10.908927    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0396ab1_e6a3_4b4e_8ebc_2f0031109a0f.slice/crio-1debf9587d2cf424f276b22703436d0a5b6d742ae001b2efd152cd3898e27959 WatchSource:0}: Error finding container 1debf9587d2cf424f276b22703436d0a5b6d742ae001b2efd152cd3898e27959: Status 404 returned error can't find the container with id 1debf9587d2cf424f276b22703436d0a5b6d742ae001b2efd152cd3898e27959
Sep 10 19:15:11.314043 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:11.313959    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-d54bdf466-f2gzj" event={"ID":"f0396ab1-e6a3-4b4e-8ebc-2f0031109a0f","Type":"ContainerStarted","Data":"1debf9587d2cf424f276b22703436d0a5b6d742ae001b2efd152cd3898e27959"}
Sep 10 19:15:15.329683 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:15.329641    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-d54bdf466-f2gzj" event={"ID":"f0396ab1-e6a3-4b4e-8ebc-2f0031109a0f","Type":"ContainerStarted","Data":"5672718def4016dde7751840819b316ca4c2663571aab9bd9ed37f6e1b148df1"}
Sep 10 19:15:16.334645 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:16.334602    2546 generic.go:356] "Generic (PLEG): container finished" podID="f0396ab1-e6a3-4b4e-8ebc-2f0031109a0f" containerID="5672718def4016dde7751840819b316ca4c2663571aab9bd9ed37f6e1b148df1" exitCode=0
Sep 10 19:15:16.335141 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:16.334671    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-d54bdf466-f2gzj" event={"ID":"f0396ab1-e6a3-4b4e-8ebc-2f0031109a0f","Type":"ContainerDied","Data":"5672718def4016dde7751840819b316ca4c2663571aab9bd9ed37f6e1b148df1"}
Sep 10 19:15:17.340381 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:17.340324    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-d54bdf466-f2gzj" event={"ID":"f0396ab1-e6a3-4b4e-8ebc-2f0031109a0f","Type":"ContainerStarted","Data":"a8ae91c4e6c5f355073354a1e745823ad5bca167a7a2c8a0f120f050d04fd5ff"}
Sep 10 19:15:17.340766 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:17.340421    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-d54bdf466-f2gzj"
Sep 10 19:15:17.381601 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:17.381548    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-d54bdf466-f2gzj" podStartSLOduration=2.427409338 podStartE2EDuration="7.381530922s" totalImagesPullingTime="3.865542281s" totalInitContainerRuntime="1.088579303s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:15:10 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:15:17.379745032 +0000 UTC m=+280.608795058" watchObservedRunningTime="2026-09-10 19:15:17.381530922 +0000 UTC m=+280.610580946"
Sep 10 19:15:28.346509 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:28.346477    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-d54bdf466-f2gzj"
Sep 10 19:15:30.621248 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:30.621212    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6b5f7cd94b-d6plt"]
Sep 10 19:15:30.624311 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:30.624292    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6b5f7cd94b-d6plt"
Sep 10 19:15:30.634842 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:30.634818    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 10 19:15:30.635073 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:30.635059    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 10 19:15:30.635483 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:30.635463    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 10 19:15:30.637191 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:30.637172    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 10 19:15:30.645533 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:30.645506    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bda5e467-10e3-458d-b507-91b0772d3d32-tls-certs\") pod \"kube-auth-proxy-6b5f7cd94b-d6plt\" (UID: \"bda5e467-10e3-458d-b507-91b0772d3d32\") " pod="openshift-ingress/kube-auth-proxy-6b5f7cd94b-d6plt"
Sep 10 19:15:30.645657 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:30.645543    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bda5e467-10e3-458d-b507-91b0772d3d32-tmp\") pod \"kube-auth-proxy-6b5f7cd94b-d6plt\" (UID: \"bda5e467-10e3-458d-b507-91b0772d3d32\") " pod="openshift-ingress/kube-auth-proxy-6b5f7cd94b-d6plt"
Sep 10 19:15:30.645657 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:30.645587    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lfm4\" (UniqueName: \"kubernetes.io/projected/bda5e467-10e3-458d-b507-91b0772d3d32-kube-api-access-2lfm4\") pod \"kube-auth-proxy-6b5f7cd94b-d6plt\" (UID: \"bda5e467-10e3-458d-b507-91b0772d3d32\") " pod="openshift-ingress/kube-auth-proxy-6b5f7cd94b-d6plt"
Sep 10 19:15:30.666205 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:30.666175    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6b5f7cd94b-d6plt"]
Sep 10 19:15:30.747059 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:30.747022    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bda5e467-10e3-458d-b507-91b0772d3d32-tls-certs\") pod \"kube-auth-proxy-6b5f7cd94b-d6plt\" (UID: \"bda5e467-10e3-458d-b507-91b0772d3d32\") " pod="openshift-ingress/kube-auth-proxy-6b5f7cd94b-d6plt"
Sep 10 19:15:30.747059 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:30.747059    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bda5e467-10e3-458d-b507-91b0772d3d32-tmp\") pod \"kube-auth-proxy-6b5f7cd94b-d6plt\" (UID: \"bda5e467-10e3-458d-b507-91b0772d3d32\") " pod="openshift-ingress/kube-auth-proxy-6b5f7cd94b-d6plt"
Sep 10 19:15:30.747313 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:15:30.747177    2546 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Sep 10 19:15:30.747313 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:30.747185    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lfm4\" (UniqueName: \"kubernetes.io/projected/bda5e467-10e3-458d-b507-91b0772d3d32-kube-api-access-2lfm4\") pod \"kube-auth-proxy-6b5f7cd94b-d6plt\" (UID: \"bda5e467-10e3-458d-b507-91b0772d3d32\") " pod="openshift-ingress/kube-auth-proxy-6b5f7cd94b-d6plt"
Sep 10 19:15:30.747313 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:15:30.747261    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bda5e467-10e3-458d-b507-91b0772d3d32-tls-certs podName:bda5e467-10e3-458d-b507-91b0772d3d32 nodeName:}" failed. No retries permitted until 2026-09-10 19:15:31.247239045 +0000 UTC m=+294.476289064 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/bda5e467-10e3-458d-b507-91b0772d3d32-tls-certs") pod "kube-auth-proxy-6b5f7cd94b-d6plt" (UID: "bda5e467-10e3-458d-b507-91b0772d3d32") : secret "kube-auth-proxy-tls" not found
Sep 10 19:15:30.749290 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:30.749268    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bda5e467-10e3-458d-b507-91b0772d3d32-tmp\") pod \"kube-auth-proxy-6b5f7cd94b-d6plt\" (UID: \"bda5e467-10e3-458d-b507-91b0772d3d32\") " pod="openshift-ingress/kube-auth-proxy-6b5f7cd94b-d6plt"
Sep 10 19:15:30.782989 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:30.782956    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lfm4\" (UniqueName: \"kubernetes.io/projected/bda5e467-10e3-458d-b507-91b0772d3d32-kube-api-access-2lfm4\") pod \"kube-auth-proxy-6b5f7cd94b-d6plt\" (UID: \"bda5e467-10e3-458d-b507-91b0772d3d32\") " pod="openshift-ingress/kube-auth-proxy-6b5f7cd94b-d6plt"
Sep 10 19:15:31.251181 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:31.251125    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bda5e467-10e3-458d-b507-91b0772d3d32-tls-certs\") pod \"kube-auth-proxy-6b5f7cd94b-d6plt\" (UID: \"bda5e467-10e3-458d-b507-91b0772d3d32\") " pod="openshift-ingress/kube-auth-proxy-6b5f7cd94b-d6plt"
Sep 10 19:15:31.253679 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:31.253645    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bda5e467-10e3-458d-b507-91b0772d3d32-tls-certs\") pod \"kube-auth-proxy-6b5f7cd94b-d6plt\" (UID: \"bda5e467-10e3-458d-b507-91b0772d3d32\") " pod="openshift-ingress/kube-auth-proxy-6b5f7cd94b-d6plt"
Sep 10 19:15:31.533755 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:31.533643    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6b5f7cd94b-d6plt"
Sep 10 19:15:31.848182 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:31.848155    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6b5f7cd94b-d6plt"]
Sep 10 19:15:31.849952 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:15:31.849924    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbda5e467_10e3_458d_b507_91b0772d3d32.slice/crio-59b9d552f874829dbf23cefc72897fe218fc5ef91435feefb0cea57540cb1311 WatchSource:0}: Error finding container 59b9d552f874829dbf23cefc72897fe218fc5ef91435feefb0cea57540cb1311: Status 404 returned error can't find the container with id 59b9d552f874829dbf23cefc72897fe218fc5ef91435feefb0cea57540cb1311
Sep 10 19:15:32.390771 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:32.390734    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6b5f7cd94b-d6plt" event={"ID":"bda5e467-10e3-458d-b507-91b0772d3d32","Type":"ContainerStarted","Data":"59b9d552f874829dbf23cefc72897fe218fc5ef91435feefb0cea57540cb1311"}
Sep 10 19:15:33.360848 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:33.360811    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x55cx"]
Sep 10 19:15:33.367493 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:33.367464    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x55cx"
Sep 10 19:15:33.371583 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:33.371556    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 10 19:15:33.371583 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:33.371557    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 10 19:15:33.375725 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:33.375698    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 10 19:15:33.375725 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:33.375716    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 10 19:15:33.375968 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:33.375950    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-z4h2z\""
Sep 10 19:15:33.390225 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:33.390192    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x55cx"]
Sep 10 19:15:33.471595 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:33.471451    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44e48482-817e-4b68-9d8d-b48fc13b5c06-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-x55cx\" (UID: \"44e48482-817e-4b68-9d8d-b48fc13b5c06\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x55cx"
Sep 10 19:15:33.471595 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:33.471510    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/44e48482-817e-4b68-9d8d-b48fc13b5c06-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-x55cx\" (UID: \"44e48482-817e-4b68-9d8d-b48fc13b5c06\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x55cx"
Sep 10 19:15:33.471595 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:33.471535    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5b9p6\" (UniqueName: \"kubernetes.io/projected/44e48482-817e-4b68-9d8d-b48fc13b5c06-kube-api-access-5b9p6\") pod \"istiod-openshift-gateway-7b7995d5cd-x55cx\" (UID: \"44e48482-817e-4b68-9d8d-b48fc13b5c06\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x55cx"
Sep 10 19:15:33.471862 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:33.471625    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/44e48482-817e-4b68-9d8d-b48fc13b5c06-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-x55cx\" (UID: \"44e48482-817e-4b68-9d8d-b48fc13b5c06\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x55cx"
Sep 10 19:15:33.471862 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:33.471663    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/44e48482-817e-4b68-9d8d-b48fc13b5c06-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-x55cx\" (UID: \"44e48482-817e-4b68-9d8d-b48fc13b5c06\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x55cx"
Sep 10 19:15:33.471862 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:33.471736    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/44e48482-817e-4b68-9d8d-b48fc13b5c06-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-x55cx\" (UID: \"44e48482-817e-4b68-9d8d-b48fc13b5c06\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x55cx"
Sep 10 19:15:33.471862 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:33.471763    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/44e48482-817e-4b68-9d8d-b48fc13b5c06-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-x55cx\" (UID: \"44e48482-817e-4b68-9d8d-b48fc13b5c06\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x55cx"
Sep 10 19:15:33.572599 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:33.572525    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/44e48482-817e-4b68-9d8d-b48fc13b5c06-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-x55cx\" (UID: \"44e48482-817e-4b68-9d8d-b48fc13b5c06\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x55cx"
Sep 10 19:15:33.572599 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:33.572578    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/44e48482-817e-4b68-9d8d-b48fc13b5c06-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-x55cx\" (UID: \"44e48482-817e-4b68-9d8d-b48fc13b5c06\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x55cx"
Sep 10 19:15:33.572841 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:33.572634    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44e48482-817e-4b68-9d8d-b48fc13b5c06-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-x55cx\" (UID: \"44e48482-817e-4b68-9d8d-b48fc13b5c06\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x55cx"
Sep 10 19:15:33.572841 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:33.572665    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/44e48482-817e-4b68-9d8d-b48fc13b5c06-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-x55cx\" (UID: \"44e48482-817e-4b68-9d8d-b48fc13b5c06\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x55cx"
Sep 10 19:15:33.572841 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:33.572689    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5b9p6\" (UniqueName: \"kubernetes.io/projected/44e48482-817e-4b68-9d8d-b48fc13b5c06-kube-api-access-5b9p6\") pod \"istiod-openshift-gateway-7b7995d5cd-x55cx\" (UID: \"44e48482-817e-4b68-9d8d-b48fc13b5c06\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x55cx"
Sep 10 19:15:33.572841 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:33.572732    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/44e48482-817e-4b68-9d8d-b48fc13b5c06-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-x55cx\" (UID: \"44e48482-817e-4b68-9d8d-b48fc13b5c06\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x55cx"
Sep 10 19:15:33.572841 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:33.572754    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/44e48482-817e-4b68-9d8d-b48fc13b5c06-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-x55cx\" (UID: \"44e48482-817e-4b68-9d8d-b48fc13b5c06\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x55cx"
Sep 10 19:15:33.573673 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:33.573362    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/44e48482-817e-4b68-9d8d-b48fc13b5c06-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-x55cx\" (UID: \"44e48482-817e-4b68-9d8d-b48fc13b5c06\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x55cx"
Sep 10 19:15:33.575655 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:33.575610    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/44e48482-817e-4b68-9d8d-b48fc13b5c06-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-x55cx\" (UID: \"44e48482-817e-4b68-9d8d-b48fc13b5c06\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x55cx"
Sep 10 19:15:33.575765 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:33.575683    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/44e48482-817e-4b68-9d8d-b48fc13b5c06-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-x55cx\" (UID: \"44e48482-817e-4b68-9d8d-b48fc13b5c06\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x55cx"
Sep 10 19:15:33.575765 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:33.575706    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/44e48482-817e-4b68-9d8d-b48fc13b5c06-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-x55cx\" (UID: \"44e48482-817e-4b68-9d8d-b48fc13b5c06\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x55cx"
Sep 10 19:15:33.575895 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:33.575868    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/44e48482-817e-4b68-9d8d-b48fc13b5c06-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-x55cx\" (UID: \"44e48482-817e-4b68-9d8d-b48fc13b5c06\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x55cx"
Sep 10 19:15:33.583771 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:33.583737    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44e48482-817e-4b68-9d8d-b48fc13b5c06-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-x55cx\" (UID: \"44e48482-817e-4b68-9d8d-b48fc13b5c06\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x55cx"
Sep 10 19:15:33.584315 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:33.584295    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5b9p6\" (UniqueName: \"kubernetes.io/projected/44e48482-817e-4b68-9d8d-b48fc13b5c06-kube-api-access-5b9p6\") pod \"istiod-openshift-gateway-7b7995d5cd-x55cx\" (UID: \"44e48482-817e-4b68-9d8d-b48fc13b5c06\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x55cx"
Sep 10 19:15:33.679687 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:33.679606    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x55cx"
Sep 10 19:15:34.283360 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:34.283283    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x55cx"]
Sep 10 19:15:34.769069 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:15:34.769036    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44e48482_817e_4b68_9d8d_b48fc13b5c06.slice/crio-191d690dc4c4942f442b97ce428000f6bd534fec5ba3e713b247ac8600a8c519 WatchSource:0}: Error finding container 191d690dc4c4942f442b97ce428000f6bd534fec5ba3e713b247ac8600a8c519: Status 404 returned error can't find the container with id 191d690dc4c4942f442b97ce428000f6bd534fec5ba3e713b247ac8600a8c519
Sep 10 19:15:35.405958 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:35.405915    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x55cx" event={"ID":"44e48482-817e-4b68-9d8d-b48fc13b5c06","Type":"ContainerStarted","Data":"191d690dc4c4942f442b97ce428000f6bd534fec5ba3e713b247ac8600a8c519"}
Sep 10 19:15:35.407364 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:35.407324    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6b5f7cd94b-d6plt" event={"ID":"bda5e467-10e3-458d-b507-91b0772d3d32","Type":"ContainerStarted","Data":"04622175f44c727764671eaf255be6e01d85bd82d3b858d08cda09be81b7d8aa"}
Sep 10 19:15:35.429240 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:35.429191    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6b5f7cd94b-d6plt" podStartSLOduration=2.460090761 podStartE2EDuration="5.429174289s" totalImagesPullingTime="2.969083528s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:15:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:15:35.428046915 +0000 UTC m=+298.657096940" watchObservedRunningTime="2026-09-10 19:15:35.429174289 +0000 UTC m=+298.658224313"
Sep 10 19:15:37.386820 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:37.386795    2546 kubelet.go:1648] "Image garbage collection succeeded"
Sep 10 19:15:37.422883 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:37.422845    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 19:15:37.422953 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:37.422918    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 19:15:38.420401 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:38.420366    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x55cx" event={"ID":"44e48482-817e-4b68-9d8d-b48fc13b5c06","Type":"ContainerStarted","Data":"5b3d047a2c7ab5e1128070845df925c9ce42964cec406626dc54a10c713fbe29"}
Sep 10 19:15:38.420773 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:38.420574    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x55cx"
Sep 10 19:15:38.422489 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:38.422460    2546 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-x55cx container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 10 19:15:38.422609 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:38.422517    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x55cx" podUID="44e48482-817e-4b68-9d8d-b48fc13b5c06" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 10 19:15:38.479149 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:38.479067    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x55cx" podStartSLOduration=2.827433983 podStartE2EDuration="5.479049168s" totalImagesPullingTime="2.651615185s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:15:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:15:38.474792696 +0000 UTC m=+301.703842721" watchObservedRunningTime="2026-09-10 19:15:38.479049168 +0000 UTC m=+301.708099183"
Sep 10 19:15:39.425505 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:39.425468    2546 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-x55cx container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 10 19:15:39.425871 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:39.425523    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x55cx" podUID="44e48482-817e-4b68-9d8d-b48fc13b5c06" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 10 19:15:42.306142 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:42.306106    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-jsgn2"]
Sep 10 19:15:42.309411 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:42.309390    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-jsgn2"
Sep 10 19:15:42.315974 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:42.315944    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Sep 10 19:15:42.316128 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:42.315950    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-jgfwm\""
Sep 10 19:15:42.334854 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:42.334819    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-jsgn2"]
Sep 10 19:15:42.339763 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:42.339736    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9qrn\" (UniqueName: \"kubernetes.io/projected/f239b2bf-edcf-4852-adc1-c905d4681e29-kube-api-access-z9qrn\") pod \"model-serving-api-54d89bd5fc-jsgn2\" (UID: \"f239b2bf-edcf-4852-adc1-c905d4681e29\") " pod="opendatahub/model-serving-api-54d89bd5fc-jsgn2"
Sep 10 19:15:42.339895 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:42.339786    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f239b2bf-edcf-4852-adc1-c905d4681e29-tls-certs\") pod \"model-serving-api-54d89bd5fc-jsgn2\" (UID: \"f239b2bf-edcf-4852-adc1-c905d4681e29\") " pod="opendatahub/model-serving-api-54d89bd5fc-jsgn2"
Sep 10 19:15:42.426400 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:42.426361    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x55cx"
Sep 10 19:15:42.441030 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:42.441000    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f239b2bf-edcf-4852-adc1-c905d4681e29-tls-certs\") pod \"model-serving-api-54d89bd5fc-jsgn2\" (UID: \"f239b2bf-edcf-4852-adc1-c905d4681e29\") " pod="opendatahub/model-serving-api-54d89bd5fc-jsgn2"
Sep 10 19:15:42.441186 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:42.441078    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9qrn\" (UniqueName: \"kubernetes.io/projected/f239b2bf-edcf-4852-adc1-c905d4681e29-kube-api-access-z9qrn\") pod \"model-serving-api-54d89bd5fc-jsgn2\" (UID: \"f239b2bf-edcf-4852-adc1-c905d4681e29\") " pod="opendatahub/model-serving-api-54d89bd5fc-jsgn2"
Sep 10 19:15:42.441186 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:15:42.441143    2546 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found
Sep 10 19:15:42.441254 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:15:42.441211    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f239b2bf-edcf-4852-adc1-c905d4681e29-tls-certs podName:f239b2bf-edcf-4852-adc1-c905d4681e29 nodeName:}" failed. No retries permitted until 2026-09-10 19:15:42.941197051 +0000 UTC m=+306.170247054 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/f239b2bf-edcf-4852-adc1-c905d4681e29-tls-certs") pod "model-serving-api-54d89bd5fc-jsgn2" (UID: "f239b2bf-edcf-4852-adc1-c905d4681e29") : secret "model-serving-api-tls" not found
Sep 10 19:15:42.452231 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:42.452199    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9qrn\" (UniqueName: \"kubernetes.io/projected/f239b2bf-edcf-4852-adc1-c905d4681e29-kube-api-access-z9qrn\") pod \"model-serving-api-54d89bd5fc-jsgn2\" (UID: \"f239b2bf-edcf-4852-adc1-c905d4681e29\") " pod="opendatahub/model-serving-api-54d89bd5fc-jsgn2"
Sep 10 19:15:42.944888 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:42.944850    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f239b2bf-edcf-4852-adc1-c905d4681e29-tls-certs\") pod \"model-serving-api-54d89bd5fc-jsgn2\" (UID: \"f239b2bf-edcf-4852-adc1-c905d4681e29\") " pod="opendatahub/model-serving-api-54d89bd5fc-jsgn2"
Sep 10 19:15:42.947237 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:42.947213    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f239b2bf-edcf-4852-adc1-c905d4681e29-tls-certs\") pod \"model-serving-api-54d89bd5fc-jsgn2\" (UID: \"f239b2bf-edcf-4852-adc1-c905d4681e29\") " pod="opendatahub/model-serving-api-54d89bd5fc-jsgn2"
Sep 10 19:15:43.219838 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:43.219749    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-jsgn2"
Sep 10 19:15:43.498264 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:43.498187    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-jsgn2"]
Sep 10 19:15:43.501547 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:15:43.501516    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf239b2bf_edcf_4852_adc1_c905d4681e29.slice/crio-2dec94ae66e53b5c8638c48f3587eb6311694f65fdeca7f8eb657ddbf6aa0b8c WatchSource:0}: Error finding container 2dec94ae66e53b5c8638c48f3587eb6311694f65fdeca7f8eb657ddbf6aa0b8c: Status 404 returned error can't find the container with id 2dec94ae66e53b5c8638c48f3587eb6311694f65fdeca7f8eb657ddbf6aa0b8c
Sep 10 19:15:43.503438 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:43.503418    2546 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 19:15:44.444675 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:44.444640    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-jsgn2" event={"ID":"f239b2bf-edcf-4852-adc1-c905d4681e29","Type":"ContainerStarted","Data":"2dec94ae66e53b5c8638c48f3587eb6311694f65fdeca7f8eb657ddbf6aa0b8c"}
Sep 10 19:15:45.449505 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:45.449462    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-jsgn2" event={"ID":"f239b2bf-edcf-4852-adc1-c905d4681e29","Type":"ContainerStarted","Data":"3d90a3175f8547713baebc573a3021a11a132dbcc806f391d2d9597fb0be5c99"}
Sep 10 19:15:45.449967 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:45.449540    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-jsgn2"
Sep 10 19:15:45.469675 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:45.469627    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-jsgn2" podStartSLOduration=2.057401822 podStartE2EDuration="3.469612607s" totalImagesPullingTime="1.412210785s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:15:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:15:45.468978504 +0000 UTC m=+308.698028530" watchObservedRunningTime="2026-09-10 19:15:45.469612607 +0000 UTC m=+308.698662632"
Sep 10 19:15:54.701359 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:54.701310    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-q6cqb"]
Sep 10 19:15:54.704720 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:54.704698    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-q6cqb"
Sep 10 19:15:54.710072 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:54.709927    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Sep 10 19:15:54.710072 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:54.709976    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-qf8x5\""
Sep 10 19:15:54.742282 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:54.742251    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-q6cqb"]
Sep 10 19:15:54.793527 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:54.793490    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-6lcpk"]
Sep 10 19:15:54.796836 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:54.796817    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6lcpk"
Sep 10 19:15:54.806586 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:54.806562    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Sep 10 19:15:54.806714 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:54.806589    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-d9xpx\""
Sep 10 19:15:54.832319 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:54.832289    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-6lcpk"]
Sep 10 19:15:54.841943 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:54.841905    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2llmw\" (UniqueName: \"kubernetes.io/projected/9977ad8b-b590-4004-93b2-7e93892c0b7f-kube-api-access-2llmw\") pod \"kserve-controller-manager-d6f4b65b8-q6cqb\" (UID: \"9977ad8b-b590-4004-93b2-7e93892c0b7f\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-q6cqb"
Sep 10 19:15:54.842108 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:54.841978    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9977ad8b-b590-4004-93b2-7e93892c0b7f-cert\") pod \"kserve-controller-manager-d6f4b65b8-q6cqb\" (UID: \"9977ad8b-b590-4004-93b2-7e93892c0b7f\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-q6cqb"
Sep 10 19:15:54.942837 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:54.942795    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/58903c8d-079b-45c6-9750-717e1fa44405-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-6lcpk\" (UID: \"58903c8d-079b-45c6-9750-717e1fa44405\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6lcpk"
Sep 10 19:15:54.943128 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:54.943102    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fjgh\" (UniqueName: \"kubernetes.io/projected/58903c8d-079b-45c6-9750-717e1fa44405-kube-api-access-4fjgh\") pod \"llmisvc-controller-manager-777c6bd7d5-6lcpk\" (UID: \"58903c8d-079b-45c6-9750-717e1fa44405\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6lcpk"
Sep 10 19:15:54.943295 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:54.943279    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2llmw\" (UniqueName: \"kubernetes.io/projected/9977ad8b-b590-4004-93b2-7e93892c0b7f-kube-api-access-2llmw\") pod \"kserve-controller-manager-d6f4b65b8-q6cqb\" (UID: \"9977ad8b-b590-4004-93b2-7e93892c0b7f\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-q6cqb"
Sep 10 19:15:54.943455 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:54.943438    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9977ad8b-b590-4004-93b2-7e93892c0b7f-cert\") pod \"kserve-controller-manager-d6f4b65b8-q6cqb\" (UID: \"9977ad8b-b590-4004-93b2-7e93892c0b7f\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-q6cqb"
Sep 10 19:15:54.943595 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:15:54.943571    2546 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Sep 10 19:15:54.943675 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:15:54.943661    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9977ad8b-b590-4004-93b2-7e93892c0b7f-cert podName:9977ad8b-b590-4004-93b2-7e93892c0b7f nodeName:}" failed. No retries permitted until 2026-09-10 19:15:55.443638599 +0000 UTC m=+318.672688616 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/9977ad8b-b590-4004-93b2-7e93892c0b7f-cert") pod "kserve-controller-manager-d6f4b65b8-q6cqb" (UID: "9977ad8b-b590-4004-93b2-7e93892c0b7f") : secret "kserve-webhook-server-cert" not found
Sep 10 19:15:54.968484 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:54.968412    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2llmw\" (UniqueName: \"kubernetes.io/projected/9977ad8b-b590-4004-93b2-7e93892c0b7f-kube-api-access-2llmw\") pod \"kserve-controller-manager-d6f4b65b8-q6cqb\" (UID: \"9977ad8b-b590-4004-93b2-7e93892c0b7f\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-q6cqb"
Sep 10 19:15:55.045019 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:55.044978    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/58903c8d-079b-45c6-9750-717e1fa44405-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-6lcpk\" (UID: \"58903c8d-079b-45c6-9750-717e1fa44405\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6lcpk"
Sep 10 19:15:55.045165 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:55.045027    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fjgh\" (UniqueName: \"kubernetes.io/projected/58903c8d-079b-45c6-9750-717e1fa44405-kube-api-access-4fjgh\") pod \"llmisvc-controller-manager-777c6bd7d5-6lcpk\" (UID: \"58903c8d-079b-45c6-9750-717e1fa44405\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6lcpk"
Sep 10 19:15:55.047502 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:55.047477    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/58903c8d-079b-45c6-9750-717e1fa44405-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-6lcpk\" (UID: \"58903c8d-079b-45c6-9750-717e1fa44405\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6lcpk"
Sep 10 19:15:55.058191 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:55.058161    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fjgh\" (UniqueName: \"kubernetes.io/projected/58903c8d-079b-45c6-9750-717e1fa44405-kube-api-access-4fjgh\") pod \"llmisvc-controller-manager-777c6bd7d5-6lcpk\" (UID: \"58903c8d-079b-45c6-9750-717e1fa44405\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6lcpk"
Sep 10 19:15:55.106042 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:55.106008    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6lcpk"
Sep 10 19:15:55.448566 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:55.448532    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9977ad8b-b590-4004-93b2-7e93892c0b7f-cert\") pod \"kserve-controller-manager-d6f4b65b8-q6cqb\" (UID: \"9977ad8b-b590-4004-93b2-7e93892c0b7f\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-q6cqb"
Sep 10 19:15:55.448734 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:55.448715    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-6lcpk"]
Sep 10 19:15:55.450908 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:55.450878    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9977ad8b-b590-4004-93b2-7e93892c0b7f-cert\") pod \"kserve-controller-manager-d6f4b65b8-q6cqb\" (UID: \"9977ad8b-b590-4004-93b2-7e93892c0b7f\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-q6cqb"
Sep 10 19:15:55.457741 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:15:55.457719    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod58903c8d_079b_45c6_9750_717e1fa44405.slice/crio-fbc51b131d4537850a5e4ee8793ce3f6e9296bf19f73a74652d5b25976e0f4ce WatchSource:0}: Error finding container fbc51b131d4537850a5e4ee8793ce3f6e9296bf19f73a74652d5b25976e0f4ce: Status 404 returned error can't find the container with id fbc51b131d4537850a5e4ee8793ce3f6e9296bf19f73a74652d5b25976e0f4ce
Sep 10 19:15:55.488142 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:55.488112    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6lcpk" event={"ID":"58903c8d-079b-45c6-9750-717e1fa44405","Type":"ContainerStarted","Data":"fbc51b131d4537850a5e4ee8793ce3f6e9296bf19f73a74652d5b25976e0f4ce"}
Sep 10 19:15:55.615149 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:55.615117    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-q6cqb"
Sep 10 19:15:55.889379 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:55.889354    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-q6cqb"]
Sep 10 19:15:55.891784 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:15:55.891757    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9977ad8b_b590_4004_93b2_7e93892c0b7f.slice/crio-f525d321717bb3915cdd8e1fa520c7634544a41cd65cae770ace1e97cf4e72d1 WatchSource:0}: Error finding container f525d321717bb3915cdd8e1fa520c7634544a41cd65cae770ace1e97cf4e72d1: Status 404 returned error can't find the container with id f525d321717bb3915cdd8e1fa520c7634544a41cd65cae770ace1e97cf4e72d1
Sep 10 19:15:56.458624 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:56.458595    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-jsgn2"
Sep 10 19:15:56.493938 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:56.493902    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-q6cqb" event={"ID":"9977ad8b-b590-4004-93b2-7e93892c0b7f","Type":"ContainerStarted","Data":"f525d321717bb3915cdd8e1fa520c7634544a41cd65cae770ace1e97cf4e72d1"}
Sep 10 19:15:57.198503 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:57.198473    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x55cx"]
Sep 10 19:15:57.198947 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:57.198741    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x55cx" podUID="44e48482-817e-4b68-9d8d-b48fc13b5c06" containerName="discovery" containerID="cri-o://5b3d047a2c7ab5e1128070845df925c9ce42964cec406626dc54a10c713fbe29" gracePeriod=30
Sep 10 19:15:57.466315 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:57.466281    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x55cx"
Sep 10 19:15:57.500832 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:57.500788    2546 generic.go:356] "Generic (PLEG): container finished" podID="44e48482-817e-4b68-9d8d-b48fc13b5c06" containerID="5b3d047a2c7ab5e1128070845df925c9ce42964cec406626dc54a10c713fbe29" exitCode=0
Sep 10 19:15:57.501012 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:57.500856    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x55cx"
Sep 10 19:15:57.501012 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:57.500859    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x55cx" event={"ID":"44e48482-817e-4b68-9d8d-b48fc13b5c06","Type":"ContainerDied","Data":"5b3d047a2c7ab5e1128070845df925c9ce42964cec406626dc54a10c713fbe29"}
Sep 10 19:15:57.501012 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:57.501003    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x55cx" event={"ID":"44e48482-817e-4b68-9d8d-b48fc13b5c06","Type":"ContainerDied","Data":"191d690dc4c4942f442b97ce428000f6bd534fec5ba3e713b247ac8600a8c519"}
Sep 10 19:15:57.501178 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:57.501027    2546 scope.go:117] "RemoveContainer" containerID="5b3d047a2c7ab5e1128070845df925c9ce42964cec406626dc54a10c713fbe29"
Sep 10 19:15:57.511321 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:57.511302    2546 scope.go:117] "RemoveContainer" containerID="5b3d047a2c7ab5e1128070845df925c9ce42964cec406626dc54a10c713fbe29"
Sep 10 19:15:57.511739 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:15:57.511672    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b3d047a2c7ab5e1128070845df925c9ce42964cec406626dc54a10c713fbe29\": container with ID starting with 5b3d047a2c7ab5e1128070845df925c9ce42964cec406626dc54a10c713fbe29 not found: ID does not exist" containerID="5b3d047a2c7ab5e1128070845df925c9ce42964cec406626dc54a10c713fbe29"
Sep 10 19:15:57.511739 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:57.511713    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b3d047a2c7ab5e1128070845df925c9ce42964cec406626dc54a10c713fbe29"} err="failed to get container status \"5b3d047a2c7ab5e1128070845df925c9ce42964cec406626dc54a10c713fbe29\": rpc error: code = NotFound desc = could not find container \"5b3d047a2c7ab5e1128070845df925c9ce42964cec406626dc54a10c713fbe29\": container with ID starting with 5b3d047a2c7ab5e1128070845df925c9ce42964cec406626dc54a10c713fbe29 not found: ID does not exist"
Sep 10 19:15:57.566646 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:57.566616    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5b9p6\" (UniqueName: \"kubernetes.io/projected/44e48482-817e-4b68-9d8d-b48fc13b5c06-kube-api-access-5b9p6\") pod \"44e48482-817e-4b68-9d8d-b48fc13b5c06\" (UID: \"44e48482-817e-4b68-9d8d-b48fc13b5c06\") "
Sep 10 19:15:57.566646 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:57.566650    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/44e48482-817e-4b68-9d8d-b48fc13b5c06-istio-csr-dns-cert\") pod \"44e48482-817e-4b68-9d8d-b48fc13b5c06\" (UID: \"44e48482-817e-4b68-9d8d-b48fc13b5c06\") "
Sep 10 19:15:57.566865 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:57.566670    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/44e48482-817e-4b68-9d8d-b48fc13b5c06-istio-csr-ca-configmap\") pod \"44e48482-817e-4b68-9d8d-b48fc13b5c06\" (UID: \"44e48482-817e-4b68-9d8d-b48fc13b5c06\") "
Sep 10 19:15:57.566865 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:57.566686    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/44e48482-817e-4b68-9d8d-b48fc13b5c06-istio-kubeconfig\") pod \"44e48482-817e-4b68-9d8d-b48fc13b5c06\" (UID: \"44e48482-817e-4b68-9d8d-b48fc13b5c06\") "
Sep 10 19:15:57.566865 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:57.566740    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/44e48482-817e-4b68-9d8d-b48fc13b5c06-local-certs\") pod \"44e48482-817e-4b68-9d8d-b48fc13b5c06\" (UID: \"44e48482-817e-4b68-9d8d-b48fc13b5c06\") "
Sep 10 19:15:57.566865 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:57.566778    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44e48482-817e-4b68-9d8d-b48fc13b5c06-istio-token\") pod \"44e48482-817e-4b68-9d8d-b48fc13b5c06\" (UID: \"44e48482-817e-4b68-9d8d-b48fc13b5c06\") "
Sep 10 19:15:57.566865 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:57.566833    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/44e48482-817e-4b68-9d8d-b48fc13b5c06-cacerts\") pod \"44e48482-817e-4b68-9d8d-b48fc13b5c06\" (UID: \"44e48482-817e-4b68-9d8d-b48fc13b5c06\") "
Sep 10 19:15:57.567111 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:57.567087    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44e48482-817e-4b68-9d8d-b48fc13b5c06-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "44e48482-817e-4b68-9d8d-b48fc13b5c06" (UID: "44e48482-817e-4b68-9d8d-b48fc13b5c06"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 19:15:57.569624 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:57.569525    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44e48482-817e-4b68-9d8d-b48fc13b5c06-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "44e48482-817e-4b68-9d8d-b48fc13b5c06" (UID: "44e48482-817e-4b68-9d8d-b48fc13b5c06"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 19:15:57.569624 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:57.569600    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44e48482-817e-4b68-9d8d-b48fc13b5c06-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "44e48482-817e-4b68-9d8d-b48fc13b5c06" (UID: "44e48482-817e-4b68-9d8d-b48fc13b5c06"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 19:15:57.569865 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:57.569827    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44e48482-817e-4b68-9d8d-b48fc13b5c06-cacerts" (OuterVolumeSpecName: "cacerts") pod "44e48482-817e-4b68-9d8d-b48fc13b5c06" (UID: "44e48482-817e-4b68-9d8d-b48fc13b5c06"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 19:15:57.569865 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:57.569832    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44e48482-817e-4b68-9d8d-b48fc13b5c06-istio-token" (OuterVolumeSpecName: "istio-token") pod "44e48482-817e-4b68-9d8d-b48fc13b5c06" (UID: "44e48482-817e-4b68-9d8d-b48fc13b5c06"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 19:15:57.569865 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:57.569849    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44e48482-817e-4b68-9d8d-b48fc13b5c06-kube-api-access-5b9p6" (OuterVolumeSpecName: "kube-api-access-5b9p6") pod "44e48482-817e-4b68-9d8d-b48fc13b5c06" (UID: "44e48482-817e-4b68-9d8d-b48fc13b5c06"). InnerVolumeSpecName "kube-api-access-5b9p6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 19:15:57.570060 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:57.569921    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44e48482-817e-4b68-9d8d-b48fc13b5c06-local-certs" (OuterVolumeSpecName: "local-certs") pod "44e48482-817e-4b68-9d8d-b48fc13b5c06" (UID: "44e48482-817e-4b68-9d8d-b48fc13b5c06"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:15:57.667817 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:57.667775    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5b9p6\" (UniqueName: \"kubernetes.io/projected/44e48482-817e-4b68-9d8d-b48fc13b5c06-kube-api-access-5b9p6\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:15:57.667817 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:57.667811    2546 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/44e48482-817e-4b68-9d8d-b48fc13b5c06-istio-csr-dns-cert\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:15:57.667817 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:57.667823    2546 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/44e48482-817e-4b68-9d8d-b48fc13b5c06-istio-csr-ca-configmap\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:15:57.668081 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:57.667834    2546 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/44e48482-817e-4b68-9d8d-b48fc13b5c06-istio-kubeconfig\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:15:57.668081 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:57.667843    2546 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/44e48482-817e-4b68-9d8d-b48fc13b5c06-local-certs\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:15:57.668081 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:57.667851    2546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44e48482-817e-4b68-9d8d-b48fc13b5c06-istio-token\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:15:57.668081 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:57.667858    2546 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/44e48482-817e-4b68-9d8d-b48fc13b5c06-cacerts\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:15:57.845935 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:57.845901    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x55cx"]
Sep 10 19:15:57.847771 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:57.847738    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x55cx"]
Sep 10 19:15:59.280578 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:59.280481    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44e48482-817e-4b68-9d8d-b48fc13b5c06" path="/var/lib/kubelet/pods/44e48482-817e-4b68-9d8d-b48fc13b5c06/volumes"
Sep 10 19:15:59.511755 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:59.511719    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-q6cqb" event={"ID":"9977ad8b-b590-4004-93b2-7e93892c0b7f","Type":"ContainerStarted","Data":"fbec3c00ec36ee2afcca086faef6806c9bae56b6ab5fb529d47ccea6b2a7348e"}
Sep 10 19:15:59.511939 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:59.511784    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-q6cqb"
Sep 10 19:15:59.513179 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:59.513156    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6lcpk" event={"ID":"58903c8d-079b-45c6-9750-717e1fa44405","Type":"ContainerStarted","Data":"1314e6712d8f3bf6d00afeb1963bb7ab7629aa9890a60d7891ee99d78890cec2"}
Sep 10 19:15:59.513274 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:59.513209    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6lcpk"
Sep 10 19:15:59.643651 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:59.643593    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-q6cqb" podStartSLOduration=2.633752083 podStartE2EDuration="5.643575143s" totalImagesPullingTime="3.00982306s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:15:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:15:59.605681304 +0000 UTC m=+322.834731328" watchObservedRunningTime="2026-09-10 19:15:59.643575143 +0000 UTC m=+322.872625168"
Sep 10 19:15:59.649419 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:15:59.649379    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6lcpk" podStartSLOduration=2.202179531 podStartE2EDuration="5.649364204s" totalImagesPullingTime="3.447184673s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:15:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:15:59.648120299 +0000 UTC m=+322.877170327" watchObservedRunningTime="2026-09-10 19:15:59.649364204 +0000 UTC m=+322.878414228"
Sep 10 19:16:02.426496 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:16:02.426440    2546 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-x55cx container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.18:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 10 19:16:02.426895 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:16:02.426523    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x55cx" podUID="44e48482-817e-4b68-9d8d-b48fc13b5c06" containerName="discovery" probeResult="failure" output="Get \"http://10.133.0.18:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 10 19:16:30.519559 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:16:30.519475    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6lcpk"
Sep 10 19:16:30.522649 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:16:30.522628    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-q6cqb"
Sep 10 19:17:01.568582 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:01.568544    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-q6kgf"]
Sep 10 19:17:01.569157 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:01.568972    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44e48482-817e-4b68-9d8d-b48fc13b5c06" containerName="discovery"
Sep 10 19:17:01.569157 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:01.568992    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="44e48482-817e-4b68-9d8d-b48fc13b5c06" containerName="discovery"
Sep 10 19:17:01.569157 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:01.569105    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="44e48482-817e-4b68-9d8d-b48fc13b5c06" containerName="discovery"
Sep 10 19:17:01.572010 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:01.571988    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-q6kgf"
Sep 10 19:17:01.577058 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:01.577029    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 10 19:17:01.577185 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:01.577080    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-89nlp\""
Sep 10 19:17:01.577185 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:01.577173    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 10 19:17:01.596070 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:01.596041    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-q6kgf"]
Sep 10 19:17:01.646804 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:01.646774    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ad379d09-49e3-4540-aded-0f1c0c866e28-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-q6kgf\" (UID: \"ad379d09-49e3-4540-aded-0f1c0c866e28\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-q6kgf"
Sep 10 19:17:01.646959 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:01.646838    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmhwx\" (UniqueName: \"kubernetes.io/projected/ad379d09-49e3-4540-aded-0f1c0c866e28-kube-api-access-wmhwx\") pod \"kuadrant-operator-controller-manager-76f8f686bf-q6kgf\" (UID: \"ad379d09-49e3-4540-aded-0f1c0c866e28\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-q6kgf"
Sep 10 19:17:01.748132 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:01.748091    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmhwx\" (UniqueName: \"kubernetes.io/projected/ad379d09-49e3-4540-aded-0f1c0c866e28-kube-api-access-wmhwx\") pod \"kuadrant-operator-controller-manager-76f8f686bf-q6kgf\" (UID: \"ad379d09-49e3-4540-aded-0f1c0c866e28\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-q6kgf"
Sep 10 19:17:01.748307 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:01.748159    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ad379d09-49e3-4540-aded-0f1c0c866e28-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-q6kgf\" (UID: \"ad379d09-49e3-4540-aded-0f1c0c866e28\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-q6kgf"
Sep 10 19:17:01.748528 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:01.748510    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ad379d09-49e3-4540-aded-0f1c0c866e28-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-q6kgf\" (UID: \"ad379d09-49e3-4540-aded-0f1c0c866e28\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-q6kgf"
Sep 10 19:17:01.770069 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:01.770036    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmhwx\" (UniqueName: \"kubernetes.io/projected/ad379d09-49e3-4540-aded-0f1c0c866e28-kube-api-access-wmhwx\") pod \"kuadrant-operator-controller-manager-76f8f686bf-q6kgf\" (UID: \"ad379d09-49e3-4540-aded-0f1c0c866e28\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-q6kgf"
Sep 10 19:17:01.882283 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:01.882250    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-q6kgf"
Sep 10 19:17:02.189413 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:02.189320    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-q6kgf"]
Sep 10 19:17:02.192363 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:17:02.192314    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad379d09_49e3_4540_aded_0f1c0c866e28.slice/crio-e9f4430edf9f8b9fc8047c111dab20bb4e10e24f31053f4873a3f686376ca46c WatchSource:0}: Error finding container e9f4430edf9f8b9fc8047c111dab20bb4e10e24f31053f4873a3f686376ca46c: Status 404 returned error can't find the container with id e9f4430edf9f8b9fc8047c111dab20bb4e10e24f31053f4873a3f686376ca46c
Sep 10 19:17:02.762702 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:02.762666    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-q6kgf" event={"ID":"ad379d09-49e3-4540-aded-0f1c0c866e28","Type":"ContainerStarted","Data":"e9f4430edf9f8b9fc8047c111dab20bb4e10e24f31053f4873a3f686376ca46c"}
Sep 10 19:17:03.329326 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:03.329290    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-66vvb"]
Sep 10 19:17:03.339010 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:03.338982    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-66vvb"
Sep 10 19:17:03.349398 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:03.348928    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-cvrb7\""
Sep 10 19:17:03.358568 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:03.358502    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-66vvb"]
Sep 10 19:17:03.462533 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:03.462489    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tf6js\" (UniqueName: \"kubernetes.io/projected/062a8b66-7428-42b5-81e3-c4f3c8730d74-kube-api-access-tf6js\") pod \"authorino-operator-76b478bd78-66vvb\" (UID: \"062a8b66-7428-42b5-81e3-c4f3c8730d74\") " pod="kuadrant-system/authorino-operator-76b478bd78-66vvb"
Sep 10 19:17:03.563996 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:03.563958    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tf6js\" (UniqueName: \"kubernetes.io/projected/062a8b66-7428-42b5-81e3-c4f3c8730d74-kube-api-access-tf6js\") pod \"authorino-operator-76b478bd78-66vvb\" (UID: \"062a8b66-7428-42b5-81e3-c4f3c8730d74\") " pod="kuadrant-system/authorino-operator-76b478bd78-66vvb"
Sep 10 19:17:03.574014 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:03.573965    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tf6js\" (UniqueName: \"kubernetes.io/projected/062a8b66-7428-42b5-81e3-c4f3c8730d74-kube-api-access-tf6js\") pod \"authorino-operator-76b478bd78-66vvb\" (UID: \"062a8b66-7428-42b5-81e3-c4f3c8730d74\") " pod="kuadrant-system/authorino-operator-76b478bd78-66vvb"
Sep 10 19:17:03.654393 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:03.654357    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-66vvb"
Sep 10 19:17:03.960868 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:03.960783    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-66vvb"]
Sep 10 19:17:04.541124 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:17:04.541083    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod062a8b66_7428_42b5_81e3_c4f3c8730d74.slice/crio-3e977f8ca5c9d3fdf2310b822fbf9da3069a541155f1ed3f60dbdee46c0d012a WatchSource:0}: Error finding container 3e977f8ca5c9d3fdf2310b822fbf9da3069a541155f1ed3f60dbdee46c0d012a: Status 404 returned error can't find the container with id 3e977f8ca5c9d3fdf2310b822fbf9da3069a541155f1ed3f60dbdee46c0d012a
Sep 10 19:17:04.775540 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:04.775501    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-66vvb" event={"ID":"062a8b66-7428-42b5-81e3-c4f3c8730d74","Type":"ContainerStarted","Data":"3e977f8ca5c9d3fdf2310b822fbf9da3069a541155f1ed3f60dbdee46c0d012a"}
Sep 10 19:17:04.791773 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:04.791686    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-8hdw7"]
Sep 10 19:17:04.795435 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:04.795409    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-8hdw7"
Sep 10 19:17:04.798155 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:04.798129    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-j8r55\""
Sep 10 19:17:04.816100 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:04.816068    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-8hdw7"]
Sep 10 19:17:04.874698 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:04.874659    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5cnl\" (UniqueName: \"kubernetes.io/projected/59396e7b-7b35-40b3-a420-3699f237fcf2-kube-api-access-c5cnl\") pod \"limitador-operator-controller-manager-74495c69f8-8hdw7\" (UID: \"59396e7b-7b35-40b3-a420-3699f237fcf2\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-8hdw7"
Sep 10 19:17:04.975916 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:04.975878    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5cnl\" (UniqueName: \"kubernetes.io/projected/59396e7b-7b35-40b3-a420-3699f237fcf2-kube-api-access-c5cnl\") pod \"limitador-operator-controller-manager-74495c69f8-8hdw7\" (UID: \"59396e7b-7b35-40b3-a420-3699f237fcf2\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-8hdw7"
Sep 10 19:17:04.986733 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:04.986694    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5cnl\" (UniqueName: \"kubernetes.io/projected/59396e7b-7b35-40b3-a420-3699f237fcf2-kube-api-access-c5cnl\") pod \"limitador-operator-controller-manager-74495c69f8-8hdw7\" (UID: \"59396e7b-7b35-40b3-a420-3699f237fcf2\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-8hdw7"
Sep 10 19:17:05.109367 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:05.109325    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-8hdw7"
Sep 10 19:17:06.509242 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:06.509215    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-8hdw7"]
Sep 10 19:17:06.512808 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:17:06.512768    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59396e7b_7b35_40b3_a420_3699f237fcf2.slice/crio-113b7d7fe8c460f59c63a9b1f2e58ee2d21b8d2f3be78613b8ffb3c42de63669 WatchSource:0}: Error finding container 113b7d7fe8c460f59c63a9b1f2e58ee2d21b8d2f3be78613b8ffb3c42de63669: Status 404 returned error can't find the container with id 113b7d7fe8c460f59c63a9b1f2e58ee2d21b8d2f3be78613b8ffb3c42de63669
Sep 10 19:17:06.786520 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:06.786425    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-8hdw7" event={"ID":"59396e7b-7b35-40b3-a420-3699f237fcf2","Type":"ContainerStarted","Data":"113b7d7fe8c460f59c63a9b1f2e58ee2d21b8d2f3be78613b8ffb3c42de63669"}
Sep 10 19:17:06.788447 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:06.788414    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-q6kgf" event={"ID":"ad379d09-49e3-4540-aded-0f1c0c866e28","Type":"ContainerStarted","Data":"e490c503ae2cd546e06531025a136e7b4864d42344a071e4c76c637de0e5731e"}
Sep 10 19:17:06.788719 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:06.788698    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-q6kgf"
Sep 10 19:17:06.824082 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:06.824022    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-q6kgf" podStartSLOduration=1.730724249 podStartE2EDuration="5.824001058s" totalImagesPullingTime="4.093276809s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:17:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:17:06.819155626 +0000 UTC m=+390.048205683" watchObservedRunningTime="2026-09-10 19:17:06.824001058 +0000 UTC m=+390.053051086"
Sep 10 19:17:07.193888 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:07.193851    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-6gwdr"]
Sep 10 19:17:07.196795 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:07.196778    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-6gwdr"
Sep 10 19:17:07.199534 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:07.199512    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-qw5jf\""
Sep 10 19:17:07.201862 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:07.201848    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 10 19:17:07.236572 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:07.236543    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-6gwdr"]
Sep 10 19:17:07.294147 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:07.294080    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxtkf\" (UniqueName: \"kubernetes.io/projected/80fb63b4-18c7-46f3-aac2-110f0ae1efe9-kube-api-access-xxtkf\") pod \"dns-operator-controller-manager-7b84cd7c9c-6gwdr\" (UID: \"80fb63b4-18c7-46f3-aac2-110f0ae1efe9\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-6gwdr"
Sep 10 19:17:07.395248 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:07.395212    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxtkf\" (UniqueName: \"kubernetes.io/projected/80fb63b4-18c7-46f3-aac2-110f0ae1efe9-kube-api-access-xxtkf\") pod \"dns-operator-controller-manager-7b84cd7c9c-6gwdr\" (UID: \"80fb63b4-18c7-46f3-aac2-110f0ae1efe9\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-6gwdr"
Sep 10 19:17:07.405962 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:07.405938    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxtkf\" (UniqueName: \"kubernetes.io/projected/80fb63b4-18c7-46f3-aac2-110f0ae1efe9-kube-api-access-xxtkf\") pod \"dns-operator-controller-manager-7b84cd7c9c-6gwdr\" (UID: \"80fb63b4-18c7-46f3-aac2-110f0ae1efe9\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-6gwdr"
Sep 10 19:17:07.506989 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:07.506952    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-6gwdr"
Sep 10 19:17:07.794390 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:07.794331    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-66vvb" event={"ID":"062a8b66-7428-42b5-81e3-c4f3c8730d74","Type":"ContainerStarted","Data":"1f6739d076de454acc94789fd94f5e2698fa4eb8ab4d1e80e87810618b036081"}
Sep 10 19:17:07.794784 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:07.794572    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-66vvb"
Sep 10 19:17:07.806238 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:07.806217    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-6gwdr"]
Sep 10 19:17:07.808799 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:17:07.808775    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80fb63b4_18c7_46f3_aac2_110f0ae1efe9.slice/crio-890b50974600ee288e4ee04b04876208097809059f0f6cccae46261b24657d31 WatchSource:0}: Error finding container 890b50974600ee288e4ee04b04876208097809059f0f6cccae46261b24657d31: Status 404 returned error can't find the container with id 890b50974600ee288e4ee04b04876208097809059f0f6cccae46261b24657d31
Sep 10 19:17:07.877883 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:07.877835    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-76b478bd78-66vvb" podStartSLOduration=2.386594206 podStartE2EDuration="4.877818011s" totalImagesPullingTime="2.491223805s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:17:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:17:07.876102129 +0000 UTC m=+391.105152154" watchObservedRunningTime="2026-09-10 19:17:07.877818011 +0000 UTC m=+391.106868035"
Sep 10 19:17:08.801269 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:08.801230    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-6gwdr" event={"ID":"80fb63b4-18c7-46f3-aac2-110f0ae1efe9","Type":"ContainerStarted","Data":"890b50974600ee288e4ee04b04876208097809059f0f6cccae46261b24657d31"}
Sep 10 19:17:09.439965 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:09.439933    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-hp4b2"]
Sep 10 19:17:09.444562 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:09.444534    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-hp4b2"
Sep 10 19:17:09.447933 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:09.447909    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 10 19:17:09.447933 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:09.447920    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 10 19:17:09.448084 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:09.447920    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-bktjk\""
Sep 10 19:17:09.454593 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:09.454566    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-hp4b2"]
Sep 10 19:17:09.513332 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:09.513296    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/290976c4-b4cb-4ca3-91b3-57b236ea7a1f-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-hp4b2\" (UID: \"290976c4-b4cb-4ca3-91b3-57b236ea7a1f\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-hp4b2"
Sep 10 19:17:09.513515 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:09.513394    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ck6l2\" (UniqueName: \"kubernetes.io/projected/290976c4-b4cb-4ca3-91b3-57b236ea7a1f-kube-api-access-ck6l2\") pod \"kuadrant-console-plugin-fbdd4f445-hp4b2\" (UID: \"290976c4-b4cb-4ca3-91b3-57b236ea7a1f\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-hp4b2"
Sep 10 19:17:09.513599 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:09.513577    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/290976c4-b4cb-4ca3-91b3-57b236ea7a1f-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-hp4b2\" (UID: \"290976c4-b4cb-4ca3-91b3-57b236ea7a1f\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-hp4b2"
Sep 10 19:17:09.614692 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:09.614652    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ck6l2\" (UniqueName: \"kubernetes.io/projected/290976c4-b4cb-4ca3-91b3-57b236ea7a1f-kube-api-access-ck6l2\") pod \"kuadrant-console-plugin-fbdd4f445-hp4b2\" (UID: \"290976c4-b4cb-4ca3-91b3-57b236ea7a1f\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-hp4b2"
Sep 10 19:17:09.614886 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:09.614724    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/290976c4-b4cb-4ca3-91b3-57b236ea7a1f-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-hp4b2\" (UID: \"290976c4-b4cb-4ca3-91b3-57b236ea7a1f\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-hp4b2"
Sep 10 19:17:09.614886 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:09.614770    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/290976c4-b4cb-4ca3-91b3-57b236ea7a1f-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-hp4b2\" (UID: \"290976c4-b4cb-4ca3-91b3-57b236ea7a1f\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-hp4b2"
Sep 10 19:17:09.615001 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:17:09.614914    2546 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Sep 10 19:17:09.615001 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:17:09.614986    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/290976c4-b4cb-4ca3-91b3-57b236ea7a1f-plugin-serving-cert podName:290976c4-b4cb-4ca3-91b3-57b236ea7a1f nodeName:}" failed. No retries permitted until 2026-09-10 19:17:10.114964702 +0000 UTC m=+393.344014711 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/290976c4-b4cb-4ca3-91b3-57b236ea7a1f-plugin-serving-cert") pod "kuadrant-console-plugin-fbdd4f445-hp4b2" (UID: "290976c4-b4cb-4ca3-91b3-57b236ea7a1f") : secret "plugin-serving-cert" not found
Sep 10 19:17:09.615509 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:09.615488    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/290976c4-b4cb-4ca3-91b3-57b236ea7a1f-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-hp4b2\" (UID: \"290976c4-b4cb-4ca3-91b3-57b236ea7a1f\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-hp4b2"
Sep 10 19:17:09.633730 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:09.633701    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ck6l2\" (UniqueName: \"kubernetes.io/projected/290976c4-b4cb-4ca3-91b3-57b236ea7a1f-kube-api-access-ck6l2\") pod \"kuadrant-console-plugin-fbdd4f445-hp4b2\" (UID: \"290976c4-b4cb-4ca3-91b3-57b236ea7a1f\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-hp4b2"
Sep 10 19:17:09.808196 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:09.808114    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-8hdw7" event={"ID":"59396e7b-7b35-40b3-a420-3699f237fcf2","Type":"ContainerStarted","Data":"f4fc0e9f8edbe79b1305014955993c11b71f85ca05569cac5c49d37eccd1f17a"}
Sep 10 19:17:09.808577 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:09.808237    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-8hdw7"
Sep 10 19:17:09.842329 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:09.842271    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-8hdw7" podStartSLOduration=2.969962003 podStartE2EDuration="5.842251061s" totalImagesPullingTime="2.872289058s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:17:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:17:09.841464186 +0000 UTC m=+393.070514211" watchObservedRunningTime="2026-09-10 19:17:09.842251061 +0000 UTC m=+393.071301087"
Sep 10 19:17:10.118848 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:10.118816    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/290976c4-b4cb-4ca3-91b3-57b236ea7a1f-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-hp4b2\" (UID: \"290976c4-b4cb-4ca3-91b3-57b236ea7a1f\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-hp4b2"
Sep 10 19:17:10.121316 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:10.121285    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/290976c4-b4cb-4ca3-91b3-57b236ea7a1f-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-hp4b2\" (UID: \"290976c4-b4cb-4ca3-91b3-57b236ea7a1f\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-hp4b2"
Sep 10 19:17:10.361121 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:10.361085    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-hp4b2"
Sep 10 19:17:10.646960 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:10.646880    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-hp4b2"]
Sep 10 19:17:10.649823 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:17:10.649794    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod290976c4_b4cb_4ca3_91b3_57b236ea7a1f.slice/crio-53f2fc150fc848334cf5351d92a6b6b477bfe5dd4906cf5674f43d82535a0079 WatchSource:0}: Error finding container 53f2fc150fc848334cf5351d92a6b6b477bfe5dd4906cf5674f43d82535a0079: Status 404 returned error can't find the container with id 53f2fc150fc848334cf5351d92a6b6b477bfe5dd4906cf5674f43d82535a0079
Sep 10 19:17:10.813333 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:10.813297    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-hp4b2" event={"ID":"290976c4-b4cb-4ca3-91b3-57b236ea7a1f","Type":"ContainerStarted","Data":"53f2fc150fc848334cf5351d92a6b6b477bfe5dd4906cf5674f43d82535a0079"}
Sep 10 19:17:10.814852 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:10.814818    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-6gwdr" event={"ID":"80fb63b4-18c7-46f3-aac2-110f0ae1efe9","Type":"ContainerStarted","Data":"bd4626a86c0f6c8ca460bb658691c9441de490db6434c844016b9cf733d7fbf0"}
Sep 10 19:17:10.815006 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:10.814988    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-6gwdr"
Sep 10 19:17:10.843651 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:10.843589    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-6gwdr" podStartSLOduration=1.779637675 podStartE2EDuration="3.843572988s" totalImagesPullingTime="2.063935313s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:17:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:17:10.842413292 +0000 UTC m=+394.071463317" watchObservedRunningTime="2026-09-10 19:17:10.843572988 +0000 UTC m=+394.072623012"
Sep 10 19:17:15.842191 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:15.842103    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-hp4b2" event={"ID":"290976c4-b4cb-4ca3-91b3-57b236ea7a1f","Type":"ContainerStarted","Data":"3ab6772ba37944fdf3e7913c45fd440d3952df5f5ec2a6e3f249a980268161f6"}
Sep 10 19:17:15.860752 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:15.860695    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-hp4b2" podStartSLOduration=1.925246525 podStartE2EDuration="6.860678278s" totalImagesPullingTime="4.935431753s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:17:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:17:15.859211751 +0000 UTC m=+399.088261778" watchObservedRunningTime="2026-09-10 19:17:15.860678278 +0000 UTC m=+399.089728302"
Sep 10 19:17:17.796572 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:17.796538    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-q6kgf"
Sep 10 19:17:18.804226 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:18.804194    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-66vvb"
Sep 10 19:17:20.272381 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:20.272333    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-wpm7h"]
Sep 10 19:17:20.275972 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:20.275956    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-wpm7h"
Sep 10 19:17:20.281078 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:20.281050    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-q6kgf"]
Sep 10 19:17:20.281263 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:20.281239    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-q6kgf" podUID="ad379d09-49e3-4540-aded-0f1c0c866e28" containerName="manager" containerID="cri-o://e490c503ae2cd546e06531025a136e7b4864d42344a071e4c76c637de0e5731e" gracePeriod=2
Sep 10 19:17:20.294256 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:20.294229    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-wpm7h"]
Sep 10 19:17:20.297828 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:20.297802    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-q6kgf"]
Sep 10 19:17:20.315797 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:20.315587    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-pxbpg"]
Sep 10 19:17:20.316074 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:20.316049    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad379d09-49e3-4540-aded-0f1c0c866e28" containerName="manager"
Sep 10 19:17:20.316074 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:20.316075    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad379d09-49e3-4540-aded-0f1c0c866e28" containerName="manager"
Sep 10 19:17:20.316236 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:20.316161    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad379d09-49e3-4540-aded-0f1c0c866e28" containerName="manager"
Sep 10 19:17:20.319554 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:20.319532    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-pxbpg"
Sep 10 19:17:20.320195 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:20.320176    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-8hdw7"]
Sep 10 19:17:20.320388 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:20.320326    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-8hdw7" podUID="59396e7b-7b35-40b3-a420-3699f237fcf2" containerName="manager" containerID="cri-o://f4fc0e9f8edbe79b1305014955993c11b71f85ca05569cac5c49d37eccd1f17a" gracePeriod=2
Sep 10 19:17:20.322100 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:20.322076    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-8hdw7"
Sep 10 19:17:20.325527 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:17:20.325482    2546 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-8hdw7\" is forbidden: User \"system:node:ip-10-0-132-222.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-222.ec2.internal' and this object" podUID="59396e7b-7b35-40b3-a420-3699f237fcf2" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-8hdw7"
Sep 10 19:17:20.327097 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:20.327074    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-8hdw7"]
Sep 10 19:17:20.328217 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:17:20.328185    2546 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-q6kgf\" is forbidden: User \"system:node:ip-10-0-132-222.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-222.ec2.internal' and this object" podUID="ad379d09-49e3-4540-aded-0f1c0c866e28" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-q6kgf"
Sep 10 19:17:20.340902 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:20.340876    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-pxbpg"]
Sep 10 19:17:20.408023 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:20.407990    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v468c\" (UniqueName: \"kubernetes.io/projected/9d59885d-35cf-4499-90da-e40cbeb9a0b9-kube-api-access-v468c\") pod \"kuadrant-operator-controller-manager-76f8f686bf-pxbpg\" (UID: \"9d59885d-35cf-4499-90da-e40cbeb9a0b9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-pxbpg"
Sep 10 19:17:20.408253 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:20.408036    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktckh\" (UniqueName: \"kubernetes.io/projected/69c8f195-819a-4f81-916f-21461bef0ad6-kube-api-access-ktckh\") pod \"kuadrant-operator-controller-manager-5769f99d9f-wpm7h\" (UID: \"69c8f195-819a-4f81-916f-21461bef0ad6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-wpm7h"
Sep 10 19:17:20.408253 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:20.408074    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9d59885d-35cf-4499-90da-e40cbeb9a0b9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-pxbpg\" (UID: \"9d59885d-35cf-4499-90da-e40cbeb9a0b9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-pxbpg"
Sep 10 19:17:20.408253 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:20.408137    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/69c8f195-819a-4f81-916f-21461bef0ad6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-wpm7h\" (UID: \"69c8f195-819a-4f81-916f-21461bef0ad6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-wpm7h"
Sep 10 19:17:20.508812 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:20.508783    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9d59885d-35cf-4499-90da-e40cbeb9a0b9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-pxbpg\" (UID: \"9d59885d-35cf-4499-90da-e40cbeb9a0b9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-pxbpg"
Sep 10 19:17:20.508942 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:20.508842    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/69c8f195-819a-4f81-916f-21461bef0ad6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-wpm7h\" (UID: \"69c8f195-819a-4f81-916f-21461bef0ad6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-wpm7h"
Sep 10 19:17:20.508942 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:20.508883    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v468c\" (UniqueName: \"kubernetes.io/projected/9d59885d-35cf-4499-90da-e40cbeb9a0b9-kube-api-access-v468c\") pod \"kuadrant-operator-controller-manager-76f8f686bf-pxbpg\" (UID: \"9d59885d-35cf-4499-90da-e40cbeb9a0b9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-pxbpg"
Sep 10 19:17:20.508942 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:20.508902    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktckh\" (UniqueName: \"kubernetes.io/projected/69c8f195-819a-4f81-916f-21461bef0ad6-kube-api-access-ktckh\") pod \"kuadrant-operator-controller-manager-5769f99d9f-wpm7h\" (UID: \"69c8f195-819a-4f81-916f-21461bef0ad6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-wpm7h"
Sep 10 19:17:20.509202 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:20.509180    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9d59885d-35cf-4499-90da-e40cbeb9a0b9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-pxbpg\" (UID: \"9d59885d-35cf-4499-90da-e40cbeb9a0b9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-pxbpg"
Sep 10 19:17:20.509281 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:20.509258    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/69c8f195-819a-4f81-916f-21461bef0ad6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-wpm7h\" (UID: \"69c8f195-819a-4f81-916f-21461bef0ad6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-wpm7h"
Sep 10 19:17:20.520632 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:20.520599    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktckh\" (UniqueName: \"kubernetes.io/projected/69c8f195-819a-4f81-916f-21461bef0ad6-kube-api-access-ktckh\") pod \"kuadrant-operator-controller-manager-5769f99d9f-wpm7h\" (UID: \"69c8f195-819a-4f81-916f-21461bef0ad6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-wpm7h"
Sep 10 19:17:20.520764 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:20.520603    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v468c\" (UniqueName: \"kubernetes.io/projected/9d59885d-35cf-4499-90da-e40cbeb9a0b9-kube-api-access-v468c\") pod \"kuadrant-operator-controller-manager-76f8f686bf-pxbpg\" (UID: \"9d59885d-35cf-4499-90da-e40cbeb9a0b9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-pxbpg"
Sep 10 19:17:20.550982 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:20.550959    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-q6kgf"
Sep 10 19:17:20.553440 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:17:20.553412    2546 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-8hdw7\" is forbidden: User \"system:node:ip-10-0-132-222.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-222.ec2.internal' and this object" podUID="59396e7b-7b35-40b3-a420-3699f237fcf2" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-8hdw7"
Sep 10 19:17:20.555608 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:17:20.555583    2546 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-q6kgf\" is forbidden: User \"system:node:ip-10-0-132-222.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-222.ec2.internal' and this object" podUID="ad379d09-49e3-4540-aded-0f1c0c866e28" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-q6kgf"
Sep 10 19:17:20.560198 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:20.560176    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-8hdw7"
Sep 10 19:17:20.562959 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:17:20.562935    2546 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-8hdw7\" is forbidden: User \"system:node:ip-10-0-132-222.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-222.ec2.internal' and this object" podUID="59396e7b-7b35-40b3-a420-3699f237fcf2" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-8hdw7"
Sep 10 19:17:20.565082 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:17:20.565048    2546 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-q6kgf\" is forbidden: User \"system:node:ip-10-0-132-222.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-222.ec2.internal' and this object" podUID="ad379d09-49e3-4540-aded-0f1c0c866e28" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-q6kgf"
Sep 10 19:17:20.587396 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:20.587372    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-wpm7h"
Sep 10 19:17:20.710936 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:20.710900    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ad379d09-49e3-4540-aded-0f1c0c866e28-extensions-socket-volume\") pod \"ad379d09-49e3-4540-aded-0f1c0c866e28\" (UID: \"ad379d09-49e3-4540-aded-0f1c0c866e28\") "
Sep 10 19:17:20.711099 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:20.710991    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wmhwx\" (UniqueName: \"kubernetes.io/projected/ad379d09-49e3-4540-aded-0f1c0c866e28-kube-api-access-wmhwx\") pod \"ad379d09-49e3-4540-aded-0f1c0c866e28\" (UID: \"ad379d09-49e3-4540-aded-0f1c0c866e28\") "
Sep 10 19:17:20.711099 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:20.711022    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c5cnl\" (UniqueName: \"kubernetes.io/projected/59396e7b-7b35-40b3-a420-3699f237fcf2-kube-api-access-c5cnl\") pod \"59396e7b-7b35-40b3-a420-3699f237fcf2\" (UID: \"59396e7b-7b35-40b3-a420-3699f237fcf2\") "
Sep 10 19:17:20.711451 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:20.711421    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad379d09-49e3-4540-aded-0f1c0c866e28-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "ad379d09-49e3-4540-aded-0f1c0c866e28" (UID: "ad379d09-49e3-4540-aded-0f1c0c866e28"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:17:20.713196 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:20.713171    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad379d09-49e3-4540-aded-0f1c0c866e28-kube-api-access-wmhwx" (OuterVolumeSpecName: "kube-api-access-wmhwx") pod "ad379d09-49e3-4540-aded-0f1c0c866e28" (UID: "ad379d09-49e3-4540-aded-0f1c0c866e28"). InnerVolumeSpecName "kube-api-access-wmhwx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 19:17:20.713266 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:20.713178    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59396e7b-7b35-40b3-a420-3699f237fcf2-kube-api-access-c5cnl" (OuterVolumeSpecName: "kube-api-access-c5cnl") pod "59396e7b-7b35-40b3-a420-3699f237fcf2" (UID: "59396e7b-7b35-40b3-a420-3699f237fcf2"). InnerVolumeSpecName "kube-api-access-c5cnl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 19:17:20.734360 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:20.734304    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-pxbpg"
Sep 10 19:17:20.811924 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:20.811851    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5cnl\" (UniqueName: \"kubernetes.io/projected/59396e7b-7b35-40b3-a420-3699f237fcf2-kube-api-access-c5cnl\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:17:20.811924 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:20.811880    2546 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ad379d09-49e3-4540-aded-0f1c0c866e28-extensions-socket-volume\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:17:20.811924 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:20.811894    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmhwx\" (UniqueName: \"kubernetes.io/projected/ad379d09-49e3-4540-aded-0f1c0c866e28-kube-api-access-wmhwx\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:17:20.864750 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:20.864723    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-wpm7h"]
Sep 10 19:17:20.867110 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:20.867081    2546 generic.go:356] "Generic (PLEG): container finished" podID="59396e7b-7b35-40b3-a420-3699f237fcf2" containerID="f4fc0e9f8edbe79b1305014955993c11b71f85ca05569cac5c49d37eccd1f17a" exitCode=0
Sep 10 19:17:20.867225 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:20.867134    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-8hdw7"
Sep 10 19:17:20.867286 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:20.867134    2546 scope.go:117] "RemoveContainer" containerID="f4fc0e9f8edbe79b1305014955993c11b71f85ca05569cac5c49d37eccd1f17a"
Sep 10 19:17:20.867860 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:17:20.867831    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69c8f195_819a_4f81_916f_21461bef0ad6.slice/crio-1d883d1864d892c7c6450e04679a0a4721b7924e8e53aa277d7118839fe1740a WatchSource:0}: Error finding container 1d883d1864d892c7c6450e04679a0a4721b7924e8e53aa277d7118839fe1740a: Status 404 returned error can't find the container with id 1d883d1864d892c7c6450e04679a0a4721b7924e8e53aa277d7118839fe1740a
Sep 10 19:17:20.868829 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:20.868803    2546 generic.go:356] "Generic (PLEG): container finished" podID="ad379d09-49e3-4540-aded-0f1c0c866e28" containerID="e490c503ae2cd546e06531025a136e7b4864d42344a071e4c76c637de0e5731e" exitCode=0
Sep 10 19:17:20.868959 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:20.868841    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-q6kgf"
Sep 10 19:17:20.869615 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:17:20.869584    2546 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-8hdw7\" is forbidden: User \"system:node:ip-10-0-132-222.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-222.ec2.internal' and this object" podUID="59396e7b-7b35-40b3-a420-3699f237fcf2" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-8hdw7"
Sep 10 19:17:20.871554 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:17:20.871528    2546 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-q6kgf\" is forbidden: User \"system:node:ip-10-0-132-222.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-222.ec2.internal' and this object" podUID="ad379d09-49e3-4540-aded-0f1c0c866e28" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-q6kgf"
Sep 10 19:17:20.873370 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:17:20.873328    2546 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-8hdw7\" is forbidden: User \"system:node:ip-10-0-132-222.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-222.ec2.internal' and this object" podUID="59396e7b-7b35-40b3-a420-3699f237fcf2" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-8hdw7"
Sep 10 19:17:20.875159 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:17:20.875135    2546 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-q6kgf\" is forbidden: User \"system:node:ip-10-0-132-222.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-222.ec2.internal' and this object" podUID="ad379d09-49e3-4540-aded-0f1c0c866e28" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-q6kgf"
Sep 10 19:17:20.876300 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:20.876278    2546 scope.go:117] "RemoveContainer" containerID="f4fc0e9f8edbe79b1305014955993c11b71f85ca05569cac5c49d37eccd1f17a"
Sep 10 19:17:20.876808 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:17:20.876663    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4fc0e9f8edbe79b1305014955993c11b71f85ca05569cac5c49d37eccd1f17a\": container with ID starting with f4fc0e9f8edbe79b1305014955993c11b71f85ca05569cac5c49d37eccd1f17a not found: ID does not exist" containerID="f4fc0e9f8edbe79b1305014955993c11b71f85ca05569cac5c49d37eccd1f17a"
Sep 10 19:17:20.876808 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:20.876698    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4fc0e9f8edbe79b1305014955993c11b71f85ca05569cac5c49d37eccd1f17a"} err="failed to get container status \"f4fc0e9f8edbe79b1305014955993c11b71f85ca05569cac5c49d37eccd1f17a\": rpc error: code = NotFound desc = could not find container \"f4fc0e9f8edbe79b1305014955993c11b71f85ca05569cac5c49d37eccd1f17a\": container with ID starting with f4fc0e9f8edbe79b1305014955993c11b71f85ca05569cac5c49d37eccd1f17a not found: ID does not exist"
Sep 10 19:17:20.876808 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:20.876722    2546 scope.go:117] "RemoveContainer" containerID="e490c503ae2cd546e06531025a136e7b4864d42344a071e4c76c637de0e5731e"
Sep 10 19:17:20.879136 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:17:20.879109    2546 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-8hdw7\" is forbidden: User \"system:node:ip-10-0-132-222.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-222.ec2.internal' and this object" podUID="59396e7b-7b35-40b3-a420-3699f237fcf2" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-8hdw7"
Sep 10 19:17:20.881074 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:17:20.881052    2546 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-q6kgf\" is forbidden: User \"system:node:ip-10-0-132-222.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-222.ec2.internal' and this object" podUID="ad379d09-49e3-4540-aded-0f1c0c866e28" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-q6kgf"
Sep 10 19:17:20.883159 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:17:20.883124    2546 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-q6kgf\" is forbidden: User \"system:node:ip-10-0-132-222.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-222.ec2.internal' and this object" podUID="ad379d09-49e3-4540-aded-0f1c0c866e28" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-q6kgf"
Sep 10 19:17:20.884877 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:20.884862    2546 scope.go:117] "RemoveContainer" containerID="e490c503ae2cd546e06531025a136e7b4864d42344a071e4c76c637de0e5731e"
Sep 10 19:17:20.885034 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:17:20.885013    2546 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-8hdw7\" is forbidden: User \"system:node:ip-10-0-132-222.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-222.ec2.internal' and this object" podUID="59396e7b-7b35-40b3-a420-3699f237fcf2" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-8hdw7"
Sep 10 19:17:20.885139 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:17:20.885125    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e490c503ae2cd546e06531025a136e7b4864d42344a071e4c76c637de0e5731e\": container with ID starting with e490c503ae2cd546e06531025a136e7b4864d42344a071e4c76c637de0e5731e not found: ID does not exist" containerID="e490c503ae2cd546e06531025a136e7b4864d42344a071e4c76c637de0e5731e"
Sep 10 19:17:20.885179 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:20.885149    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e490c503ae2cd546e06531025a136e7b4864d42344a071e4c76c637de0e5731e"} err="failed to get container status \"e490c503ae2cd546e06531025a136e7b4864d42344a071e4c76c637de0e5731e\": rpc error: code = NotFound desc = could not find container \"e490c503ae2cd546e06531025a136e7b4864d42344a071e4c76c637de0e5731e\": container with ID starting with e490c503ae2cd546e06531025a136e7b4864d42344a071e4c76c637de0e5731e not found: ID does not exist"
Sep 10 19:17:21.011285 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:21.011249    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-pxbpg"]
Sep 10 19:17:21.014624 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:17:21.014599    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d59885d_35cf_4499_90da_e40cbeb9a0b9.slice/crio-9eba45ac0da30415cf9e8ac38a629eb6ec7ba1e23d6360f5d31d3709513516c7 WatchSource:0}: Error finding container 9eba45ac0da30415cf9e8ac38a629eb6ec7ba1e23d6360f5d31d3709513516c7: Status 404 returned error can't find the container with id 9eba45ac0da30415cf9e8ac38a629eb6ec7ba1e23d6360f5d31d3709513516c7
Sep 10 19:17:21.280785 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:21.280750    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59396e7b-7b35-40b3-a420-3699f237fcf2" path="/var/lib/kubelet/pods/59396e7b-7b35-40b3-a420-3699f237fcf2/volumes"
Sep 10 19:17:21.281153 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:21.281138    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad379d09-49e3-4540-aded-0f1c0c866e28" path="/var/lib/kubelet/pods/ad379d09-49e3-4540-aded-0f1c0c866e28/volumes"
Sep 10 19:17:21.823406 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:21.823367    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-6gwdr"
Sep 10 19:17:21.876733 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:21.876696    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-wpm7h" event={"ID":"69c8f195-819a-4f81-916f-21461bef0ad6","Type":"ContainerStarted","Data":"e123fae4c4770984c0cb9c4b6ee1fb74ce929da86d26df883d2d98b00650bef4"}
Sep 10 19:17:21.876922 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:21.876757    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-wpm7h" event={"ID":"69c8f195-819a-4f81-916f-21461bef0ad6","Type":"ContainerStarted","Data":"1d883d1864d892c7c6450e04679a0a4721b7924e8e53aa277d7118839fe1740a"}
Sep 10 19:17:21.876922 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:21.876874    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-wpm7h"
Sep 10 19:17:21.879322 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:21.879299    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-pxbpg" event={"ID":"9d59885d-35cf-4499-90da-e40cbeb9a0b9","Type":"ContainerStarted","Data":"0bae66b6974cfe9b0f2301b1b6952400ffdf80b91b2079046fb0d2141bf4cff3"}
Sep 10 19:17:21.879447 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:21.879366    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-pxbpg"
Sep 10 19:17:21.879447 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:21.879385    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-pxbpg" event={"ID":"9d59885d-35cf-4499-90da-e40cbeb9a0b9","Type":"ContainerStarted","Data":"9eba45ac0da30415cf9e8ac38a629eb6ec7ba1e23d6360f5d31d3709513516c7"}
Sep 10 19:17:21.905415 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:21.904152    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-wpm7h" podStartSLOduration=1.904131065 podStartE2EDuration="1.904131065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:17:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:17:21.903449146 +0000 UTC m=+405.132499171" watchObservedRunningTime="2026-09-10 19:17:21.904131065 +0000 UTC m=+405.133181190"
Sep 10 19:17:21.928732 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:21.928686    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-pxbpg" podStartSLOduration=1.928669865 podStartE2EDuration="1.928669865s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:17:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:17:21.927137916 +0000 UTC m=+405.156187941" watchObservedRunningTime="2026-09-10 19:17:21.928669865 +0000 UTC m=+405.157719889"
Sep 10 19:17:32.885756 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:32.885722    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-pxbpg"
Sep 10 19:17:32.886233 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:32.885793    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-wpm7h"
Sep 10 19:17:32.987956 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:32.987917    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-pxbpg"]
Sep 10 19:17:32.988134 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:32.988108    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-pxbpg" podUID="9d59885d-35cf-4499-90da-e40cbeb9a0b9" containerName="manager" containerID="cri-o://0bae66b6974cfe9b0f2301b1b6952400ffdf80b91b2079046fb0d2141bf4cff3" gracePeriod=10
Sep 10 19:17:33.767945 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:33.767916    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-pxbpg"
Sep 10 19:17:33.925998 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:33.925904    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v468c\" (UniqueName: \"kubernetes.io/projected/9d59885d-35cf-4499-90da-e40cbeb9a0b9-kube-api-access-v468c\") pod \"9d59885d-35cf-4499-90da-e40cbeb9a0b9\" (UID: \"9d59885d-35cf-4499-90da-e40cbeb9a0b9\") "
Sep 10 19:17:33.925998 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:33.925973    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9d59885d-35cf-4499-90da-e40cbeb9a0b9-extensions-socket-volume\") pod \"9d59885d-35cf-4499-90da-e40cbeb9a0b9\" (UID: \"9d59885d-35cf-4499-90da-e40cbeb9a0b9\") "
Sep 10 19:17:33.926450 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:33.926334    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d59885d-35cf-4499-90da-e40cbeb9a0b9-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "9d59885d-35cf-4499-90da-e40cbeb9a0b9" (UID: "9d59885d-35cf-4499-90da-e40cbeb9a0b9"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:17:33.928202 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:33.928176    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d59885d-35cf-4499-90da-e40cbeb9a0b9-kube-api-access-v468c" (OuterVolumeSpecName: "kube-api-access-v468c") pod "9d59885d-35cf-4499-90da-e40cbeb9a0b9" (UID: "9d59885d-35cf-4499-90da-e40cbeb9a0b9"). InnerVolumeSpecName "kube-api-access-v468c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 19:17:33.931616 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:33.931592    2546 generic.go:356] "Generic (PLEG): container finished" podID="9d59885d-35cf-4499-90da-e40cbeb9a0b9" containerID="0bae66b6974cfe9b0f2301b1b6952400ffdf80b91b2079046fb0d2141bf4cff3" exitCode=0
Sep 10 19:17:33.931740 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:33.931648    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-pxbpg"
Sep 10 19:17:33.931740 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:33.931675    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-pxbpg" event={"ID":"9d59885d-35cf-4499-90da-e40cbeb9a0b9","Type":"ContainerDied","Data":"0bae66b6974cfe9b0f2301b1b6952400ffdf80b91b2079046fb0d2141bf4cff3"}
Sep 10 19:17:33.931852 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:33.931742    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-pxbpg" event={"ID":"9d59885d-35cf-4499-90da-e40cbeb9a0b9","Type":"ContainerDied","Data":"9eba45ac0da30415cf9e8ac38a629eb6ec7ba1e23d6360f5d31d3709513516c7"}
Sep 10 19:17:33.931852 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:33.931775    2546 scope.go:117] "RemoveContainer" containerID="0bae66b6974cfe9b0f2301b1b6952400ffdf80b91b2079046fb0d2141bf4cff3"
Sep 10 19:17:33.940427 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:33.940402    2546 scope.go:117] "RemoveContainer" containerID="0bae66b6974cfe9b0f2301b1b6952400ffdf80b91b2079046fb0d2141bf4cff3"
Sep 10 19:17:33.940781 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:17:33.940758    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bae66b6974cfe9b0f2301b1b6952400ffdf80b91b2079046fb0d2141bf4cff3\": container with ID starting with 0bae66b6974cfe9b0f2301b1b6952400ffdf80b91b2079046fb0d2141bf4cff3 not found: ID does not exist" containerID="0bae66b6974cfe9b0f2301b1b6952400ffdf80b91b2079046fb0d2141bf4cff3"
Sep 10 19:17:33.940865 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:33.940789    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bae66b6974cfe9b0f2301b1b6952400ffdf80b91b2079046fb0d2141bf4cff3"} err="failed to get container status \"0bae66b6974cfe9b0f2301b1b6952400ffdf80b91b2079046fb0d2141bf4cff3\": rpc error: code = NotFound desc = could not find container \"0bae66b6974cfe9b0f2301b1b6952400ffdf80b91b2079046fb0d2141bf4cff3\": container with ID starting with 0bae66b6974cfe9b0f2301b1b6952400ffdf80b91b2079046fb0d2141bf4cff3 not found: ID does not exist"
Sep 10 19:17:33.954350 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:33.954319    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-pxbpg"]
Sep 10 19:17:33.960386 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:33.960363    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-pxbpg"]
Sep 10 19:17:34.026475 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:34.026443    2546 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9d59885d-35cf-4499-90da-e40cbeb9a0b9-extensions-socket-volume\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:17:34.026475 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:34.026471    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v468c\" (UniqueName: \"kubernetes.io/projected/9d59885d-35cf-4499-90da-e40cbeb9a0b9-kube-api-access-v468c\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:17:35.280683 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:35.280639    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d59885d-35cf-4499-90da-e40cbeb9a0b9" path="/var/lib/kubelet/pods/9d59885d-35cf-4499-90da-e40cbeb9a0b9/volumes"
Sep 10 19:17:49.467160 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:49.467127    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-ps4m9"]
Sep 10 19:17:49.467658 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:49.467456    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59396e7b-7b35-40b3-a420-3699f237fcf2" containerName="manager"
Sep 10 19:17:49.467658 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:49.467468    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="59396e7b-7b35-40b3-a420-3699f237fcf2" containerName="manager"
Sep 10 19:17:49.467658 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:49.467480    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d59885d-35cf-4499-90da-e40cbeb9a0b9" containerName="manager"
Sep 10 19:17:49.467658 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:49.467486    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d59885d-35cf-4499-90da-e40cbeb9a0b9" containerName="manager"
Sep 10 19:17:49.467658 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:49.467534    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="59396e7b-7b35-40b3-a420-3699f237fcf2" containerName="manager"
Sep 10 19:17:49.467658 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:49.467541    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d59885d-35cf-4499-90da-e40cbeb9a0b9" containerName="manager"
Sep 10 19:17:49.470271 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:49.470252    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-ps4m9"
Sep 10 19:17:49.475262 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:49.475238    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-ddr6k\""
Sep 10 19:17:49.475396 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:49.475276    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 10 19:17:49.490559 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:49.490528    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-ps4m9"]
Sep 10 19:17:49.546616 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:49.546578    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/698ce0a1-dce5-40d0-8bd4-c2042400e6f2-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-ps4m9\" (UID: \"698ce0a1-dce5-40d0-8bd4-c2042400e6f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-ps4m9"
Sep 10 19:17:49.546616 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:49.546617    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/698ce0a1-dce5-40d0-8bd4-c2042400e6f2-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-ps4m9\" (UID: \"698ce0a1-dce5-40d0-8bd4-c2042400e6f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-ps4m9"
Sep 10 19:17:49.546844 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:49.546642    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/698ce0a1-dce5-40d0-8bd4-c2042400e6f2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-ps4m9\" (UID: \"698ce0a1-dce5-40d0-8bd4-c2042400e6f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-ps4m9"
Sep 10 19:17:49.546844 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:49.546731    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/698ce0a1-dce5-40d0-8bd4-c2042400e6f2-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-ps4m9\" (UID: \"698ce0a1-dce5-40d0-8bd4-c2042400e6f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-ps4m9"
Sep 10 19:17:49.546844 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:49.546764    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/698ce0a1-dce5-40d0-8bd4-c2042400e6f2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-ps4m9\" (UID: \"698ce0a1-dce5-40d0-8bd4-c2042400e6f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-ps4m9"
Sep 10 19:17:49.546844 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:49.546800    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/698ce0a1-dce5-40d0-8bd4-c2042400e6f2-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-ps4m9\" (UID: \"698ce0a1-dce5-40d0-8bd4-c2042400e6f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-ps4m9"
Sep 10 19:17:49.546844 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:49.546815    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/698ce0a1-dce5-40d0-8bd4-c2042400e6f2-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-ps4m9\" (UID: \"698ce0a1-dce5-40d0-8bd4-c2042400e6f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-ps4m9"
Sep 10 19:17:49.546844 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:49.546829    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/698ce0a1-dce5-40d0-8bd4-c2042400e6f2-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-ps4m9\" (UID: \"698ce0a1-dce5-40d0-8bd4-c2042400e6f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-ps4m9"
Sep 10 19:17:49.547094 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:49.546851    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rj74w\" (UniqueName: \"kubernetes.io/projected/698ce0a1-dce5-40d0-8bd4-c2042400e6f2-kube-api-access-rj74w\") pod \"maas-default-gateway-openshift-default-5ccb968c66-ps4m9\" (UID: \"698ce0a1-dce5-40d0-8bd4-c2042400e6f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-ps4m9"
Sep 10 19:17:49.647377 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:49.647319    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/698ce0a1-dce5-40d0-8bd4-c2042400e6f2-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-ps4m9\" (UID: \"698ce0a1-dce5-40d0-8bd4-c2042400e6f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-ps4m9"
Sep 10 19:17:49.647377 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:49.647378    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/698ce0a1-dce5-40d0-8bd4-c2042400e6f2-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-ps4m9\" (UID: \"698ce0a1-dce5-40d0-8bd4-c2042400e6f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-ps4m9"
Sep 10 19:17:49.647589 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:49.647400    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/698ce0a1-dce5-40d0-8bd4-c2042400e6f2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-ps4m9\" (UID: \"698ce0a1-dce5-40d0-8bd4-c2042400e6f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-ps4m9"
Sep 10 19:17:49.647589 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:49.647531    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/698ce0a1-dce5-40d0-8bd4-c2042400e6f2-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-ps4m9\" (UID: \"698ce0a1-dce5-40d0-8bd4-c2042400e6f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-ps4m9"
Sep 10 19:17:49.647589 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:49.647579    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/698ce0a1-dce5-40d0-8bd4-c2042400e6f2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-ps4m9\" (UID: \"698ce0a1-dce5-40d0-8bd4-c2042400e6f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-ps4m9"
Sep 10 19:17:49.647714 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:49.647644    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/698ce0a1-dce5-40d0-8bd4-c2042400e6f2-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-ps4m9\" (UID: \"698ce0a1-dce5-40d0-8bd4-c2042400e6f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-ps4m9"
Sep 10 19:17:49.647714 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:49.647670    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/698ce0a1-dce5-40d0-8bd4-c2042400e6f2-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-ps4m9\" (UID: \"698ce0a1-dce5-40d0-8bd4-c2042400e6f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-ps4m9"
Sep 10 19:17:49.647714 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:49.647697    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/698ce0a1-dce5-40d0-8bd4-c2042400e6f2-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-ps4m9\" (UID: \"698ce0a1-dce5-40d0-8bd4-c2042400e6f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-ps4m9"
Sep 10 19:17:49.647861 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:49.647734    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj74w\" (UniqueName: \"kubernetes.io/projected/698ce0a1-dce5-40d0-8bd4-c2042400e6f2-kube-api-access-rj74w\") pod \"maas-default-gateway-openshift-default-5ccb968c66-ps4m9\" (UID: \"698ce0a1-dce5-40d0-8bd4-c2042400e6f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-ps4m9"
Sep 10 19:17:49.647861 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:49.647774    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/698ce0a1-dce5-40d0-8bd4-c2042400e6f2-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-ps4m9\" (UID: \"698ce0a1-dce5-40d0-8bd4-c2042400e6f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-ps4m9"
Sep 10 19:17:49.648084 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:49.648046    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/698ce0a1-dce5-40d0-8bd4-c2042400e6f2-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-ps4m9\" (UID: \"698ce0a1-dce5-40d0-8bd4-c2042400e6f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-ps4m9"
Sep 10 19:17:49.648084 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:49.648068    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/698ce0a1-dce5-40d0-8bd4-c2042400e6f2-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-ps4m9\" (UID: \"698ce0a1-dce5-40d0-8bd4-c2042400e6f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-ps4m9"
Sep 10 19:17:49.648245 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:49.648153    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/698ce0a1-dce5-40d0-8bd4-c2042400e6f2-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-ps4m9\" (UID: \"698ce0a1-dce5-40d0-8bd4-c2042400e6f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-ps4m9"
Sep 10 19:17:49.648245 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:49.648187    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/698ce0a1-dce5-40d0-8bd4-c2042400e6f2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-ps4m9\" (UID: \"698ce0a1-dce5-40d0-8bd4-c2042400e6f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-ps4m9"
Sep 10 19:17:49.650061 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:49.650035    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/698ce0a1-dce5-40d0-8bd4-c2042400e6f2-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-ps4m9\" (UID: \"698ce0a1-dce5-40d0-8bd4-c2042400e6f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-ps4m9"
Sep 10 19:17:49.650229 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:49.650212    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/698ce0a1-dce5-40d0-8bd4-c2042400e6f2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-ps4m9\" (UID: \"698ce0a1-dce5-40d0-8bd4-c2042400e6f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-ps4m9"
Sep 10 19:17:49.657616 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:49.657594    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rj74w\" (UniqueName: \"kubernetes.io/projected/698ce0a1-dce5-40d0-8bd4-c2042400e6f2-kube-api-access-rj74w\") pod \"maas-default-gateway-openshift-default-5ccb968c66-ps4m9\" (UID: \"698ce0a1-dce5-40d0-8bd4-c2042400e6f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-ps4m9"
Sep 10 19:17:49.657718 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:49.657700    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/698ce0a1-dce5-40d0-8bd4-c2042400e6f2-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-ps4m9\" (UID: \"698ce0a1-dce5-40d0-8bd4-c2042400e6f2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-ps4m9"
Sep 10 19:17:49.780857 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:49.780770    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-ps4m9"
Sep 10 19:17:50.059458 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:50.059365    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-ps4m9"]
Sep 10 19:17:50.062131 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:17:50.062102    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod698ce0a1_dce5_40d0_8bd4_c2042400e6f2.slice/crio-aeb9c21ddfae9326d3bbc287541262d5b5360e6b1e0dfe9df71dec280246e3b5 WatchSource:0}: Error finding container aeb9c21ddfae9326d3bbc287541262d5b5360e6b1e0dfe9df71dec280246e3b5: Status 404 returned error can't find the container with id aeb9c21ddfae9326d3bbc287541262d5b5360e6b1e0dfe9df71dec280246e3b5
Sep 10 19:17:51.008864 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:51.008815    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-ps4m9" event={"ID":"698ce0a1-dce5-40d0-8bd4-c2042400e6f2","Type":"ContainerStarted","Data":"aeb9c21ddfae9326d3bbc287541262d5b5360e6b1e0dfe9df71dec280246e3b5"}
Sep 10 19:17:52.693004 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:52.692959    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 19:17:52.693250 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:52.693051    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 19:17:52.693250 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:52.693082    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 19:17:53.019211 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:53.019118    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-ps4m9" event={"ID":"698ce0a1-dce5-40d0-8bd4-c2042400e6f2","Type":"ContainerStarted","Data":"24064752b66e0f1417fe30229f5d02ab40fd9b2d68feb62e9e26f8c8925c7663"}
Sep 10 19:17:53.781948 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:53.781900    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-ps4m9"
Sep 10 19:17:53.787140 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:53.787110    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-ps4m9"
Sep 10 19:17:53.819029 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:53.818975    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-ps4m9" podStartSLOduration=2.190277687 podStartE2EDuration="4.818957364s" totalImagesPullingTime="2.628679677s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:17:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:17:53.056536676 +0000 UTC m=+436.285586702" watchObservedRunningTime="2026-09-10 19:17:53.818957364 +0000 UTC m=+437.048007389"
Sep 10 19:17:54.024141 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:54.024108    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-ps4m9"
Sep 10 19:17:54.025034 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:54.025010    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-ps4m9"
Sep 10 19:17:54.070329 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:54.070228    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-gfl8n"]
Sep 10 19:17:54.073097 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:54.073063    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-gfl8n"
Sep 10 19:17:54.076476 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:54.076440    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 10 19:17:54.083011 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:54.082981    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-gfl8n"]
Sep 10 19:17:54.158642 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:54.158606    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-gfl8n"]
Sep 10 19:17:54.187738 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:54.187693    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c9395621-008d-4316-8b74-f4bff4ba9151-config-file\") pod \"limitador-limitador-8588749977-gfl8n\" (UID: \"c9395621-008d-4316-8b74-f4bff4ba9151\") " pod="kuadrant-system/limitador-limitador-8588749977-gfl8n"
Sep 10 19:17:54.187903 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:54.187755    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7nm5\" (UniqueName: \"kubernetes.io/projected/c9395621-008d-4316-8b74-f4bff4ba9151-kube-api-access-w7nm5\") pod \"limitador-limitador-8588749977-gfl8n\" (UID: \"c9395621-008d-4316-8b74-f4bff4ba9151\") " pod="kuadrant-system/limitador-limitador-8588749977-gfl8n"
Sep 10 19:17:54.288712 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:54.288674    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c9395621-008d-4316-8b74-f4bff4ba9151-config-file\") pod \"limitador-limitador-8588749977-gfl8n\" (UID: \"c9395621-008d-4316-8b74-f4bff4ba9151\") " pod="kuadrant-system/limitador-limitador-8588749977-gfl8n"
Sep 10 19:17:54.288898 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:54.288721    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7nm5\" (UniqueName: \"kubernetes.io/projected/c9395621-008d-4316-8b74-f4bff4ba9151-kube-api-access-w7nm5\") pod \"limitador-limitador-8588749977-gfl8n\" (UID: \"c9395621-008d-4316-8b74-f4bff4ba9151\") " pod="kuadrant-system/limitador-limitador-8588749977-gfl8n"
Sep 10 19:17:54.289353 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:54.289324    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c9395621-008d-4316-8b74-f4bff4ba9151-config-file\") pod \"limitador-limitador-8588749977-gfl8n\" (UID: \"c9395621-008d-4316-8b74-f4bff4ba9151\") " pod="kuadrant-system/limitador-limitador-8588749977-gfl8n"
Sep 10 19:17:54.299323 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:54.299284    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7nm5\" (UniqueName: \"kubernetes.io/projected/c9395621-008d-4316-8b74-f4bff4ba9151-kube-api-access-w7nm5\") pod \"limitador-limitador-8588749977-gfl8n\" (UID: \"c9395621-008d-4316-8b74-f4bff4ba9151\") " pod="kuadrant-system/limitador-limitador-8588749977-gfl8n"
Sep 10 19:17:54.385361 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:54.385302    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-gfl8n"
Sep 10 19:17:54.396664 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:54.396639    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-88n8f"]
Sep 10 19:17:54.398848 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:54.398828    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-88n8f"
Sep 10 19:17:54.411715 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:54.411686    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-88n8f"]
Sep 10 19:17:54.447855 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:54.447823    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-88n8f"]
Sep 10 19:17:54.490745 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:54.490708    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwp2s\" (UniqueName: \"kubernetes.io/projected/9127a66c-9536-49d4-ae69-0910b3374c49-kube-api-access-wwp2s\") pod \"limitador-limitador-84994f7ffc-88n8f\" (UID: \"9127a66c-9536-49d4-ae69-0910b3374c49\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-88n8f"
Sep 10 19:17:54.490970 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:54.490768    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9127a66c-9536-49d4-ae69-0910b3374c49-config-file\") pod \"limitador-limitador-84994f7ffc-88n8f\" (UID: \"9127a66c-9536-49d4-ae69-0910b3374c49\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-88n8f"
Sep 10 19:17:54.591588 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:54.591547    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwp2s\" (UniqueName: \"kubernetes.io/projected/9127a66c-9536-49d4-ae69-0910b3374c49-kube-api-access-wwp2s\") pod \"limitador-limitador-84994f7ffc-88n8f\" (UID: \"9127a66c-9536-49d4-ae69-0910b3374c49\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-88n8f"
Sep 10 19:17:54.591777 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:54.591622    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9127a66c-9536-49d4-ae69-0910b3374c49-config-file\") pod \"limitador-limitador-84994f7ffc-88n8f\" (UID: \"9127a66c-9536-49d4-ae69-0910b3374c49\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-88n8f"
Sep 10 19:17:54.592298 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:54.592264    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9127a66c-9536-49d4-ae69-0910b3374c49-config-file\") pod \"limitador-limitador-84994f7ffc-88n8f\" (UID: \"9127a66c-9536-49d4-ae69-0910b3374c49\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-88n8f"
Sep 10 19:17:54.600550 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:54.600517    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwp2s\" (UniqueName: \"kubernetes.io/projected/9127a66c-9536-49d4-ae69-0910b3374c49-kube-api-access-wwp2s\") pod \"limitador-limitador-84994f7ffc-88n8f\" (UID: \"9127a66c-9536-49d4-ae69-0910b3374c49\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-88n8f"
Sep 10 19:17:54.666655 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:54.666482    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-gfl8n"]
Sep 10 19:17:54.668771 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:17:54.668740    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9395621_008d_4316_8b74_f4bff4ba9151.slice/crio-4a31c03ffa4b24a19967d18878ab3544309ea58edf4123b95f5e0fd4663f069c WatchSource:0}: Error finding container 4a31c03ffa4b24a19967d18878ab3544309ea58edf4123b95f5e0fd4663f069c: Status 404 returned error can't find the container with id 4a31c03ffa4b24a19967d18878ab3544309ea58edf4123b95f5e0fd4663f069c
Sep 10 19:17:54.721971 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:54.721929    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-88n8f"
Sep 10 19:17:54.997226 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:54.997113    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-88n8f"]
Sep 10 19:17:54.999796 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:17:54.999773    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9127a66c_9536_49d4_ae69_0910b3374c49.slice/crio-e2a590b5da80029a760686423ad3bc6754baac531742a9a4408fe03b5c4f08ce WatchSource:0}: Error finding container e2a590b5da80029a760686423ad3bc6754baac531742a9a4408fe03b5c4f08ce: Status 404 returned error can't find the container with id e2a590b5da80029a760686423ad3bc6754baac531742a9a4408fe03b5c4f08ce
Sep 10 19:17:55.030277 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:55.030239    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-gfl8n" event={"ID":"c9395621-008d-4316-8b74-f4bff4ba9151","Type":"ContainerStarted","Data":"4a31c03ffa4b24a19967d18878ab3544309ea58edf4123b95f5e0fd4663f069c"}
Sep 10 19:17:55.031492 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:55.031460    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-88n8f" event={"ID":"9127a66c-9536-49d4-ae69-0910b3374c49","Type":"ContainerStarted","Data":"e2a590b5da80029a760686423ad3bc6754baac531742a9a4408fe03b5c4f08ce"}
Sep 10 19:17:56.037008 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:56.036914    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-gfl8n" event={"ID":"c9395621-008d-4316-8b74-f4bff4ba9151","Type":"ContainerStarted","Data":"d16901ddb02688a10e4093d17bb32fe9b8571a29c581aaecd92621c041440642"}
Sep 10 19:17:56.037468 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:56.037017    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-gfl8n"
Sep 10 19:17:56.038615 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:56.038582    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-88n8f" event={"ID":"9127a66c-9536-49d4-ae69-0910b3374c49","Type":"ContainerStarted","Data":"f908b8b9df0cffc67d4ace63280891eacf0a8004eae3973f2e182eae39fc134c"}
Sep 10 19:17:56.038764 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:56.038752    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-88n8f"
Sep 10 19:17:56.056596 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:56.056539    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-gfl8n" podStartSLOduration=1.024298239 podStartE2EDuration="2.056522859s" totalImagesPullingTime="1.03222462s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:17:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:17:56.054221499 +0000 UTC m=+439.283271535" watchObservedRunningTime="2026-09-10 19:17:56.056522859 +0000 UTC m=+439.285572883"
Sep 10 19:17:56.071130 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:17:56.071081    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-88n8f" podStartSLOduration=1.36749241 podStartE2EDuration="2.071069492s" totalImagesPullingTime="703.577082ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:17:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:17:56.070191327 +0000 UTC m=+439.299241352" watchObservedRunningTime="2026-09-10 19:17:56.071069492 +0000 UTC m=+439.300119516"
Sep 10 19:18:07.044521 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:18:07.044439    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-88n8f"
Sep 10 19:18:07.044910 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:18:07.044608    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-gfl8n"
Sep 10 19:18:07.110154 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:18:07.110118    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-gfl8n"]
Sep 10 19:18:07.110383 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:18:07.110356    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-gfl8n" podUID="c9395621-008d-4316-8b74-f4bff4ba9151" containerName="limitador" containerID="cri-o://d16901ddb02688a10e4093d17bb32fe9b8571a29c581aaecd92621c041440642" gracePeriod=30
Sep 10 19:18:07.652261 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:18:07.652219    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-gfl8n"
Sep 10 19:18:07.707666 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:18:07.707583    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c9395621-008d-4316-8b74-f4bff4ba9151-config-file\") pod \"c9395621-008d-4316-8b74-f4bff4ba9151\" (UID: \"c9395621-008d-4316-8b74-f4bff4ba9151\") "
Sep 10 19:18:07.707666 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:18:07.707617    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w7nm5\" (UniqueName: \"kubernetes.io/projected/c9395621-008d-4316-8b74-f4bff4ba9151-kube-api-access-w7nm5\") pod \"c9395621-008d-4316-8b74-f4bff4ba9151\" (UID: \"c9395621-008d-4316-8b74-f4bff4ba9151\") "
Sep 10 19:18:07.707960 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:18:07.707937    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9395621-008d-4316-8b74-f4bff4ba9151-config-file" (OuterVolumeSpecName: "config-file") pod "c9395621-008d-4316-8b74-f4bff4ba9151" (UID: "c9395621-008d-4316-8b74-f4bff4ba9151"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 19:18:07.709686 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:18:07.709655    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9395621-008d-4316-8b74-f4bff4ba9151-kube-api-access-w7nm5" (OuterVolumeSpecName: "kube-api-access-w7nm5") pod "c9395621-008d-4316-8b74-f4bff4ba9151" (UID: "c9395621-008d-4316-8b74-f4bff4ba9151"). InnerVolumeSpecName "kube-api-access-w7nm5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 19:18:07.808748 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:18:07.808716    2546 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c9395621-008d-4316-8b74-f4bff4ba9151-config-file\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:18:07.808748 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:18:07.808742    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7nm5\" (UniqueName: \"kubernetes.io/projected/c9395621-008d-4316-8b74-f4bff4ba9151-kube-api-access-w7nm5\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:18:08.090930 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:18:08.090841    2546 generic.go:356] "Generic (PLEG): container finished" podID="c9395621-008d-4316-8b74-f4bff4ba9151" containerID="d16901ddb02688a10e4093d17bb32fe9b8571a29c581aaecd92621c041440642" exitCode=0
Sep 10 19:18:08.090930 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:18:08.090886    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-gfl8n"
Sep 10 19:18:08.091441 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:18:08.090927    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-gfl8n" event={"ID":"c9395621-008d-4316-8b74-f4bff4ba9151","Type":"ContainerDied","Data":"d16901ddb02688a10e4093d17bb32fe9b8571a29c581aaecd92621c041440642"}
Sep 10 19:18:08.091441 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:18:08.090983    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-gfl8n" event={"ID":"c9395621-008d-4316-8b74-f4bff4ba9151","Type":"ContainerDied","Data":"4a31c03ffa4b24a19967d18878ab3544309ea58edf4123b95f5e0fd4663f069c"}
Sep 10 19:18:08.091441 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:18:08.091015    2546 scope.go:117] "RemoveContainer" containerID="d16901ddb02688a10e4093d17bb32fe9b8571a29c581aaecd92621c041440642"
Sep 10 19:18:08.099797 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:18:08.099780    2546 scope.go:117] "RemoveContainer" containerID="d16901ddb02688a10e4093d17bb32fe9b8571a29c581aaecd92621c041440642"
Sep 10 19:18:08.100061 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:18:08.100040    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d16901ddb02688a10e4093d17bb32fe9b8571a29c581aaecd92621c041440642\": container with ID starting with d16901ddb02688a10e4093d17bb32fe9b8571a29c581aaecd92621c041440642 not found: ID does not exist" containerID="d16901ddb02688a10e4093d17bb32fe9b8571a29c581aaecd92621c041440642"
Sep 10 19:18:08.100107 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:18:08.100071    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d16901ddb02688a10e4093d17bb32fe9b8571a29c581aaecd92621c041440642"} err="failed to get container status \"d16901ddb02688a10e4093d17bb32fe9b8571a29c581aaecd92621c041440642\": rpc error: code = NotFound desc = could not find container \"d16901ddb02688a10e4093d17bb32fe9b8571a29c581aaecd92621c041440642\": container with ID starting with d16901ddb02688a10e4093d17bb32fe9b8571a29c581aaecd92621c041440642 not found: ID does not exist"
Sep 10 19:18:08.112805 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:18:08.112782    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-gfl8n"]
Sep 10 19:18:08.121647 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:18:08.118458    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-gfl8n"]
Sep 10 19:18:09.280129 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:18:09.280095    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c9395621-008d-4316-8b74-f4bff4ba9151" path="/var/lib/kubelet/pods/c9395621-008d-4316-8b74-f4bff4ba9151/volumes"
Sep 10 19:18:24.035656 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:18:24.035622    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-55577c8988-dvdgp"]
Sep 10 19:18:24.036012 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:18:24.035926    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9395621-008d-4316-8b74-f4bff4ba9151" containerName="limitador"
Sep 10 19:18:24.036012 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:18:24.035937    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9395621-008d-4316-8b74-f4bff4ba9151" containerName="limitador"
Sep 10 19:18:24.036012 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:18:24.036002    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9395621-008d-4316-8b74-f4bff4ba9151" containerName="limitador"
Sep 10 19:18:24.039945 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:18:24.039925    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-55577c8988-dvdgp"
Sep 10 19:18:24.047066 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:18:24.047048    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-77n47\""
Sep 10 19:18:24.047146 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:18:24.047063    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 10 19:18:24.128916 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:18:24.128882    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-55577c8988-dvdgp"]
Sep 10 19:18:24.146418 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:18:24.146389    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pptf8\" (UniqueName: \"kubernetes.io/projected/a7f402db-3f90-4dd0-b5de-f75d6aef23bf-kube-api-access-pptf8\") pod \"authorino-55577c8988-dvdgp\" (UID: \"a7f402db-3f90-4dd0-b5de-f75d6aef23bf\") " pod="kuadrant-system/authorino-55577c8988-dvdgp"
Sep 10 19:18:24.146576 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:18:24.146461    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a7f402db-3f90-4dd0-b5de-f75d6aef23bf-tls-cert\") pod \"authorino-55577c8988-dvdgp\" (UID: \"a7f402db-3f90-4dd0-b5de-f75d6aef23bf\") " pod="kuadrant-system/authorino-55577c8988-dvdgp"
Sep 10 19:18:24.247409 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:18:24.247372    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a7f402db-3f90-4dd0-b5de-f75d6aef23bf-tls-cert\") pod \"authorino-55577c8988-dvdgp\" (UID: \"a7f402db-3f90-4dd0-b5de-f75d6aef23bf\") " pod="kuadrant-system/authorino-55577c8988-dvdgp"
Sep 10 19:18:24.247573 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:18:24.247433    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pptf8\" (UniqueName: \"kubernetes.io/projected/a7f402db-3f90-4dd0-b5de-f75d6aef23bf-kube-api-access-pptf8\") pod \"authorino-55577c8988-dvdgp\" (UID: \"a7f402db-3f90-4dd0-b5de-f75d6aef23bf\") " pod="kuadrant-system/authorino-55577c8988-dvdgp"
Sep 10 19:18:24.249769 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:18:24.249748    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a7f402db-3f90-4dd0-b5de-f75d6aef23bf-tls-cert\") pod \"authorino-55577c8988-dvdgp\" (UID: \"a7f402db-3f90-4dd0-b5de-f75d6aef23bf\") " pod="kuadrant-system/authorino-55577c8988-dvdgp"
Sep 10 19:18:24.256675 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:18:24.256655    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pptf8\" (UniqueName: \"kubernetes.io/projected/a7f402db-3f90-4dd0-b5de-f75d6aef23bf-kube-api-access-pptf8\") pod \"authorino-55577c8988-dvdgp\" (UID: \"a7f402db-3f90-4dd0-b5de-f75d6aef23bf\") " pod="kuadrant-system/authorino-55577c8988-dvdgp"
Sep 10 19:18:24.348864 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:18:24.348785    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-55577c8988-dvdgp"
Sep 10 19:18:24.616635 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:18:24.616459    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-55577c8988-dvdgp"]
Sep 10 19:18:24.618802 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:18:24.618774    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda7f402db_3f90_4dd0_b5de_f75d6aef23bf.slice/crio-855880ba23b367e4980dd447732d4a7cb1403a37505fa2221b56ab647c60eeb2 WatchSource:0}: Error finding container 855880ba23b367e4980dd447732d4a7cb1403a37505fa2221b56ab647c60eeb2: Status 404 returned error can't find the container with id 855880ba23b367e4980dd447732d4a7cb1403a37505fa2221b56ab647c60eeb2
Sep 10 19:18:25.169525 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:18:25.169490    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-55577c8988-dvdgp" event={"ID":"a7f402db-3f90-4dd0-b5de-f75d6aef23bf","Type":"ContainerStarted","Data":"855880ba23b367e4980dd447732d4a7cb1403a37505fa2221b56ab647c60eeb2"}
Sep 10 19:18:27.180787 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:18:27.180749    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-55577c8988-dvdgp" event={"ID":"a7f402db-3f90-4dd0-b5de-f75d6aef23bf","Type":"ContainerStarted","Data":"a48b4551297e614a2bdb04e2fdd4f7333b1750cc0edf129b1fa9d02c8a929bf6"}
Sep 10 19:18:27.199167 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:18:27.199122    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-55577c8988-dvdgp" podStartSLOduration=1.08315538 podStartE2EDuration="3.199102326s" totalImagesPullingTime="2.115946946s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:18:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:18:27.198033232 +0000 UTC m=+470.427083257" watchObservedRunningTime="2026-09-10 19:18:27.199102326 +0000 UTC m=+470.428152351"
Sep 10 19:19:07.181721 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:19:07.181681    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-ff9f6d487-dwfz2"]
Sep 10 19:19:07.185370 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:19:07.185327    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-ff9f6d487-dwfz2"
Sep 10 19:19:07.188257 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:19:07.188230    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-lw7bv\""
Sep 10 19:19:07.188401 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:19:07.188229    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 19:19:07.188401 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:19:07.188290    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 10 19:19:07.189136 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:19:07.189117    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 10 19:19:07.189307 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:19:07.189293    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 10 19:19:07.195557 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:19:07.195531    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-ff9f6d487-dwfz2"]
Sep 10 19:19:07.298908 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:19:07.298878    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhtzd\" (UniqueName: \"kubernetes.io/projected/019b7970-6f5b-4637-b611-894853080e70-kube-api-access-xhtzd\") pod \"maas-api-ff9f6d487-dwfz2\" (UID: \"019b7970-6f5b-4637-b611-894853080e70\") " pod="odh-ai-gateway-infra/maas-api-ff9f6d487-dwfz2"
Sep 10 19:19:07.299106 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:19:07.298969    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/019b7970-6f5b-4637-b611-894853080e70-metrics-certs\") pod \"maas-api-ff9f6d487-dwfz2\" (UID: \"019b7970-6f5b-4637-b611-894853080e70\") " pod="odh-ai-gateway-infra/maas-api-ff9f6d487-dwfz2"
Sep 10 19:19:07.299106 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:19:07.299004    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/019b7970-6f5b-4637-b611-894853080e70-openshift-service-ca\") pod \"maas-api-ff9f6d487-dwfz2\" (UID: \"019b7970-6f5b-4637-b611-894853080e70\") " pod="odh-ai-gateway-infra/maas-api-ff9f6d487-dwfz2"
Sep 10 19:19:07.299106 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:19:07.299031    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/019b7970-6f5b-4637-b611-894853080e70-maas-api-tls\") pod \"maas-api-ff9f6d487-dwfz2\" (UID: \"019b7970-6f5b-4637-b611-894853080e70\") " pod="odh-ai-gateway-infra/maas-api-ff9f6d487-dwfz2"
Sep 10 19:19:07.400430 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:19:07.400401    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/019b7970-6f5b-4637-b611-894853080e70-metrics-certs\") pod \"maas-api-ff9f6d487-dwfz2\" (UID: \"019b7970-6f5b-4637-b611-894853080e70\") " pod="odh-ai-gateway-infra/maas-api-ff9f6d487-dwfz2"
Sep 10 19:19:07.400630 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:19:07.400437    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/019b7970-6f5b-4637-b611-894853080e70-openshift-service-ca\") pod \"maas-api-ff9f6d487-dwfz2\" (UID: \"019b7970-6f5b-4637-b611-894853080e70\") " pod="odh-ai-gateway-infra/maas-api-ff9f6d487-dwfz2"
Sep 10 19:19:07.400630 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:19:07.400460    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/019b7970-6f5b-4637-b611-894853080e70-maas-api-tls\") pod \"maas-api-ff9f6d487-dwfz2\" (UID: \"019b7970-6f5b-4637-b611-894853080e70\") " pod="odh-ai-gateway-infra/maas-api-ff9f6d487-dwfz2"
Sep 10 19:19:07.400630 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:19:07.400511    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhtzd\" (UniqueName: \"kubernetes.io/projected/019b7970-6f5b-4637-b611-894853080e70-kube-api-access-xhtzd\") pod \"maas-api-ff9f6d487-dwfz2\" (UID: \"019b7970-6f5b-4637-b611-894853080e70\") " pod="odh-ai-gateway-infra/maas-api-ff9f6d487-dwfz2"
Sep 10 19:19:07.401058 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:19:07.401027    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/019b7970-6f5b-4637-b611-894853080e70-openshift-service-ca\") pod \"maas-api-ff9f6d487-dwfz2\" (UID: \"019b7970-6f5b-4637-b611-894853080e70\") " pod="odh-ai-gateway-infra/maas-api-ff9f6d487-dwfz2"
Sep 10 19:19:07.402925 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:19:07.402901    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/019b7970-6f5b-4637-b611-894853080e70-metrics-certs\") pod \"maas-api-ff9f6d487-dwfz2\" (UID: \"019b7970-6f5b-4637-b611-894853080e70\") " pod="odh-ai-gateway-infra/maas-api-ff9f6d487-dwfz2"
Sep 10 19:19:07.403080 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:19:07.402938    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/019b7970-6f5b-4637-b611-894853080e70-maas-api-tls\") pod \"maas-api-ff9f6d487-dwfz2\" (UID: \"019b7970-6f5b-4637-b611-894853080e70\") " pod="odh-ai-gateway-infra/maas-api-ff9f6d487-dwfz2"
Sep 10 19:19:07.411411 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:19:07.411380    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhtzd\" (UniqueName: \"kubernetes.io/projected/019b7970-6f5b-4637-b611-894853080e70-kube-api-access-xhtzd\") pod \"maas-api-ff9f6d487-dwfz2\" (UID: \"019b7970-6f5b-4637-b611-894853080e70\") " pod="odh-ai-gateway-infra/maas-api-ff9f6d487-dwfz2"
Sep 10 19:19:07.496714 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:19:07.496638    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-ff9f6d487-dwfz2"
Sep 10 19:19:07.792683 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:19:07.792652    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod019b7970_6f5b_4637_b611_894853080e70.slice/crio-0866aa354f526cbadca8c9defdab7138aaa685f4d9db2174337305753ec0930a WatchSource:0}: Error finding container 0866aa354f526cbadca8c9defdab7138aaa685f4d9db2174337305753ec0930a: Status 404 returned error can't find the container with id 0866aa354f526cbadca8c9defdab7138aaa685f4d9db2174337305753ec0930a
Sep 10 19:19:07.811864 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:19:07.811833    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-ff9f6d487-dwfz2"]
Sep 10 19:19:08.363726 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:19:08.363683    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-ff9f6d487-dwfz2" event={"ID":"019b7970-6f5b-4637-b611-894853080e70","Type":"ContainerStarted","Data":"0866aa354f526cbadca8c9defdab7138aaa685f4d9db2174337305753ec0930a"}
Sep 10 19:19:08.389869 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:19:08.389837    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-88n8f"]
Sep 10 19:19:11.396262 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:19:11.396226    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-ff9f6d487-dwfz2" event={"ID":"019b7970-6f5b-4637-b611-894853080e70","Type":"ContainerStarted","Data":"15d58ad1a0c21007baa46d710559909d21fc79b6e0562a2f0c1f5829ef35fa69"}
Sep 10 19:19:11.396696 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:19:11.396333    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-ff9f6d487-dwfz2"
Sep 10 19:19:11.414147 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:19:11.414095    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-ff9f6d487-dwfz2" podStartSLOduration=1.2608259419999999 podStartE2EDuration="4.414079387s" totalImagesPullingTime="3.153253445s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:19:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:19:11.413127072 +0000 UTC m=+514.642177091" watchObservedRunningTime="2026-09-10 19:19:11.414079387 +0000 UTC m=+514.643129411"
Sep 10 19:19:17.407398 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:19:17.407370    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-ff9f6d487-dwfz2"
Sep 10 19:19:17.426225 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:19:17.426193    2546 generic.go:356] "Generic (PLEG): container finished" podID="698ce0a1-dce5-40d0-8bd4-c2042400e6f2" containerID="24064752b66e0f1417fe30229f5d02ab40fd9b2d68feb62e9e26f8c8925c7663" exitCode=0
Sep 10 19:19:17.426376 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:19:17.426232    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-ps4m9" event={"ID":"698ce0a1-dce5-40d0-8bd4-c2042400e6f2","Type":"ContainerDied","Data":"24064752b66e0f1417fe30229f5d02ab40fd9b2d68feb62e9e26f8c8925c7663"}
Sep 10 19:19:17.426622 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:19:17.426606    2546 scope.go:117] "RemoveContainer" containerID="24064752b66e0f1417fe30229f5d02ab40fd9b2d68feb62e9e26f8c8925c7663"
Sep 10 19:19:17.427637 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:19:17.427608    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 19:19:17.427745 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:19:17.427686    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 19:19:17.427745 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:19:17.427732    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 19:19:18.433819 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:19:18.433775    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-ps4m9" event={"ID":"698ce0a1-dce5-40d0-8bd4-c2042400e6f2","Type":"ContainerStarted","Data":"8db5ad6a029c95742ab4c475d75fb89d61271b979a1ddd9188cfff43c23b7aaa"}
Sep 10 19:19:18.781293 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:19:18.781201    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-ps4m9"
Sep 10 19:19:18.781508 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:19:18.781478    2546 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-ps4m9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body=
Sep 10 19:19:18.781570 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:19:18.781545    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-ps4m9" podUID="698ce0a1-dce5-40d0-8bd4-c2042400e6f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused"
Sep 10 19:19:19.782210 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:19:19.782172    2546 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-ps4m9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body=
Sep 10 19:19:19.782720 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:19:19.782237    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-ps4m9" podUID="698ce0a1-dce5-40d0-8bd4-c2042400e6f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused"
Sep 10 19:19:20.781887 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:19:20.781850    2546 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-ps4m9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body=
Sep 10 19:19:20.782046 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:19:20.781918    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-ps4m9" podUID="698ce0a1-dce5-40d0-8bd4-c2042400e6f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused"
Sep 10 19:19:21.782006 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:19:21.781962    2546 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-ps4m9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body=
Sep 10 19:19:21.782462 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:19:21.782050    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-ps4m9" podUID="698ce0a1-dce5-40d0-8bd4-c2042400e6f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused"
Sep 10 19:19:22.782166 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:19:22.782116    2546 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-ps4m9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body=
Sep 10 19:19:22.782708 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:19:22.782188    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-ps4m9" podUID="698ce0a1-dce5-40d0-8bd4-c2042400e6f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused"
Sep 10 19:19:23.785740 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:19:23.785703    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-ps4m9"
Sep 10 19:19:23.786134 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:19:23.786113    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-ps4m9"
Sep 10 19:19:23.786797 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:19:23.786778    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-ps4m9"
Sep 10 19:19:24.913411 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:19:24.913371    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-88n8f"]
Sep 10 19:19:36.508860 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:19:36.508773    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-88n8f"]
Sep 10 19:19:46.617793 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:19:46.617745    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-88n8f"]
Sep 10 19:20:07.422459 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:20:07.422409    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-88n8f"]
Sep 10 19:20:17.129976 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:20:17.129943    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjv585"]
Sep 10 19:20:17.133371 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:20:17.133335    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjv585"
Sep 10 19:20:17.139618 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:20:17.139511    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Sep 10 19:20:17.139618 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:20:17.139517    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-xvnhg\""
Sep 10 19:20:17.139849 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:20:17.139699    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 10 19:20:17.139849 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:20:17.139759    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 10 19:20:17.147663 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:20:17.147636    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjv585"]
Sep 10 19:20:17.276079 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:20:17.276033    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbktd\" (UniqueName: \"kubernetes.io/projected/e3d89b4d-5fa9-4ddd-88e0-6034a43ad6f8-kube-api-access-mbktd\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjv585\" (UID: \"e3d89b4d-5fa9-4ddd-88e0-6034a43ad6f8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjv585"
Sep 10 19:20:17.276260 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:20:17.276133    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3d89b4d-5fa9-4ddd-88e0-6034a43ad6f8-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjv585\" (UID: \"e3d89b4d-5fa9-4ddd-88e0-6034a43ad6f8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjv585"
Sep 10 19:20:17.276260 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:20:17.276177    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3d89b4d-5fa9-4ddd-88e0-6034a43ad6f8-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjv585\" (UID: \"e3d89b4d-5fa9-4ddd-88e0-6034a43ad6f8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjv585"
Sep 10 19:20:17.276260 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:20:17.276233    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3d89b4d-5fa9-4ddd-88e0-6034a43ad6f8-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjv585\" (UID: \"e3d89b4d-5fa9-4ddd-88e0-6034a43ad6f8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjv585"
Sep 10 19:20:17.276407 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:20:17.276296    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3d89b4d-5fa9-4ddd-88e0-6034a43ad6f8-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjv585\" (UID: \"e3d89b4d-5fa9-4ddd-88e0-6034a43ad6f8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjv585"
Sep 10 19:20:17.276407 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:20:17.276333    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3d89b4d-5fa9-4ddd-88e0-6034a43ad6f8-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjv585\" (UID: \"e3d89b4d-5fa9-4ddd-88e0-6034a43ad6f8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjv585"
Sep 10 19:20:17.377814 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:20:17.377773    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3d89b4d-5fa9-4ddd-88e0-6034a43ad6f8-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjv585\" (UID: \"e3d89b4d-5fa9-4ddd-88e0-6034a43ad6f8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjv585"
Sep 10 19:20:17.378025 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:20:17.377825    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3d89b4d-5fa9-4ddd-88e0-6034a43ad6f8-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjv585\" (UID: \"e3d89b4d-5fa9-4ddd-88e0-6034a43ad6f8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjv585"
Sep 10 19:20:17.378025 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:20:17.377864    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3d89b4d-5fa9-4ddd-88e0-6034a43ad6f8-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjv585\" (UID: \"e3d89b4d-5fa9-4ddd-88e0-6034a43ad6f8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjv585"
Sep 10 19:20:17.378025 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:20:17.377886    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3d89b4d-5fa9-4ddd-88e0-6034a43ad6f8-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjv585\" (UID: \"e3d89b4d-5fa9-4ddd-88e0-6034a43ad6f8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjv585"
Sep 10 19:20:17.378025 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:20:17.377908    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbktd\" (UniqueName: \"kubernetes.io/projected/e3d89b4d-5fa9-4ddd-88e0-6034a43ad6f8-kube-api-access-mbktd\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjv585\" (UID: \"e3d89b4d-5fa9-4ddd-88e0-6034a43ad6f8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjv585"
Sep 10 19:20:17.378234 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:20:17.378112    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3d89b4d-5fa9-4ddd-88e0-6034a43ad6f8-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjv585\" (UID: \"e3d89b4d-5fa9-4ddd-88e0-6034a43ad6f8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjv585"
Sep 10 19:20:17.378302 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:20:17.378233    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3d89b4d-5fa9-4ddd-88e0-6034a43ad6f8-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjv585\" (UID: \"e3d89b4d-5fa9-4ddd-88e0-6034a43ad6f8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjv585"
Sep 10 19:20:17.378302 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:20:17.378261    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3d89b4d-5fa9-4ddd-88e0-6034a43ad6f8-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjv585\" (UID: \"e3d89b4d-5fa9-4ddd-88e0-6034a43ad6f8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjv585"
Sep 10 19:20:17.378437 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:20:17.378318    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3d89b4d-5fa9-4ddd-88e0-6034a43ad6f8-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjv585\" (UID: \"e3d89b4d-5fa9-4ddd-88e0-6034a43ad6f8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjv585"
Sep 10 19:20:17.380421 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:20:17.380357    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3d89b4d-5fa9-4ddd-88e0-6034a43ad6f8-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjv585\" (UID: \"e3d89b4d-5fa9-4ddd-88e0-6034a43ad6f8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjv585"
Sep 10 19:20:17.380700 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:20:17.380683    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3d89b4d-5fa9-4ddd-88e0-6034a43ad6f8-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjv585\" (UID: \"e3d89b4d-5fa9-4ddd-88e0-6034a43ad6f8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjv585"
Sep 10 19:20:17.387352 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:20:17.387312    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbktd\" (UniqueName: \"kubernetes.io/projected/e3d89b4d-5fa9-4ddd-88e0-6034a43ad6f8-kube-api-access-mbktd\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjv585\" (UID: \"e3d89b4d-5fa9-4ddd-88e0-6034a43ad6f8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjv585"
Sep 10 19:20:17.444443 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:20:17.444394    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjv585"
Sep 10 19:20:17.722385 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:20:17.722200    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjv585"]
Sep 10 19:20:17.725132 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:20:17.725099    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3d89b4d_5fa9_4ddd_88e0_6034a43ad6f8.slice/crio-8db55c8aae331c9d5161a6722dc3d2fc4706e8994274bcc38810c772687fff21 WatchSource:0}: Error finding container 8db55c8aae331c9d5161a6722dc3d2fc4706e8994274bcc38810c772687fff21: Status 404 returned error can't find the container with id 8db55c8aae331c9d5161a6722dc3d2fc4706e8994274bcc38810c772687fff21
Sep 10 19:20:18.710261 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:20:18.710216    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjv585" event={"ID":"e3d89b4d-5fa9-4ddd-88e0-6034a43ad6f8","Type":"ContainerStarted","Data":"8db55c8aae331c9d5161a6722dc3d2fc4706e8994274bcc38810c772687fff21"}
Sep 10 19:20:18.936759 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:20:18.936720    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-88n8f"]
Sep 10 19:20:19.716968 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:20:19.716928    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjv585" event={"ID":"e3d89b4d-5fa9-4ddd-88e0-6034a43ad6f8","Type":"ContainerStarted","Data":"332b7f3cab637dc5b8718176bf8f88b9398090947b4ca935e14d29af37e114f0"}
Sep 10 19:20:19.737013 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:20:19.736958    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjv585" podStartSLOduration=1.523786468 podStartE2EDuration="2.736935356s" totalImagesPullingTime="1.213148888s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:20:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:20:19.736544317 +0000 UTC m=+582.965594344" watchObservedRunningTime="2026-09-10 19:20:19.736935356 +0000 UTC m=+582.965985409"
Sep 10 19:20:27.445125 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:20:27.445081    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjv585"
Sep 10 19:20:27.445586 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:20:27.445156    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjv585"
Sep 10 19:20:27.458179 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:20:27.458151    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjv585"
Sep 10 19:20:27.765621 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:20:27.765532    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjv585"
Sep 10 19:20:34.816492 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:20:34.816457    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-t4zlp"]
Sep 10 19:20:34.820985 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:20:34.820953    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t4zlp"
Sep 10 19:20:34.824536 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:20:34.824513    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Sep 10 19:20:34.831190 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:20:34.831159    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-t4zlp"]
Sep 10 19:20:34.931315 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:20:34.931269    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e58e6e4-8e40-45e2-a24f-8d1f800382be-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-t4zlp\" (UID: \"7e58e6e4-8e40-45e2-a24f-8d1f800382be\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t4zlp"
Sep 10 19:20:34.931315 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:20:34.931318    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7nsn\" (UniqueName: \"kubernetes.io/projected/7e58e6e4-8e40-45e2-a24f-8d1f800382be-kube-api-access-z7nsn\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-t4zlp\" (UID: \"7e58e6e4-8e40-45e2-a24f-8d1f800382be\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t4zlp"
Sep 10 19:20:34.931569 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:20:34.931394    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e58e6e4-8e40-45e2-a24f-8d1f800382be-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-t4zlp\" (UID: \"7e58e6e4-8e40-45e2-a24f-8d1f800382be\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t4zlp"
Sep 10 19:20:34.931569 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:20:34.931442    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e58e6e4-8e40-45e2-a24f-8d1f800382be-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-t4zlp\" (UID: \"7e58e6e4-8e40-45e2-a24f-8d1f800382be\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t4zlp"
Sep 10 19:20:34.931569 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:20:34.931474    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e58e6e4-8e40-45e2-a24f-8d1f800382be-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-t4zlp\" (UID: \"7e58e6e4-8e40-45e2-a24f-8d1f800382be\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t4zlp"
Sep 10 19:20:34.931685 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:20:34.931555    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e58e6e4-8e40-45e2-a24f-8d1f800382be-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-t4zlp\" (UID: \"7e58e6e4-8e40-45e2-a24f-8d1f800382be\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t4zlp"
Sep 10 19:20:35.032297 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:20:35.032253    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e58e6e4-8e40-45e2-a24f-8d1f800382be-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-t4zlp\" (UID: \"7e58e6e4-8e40-45e2-a24f-8d1f800382be\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t4zlp"
Sep 10 19:20:35.032297 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:20:35.032298    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7nsn\" (UniqueName: \"kubernetes.io/projected/7e58e6e4-8e40-45e2-a24f-8d1f800382be-kube-api-access-z7nsn\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-t4zlp\" (UID: \"7e58e6e4-8e40-45e2-a24f-8d1f800382be\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t4zlp"
Sep 10 19:20:35.032559 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:20:35.032322    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e58e6e4-8e40-45e2-a24f-8d1f800382be-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-t4zlp\" (UID: \"7e58e6e4-8e40-45e2-a24f-8d1f800382be\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t4zlp"
Sep 10 19:20:35.032559 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:20:35.032377    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e58e6e4-8e40-45e2-a24f-8d1f800382be-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-t4zlp\" (UID: \"7e58e6e4-8e40-45e2-a24f-8d1f800382be\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t4zlp"
Sep 10 19:20:35.032559 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:20:35.032406    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e58e6e4-8e40-45e2-a24f-8d1f800382be-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-t4zlp\" (UID: \"7e58e6e4-8e40-45e2-a24f-8d1f800382be\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t4zlp"
Sep 10 19:20:35.032559 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:20:35.032437    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e58e6e4-8e40-45e2-a24f-8d1f800382be-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-t4zlp\" (UID: \"7e58e6e4-8e40-45e2-a24f-8d1f800382be\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t4zlp"
Sep 10 19:20:35.032747 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:20:35.032703    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e58e6e4-8e40-45e2-a24f-8d1f800382be-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-t4zlp\" (UID: \"7e58e6e4-8e40-45e2-a24f-8d1f800382be\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t4zlp"
Sep 10 19:20:35.032804 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:20:35.032767    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e58e6e4-8e40-45e2-a24f-8d1f800382be-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-t4zlp\" (UID: \"7e58e6e4-8e40-45e2-a24f-8d1f800382be\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t4zlp"
Sep 10 19:20:35.032804 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:20:35.032783    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e58e6e4-8e40-45e2-a24f-8d1f800382be-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-t4zlp\" (UID: \"7e58e6e4-8e40-45e2-a24f-8d1f800382be\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t4zlp"
Sep 10 19:20:35.034787 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:20:35.034765    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e58e6e4-8e40-45e2-a24f-8d1f800382be-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-t4zlp\" (UID: \"7e58e6e4-8e40-45e2-a24f-8d1f800382be\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t4zlp"
Sep 10 19:20:35.034869 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:20:35.034850    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e58e6e4-8e40-45e2-a24f-8d1f800382be-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-t4zlp\" (UID: \"7e58e6e4-8e40-45e2-a24f-8d1f800382be\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t4zlp"
Sep 10 19:20:35.043008 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:20:35.042987    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7nsn\" (UniqueName: \"kubernetes.io/projected/7e58e6e4-8e40-45e2-a24f-8d1f800382be-kube-api-access-z7nsn\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-t4zlp\" (UID: \"7e58e6e4-8e40-45e2-a24f-8d1f800382be\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t4zlp"
Sep 10 19:20:35.134486 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:20:35.134447    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t4zlp"
Sep 10 19:20:35.407324 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:20:35.407295    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-t4zlp"]
Sep 10 19:20:35.409458 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:20:35.409430    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e58e6e4_8e40_45e2_a24f_8d1f800382be.slice/crio-0592221fbb1ff7a6d0edf3b9addd5f51bfe0098bb353a5bb5832b2a1ecec3fc5 WatchSource:0}: Error finding container 0592221fbb1ff7a6d0edf3b9addd5f51bfe0098bb353a5bb5832b2a1ecec3fc5: Status 404 returned error can't find the container with id 0592221fbb1ff7a6d0edf3b9addd5f51bfe0098bb353a5bb5832b2a1ecec3fc5
Sep 10 19:20:35.792422 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:20:35.792311    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t4zlp" event={"ID":"7e58e6e4-8e40-45e2-a24f-8d1f800382be","Type":"ContainerStarted","Data":"77f3729bc1b6914005c5f659a9a76f58a5db439ca9e0d30f201bddd923403a3c"}
Sep 10 19:20:35.792422 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:20:35.792389    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t4zlp" event={"ID":"7e58e6e4-8e40-45e2-a24f-8d1f800382be","Type":"ContainerStarted","Data":"0592221fbb1ff7a6d0edf3b9addd5f51bfe0098bb353a5bb5832b2a1ecec3fc5"}
Sep 10 19:20:35.813330 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:20:35.813276    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t4zlp" podStartSLOduration=1.626898165 podStartE2EDuration="1.813261323s" totalImagesPullingTime="186.363158ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:20:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:20:35.811144557 +0000 UTC m=+599.040194579" watchObservedRunningTime="2026-09-10 19:20:35.813261323 +0000 UTC m=+599.042311348"
Sep 10 19:20:36.829749 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:20:36.829715    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-88n8f"]
Sep 10 19:20:45.135080 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:20:45.135040    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t4zlp"
Sep 10 19:20:45.135604 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:20:45.135114    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t4zlp"
Sep 10 19:20:45.147729 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:20:45.147698    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t4zlp"
Sep 10 19:20:45.851362 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:20:45.851307    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t4zlp"
Sep 10 19:20:50.435510 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:20:50.435476    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-rp6w8"]
Sep 10 19:20:50.438934 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:20:50.438914    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rp6w8"
Sep 10 19:20:50.441412 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:20:50.441383    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 10 19:20:50.447684 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:20:50.447660    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-rp6w8"]
Sep 10 19:20:50.463306 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:20:50.463280    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0679a40-f013-4921-9453-4beb3f651af7-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rp6w8\" (UID: \"f0679a40-f013-4921-9453-4beb3f651af7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rp6w8"
Sep 10 19:20:50.463473 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:20:50.463330    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpzh5\" (UniqueName: \"kubernetes.io/projected/f0679a40-f013-4921-9453-4beb3f651af7-kube-api-access-zpzh5\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rp6w8\" (UID: \"f0679a40-f013-4921-9453-4beb3f651af7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rp6w8"
Sep 10 19:20:50.463473 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:20:50.463380    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0679a40-f013-4921-9453-4beb3f651af7-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rp6w8\" (UID: \"f0679a40-f013-4921-9453-4beb3f651af7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rp6w8"
Sep 10 19:20:50.463473 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:20:50.463437    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0679a40-f013-4921-9453-4beb3f651af7-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rp6w8\" (UID: \"f0679a40-f013-4921-9453-4beb3f651af7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rp6w8"
Sep 10 19:20:50.463473 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:20:50.463464    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0679a40-f013-4921-9453-4beb3f651af7-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rp6w8\" (UID: \"f0679a40-f013-4921-9453-4beb3f651af7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rp6w8"
Sep 10 19:20:50.463603 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:20:50.463502    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0679a40-f013-4921-9453-4beb3f651af7-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rp6w8\" (UID: \"f0679a40-f013-4921-9453-4beb3f651af7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rp6w8"
Sep 10 19:20:50.564066 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:20:50.564032    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0679a40-f013-4921-9453-4beb3f651af7-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rp6w8\" (UID: \"f0679a40-f013-4921-9453-4beb3f651af7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rp6w8"
Sep 10 19:20:50.564066 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:20:50.564070    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0679a40-f013-4921-9453-4beb3f651af7-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rp6w8\" (UID: \"f0679a40-f013-4921-9453-4beb3f651af7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rp6w8"
Sep 10 19:20:50.564300 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:20:50.564103    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0679a40-f013-4921-9453-4beb3f651af7-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rp6w8\" (UID: \"f0679a40-f013-4921-9453-4beb3f651af7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rp6w8"
Sep 10 19:20:50.564300 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:20:50.564142    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0679a40-f013-4921-9453-4beb3f651af7-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rp6w8\" (UID: \"f0679a40-f013-4921-9453-4beb3f651af7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rp6w8"
Sep 10 19:20:50.564300 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:20:50.564215    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpzh5\" (UniqueName: \"kubernetes.io/projected/f0679a40-f013-4921-9453-4beb3f651af7-kube-api-access-zpzh5\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rp6w8\" (UID: \"f0679a40-f013-4921-9453-4beb3f651af7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rp6w8"
Sep 10 19:20:50.564300 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:20:50.564266    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0679a40-f013-4921-9453-4beb3f651af7-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rp6w8\" (UID: \"f0679a40-f013-4921-9453-4beb3f651af7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rp6w8"
Sep 10 19:20:50.564551 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:20:50.564526    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0679a40-f013-4921-9453-4beb3f651af7-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rp6w8\" (UID: \"f0679a40-f013-4921-9453-4beb3f651af7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rp6w8"
Sep 10 19:20:50.564623 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:20:50.564602    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0679a40-f013-4921-9453-4beb3f651af7-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rp6w8\" (UID: \"f0679a40-f013-4921-9453-4beb3f651af7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rp6w8"
Sep 10 19:20:50.564750 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:20:50.564717    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0679a40-f013-4921-9453-4beb3f651af7-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rp6w8\" (UID: \"f0679a40-f013-4921-9453-4beb3f651af7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rp6w8"
Sep 10 19:20:50.566583 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:20:50.566561    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0679a40-f013-4921-9453-4beb3f651af7-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rp6w8\" (UID: \"f0679a40-f013-4921-9453-4beb3f651af7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rp6w8"
Sep 10 19:20:50.566842 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:20:50.566820    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0679a40-f013-4921-9453-4beb3f651af7-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rp6w8\" (UID: \"f0679a40-f013-4921-9453-4beb3f651af7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rp6w8"
Sep 10 19:20:50.572683 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:20:50.572657    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpzh5\" (UniqueName: \"kubernetes.io/projected/f0679a40-f013-4921-9453-4beb3f651af7-kube-api-access-zpzh5\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-rp6w8\" (UID: \"f0679a40-f013-4921-9453-4beb3f651af7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rp6w8"
Sep 10 19:20:50.748584 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:20:50.748495    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rp6w8"
Sep 10 19:20:51.022150 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:20:51.021877    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-rp6w8"]
Sep 10 19:20:51.024572 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:20:51.024542    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf0679a40_f013_4921_9453_4beb3f651af7.slice/crio-e01aa30c12a70f6c8962f623251228f5e843847babd27bde517e2a9bc13fb9c8 WatchSource:0}: Error finding container e01aa30c12a70f6c8962f623251228f5e843847babd27bde517e2a9bc13fb9c8: Status 404 returned error can't find the container with id e01aa30c12a70f6c8962f623251228f5e843847babd27bde517e2a9bc13fb9c8
Sep 10 19:20:51.025902 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:20:51.025886    2546 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 19:20:51.870408 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:20:51.870368    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rp6w8" event={"ID":"f0679a40-f013-4921-9453-4beb3f651af7","Type":"ContainerStarted","Data":"fdd962a3f150b162327d18465ce685b9e0b7224bfe643feb76f5430e76a9c810"}
Sep 10 19:20:51.870776 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:20:51.870421    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rp6w8" event={"ID":"f0679a40-f013-4921-9453-4beb3f651af7","Type":"ContainerStarted","Data":"e01aa30c12a70f6c8962f623251228f5e843847babd27bde517e2a9bc13fb9c8"}
Sep 10 19:20:51.890986 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:20:51.890932    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rp6w8" podStartSLOduration=1.721653616 podStartE2EDuration="1.890915467s" totalImagesPullingTime="169.261851ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:20:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:20:51.888222729 +0000 UTC m=+615.117272755" watchObservedRunningTime="2026-09-10 19:20:51.890915467 +0000 UTC m=+615.119965492"
Sep 10 19:20:51.919187 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:20:51.919154    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-88n8f"]
Sep 10 19:21:00.749131 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:21:00.749036    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rp6w8"
Sep 10 19:21:00.749131 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:21:00.749093    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rp6w8"
Sep 10 19:21:00.761537 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:21:00.761511    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rp6w8"
Sep 10 19:21:00.919286 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:21:00.919251    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-rp6w8"
Sep 10 19:21:42.229403 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:21:42.229366    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-55577c8988-dvdgp"]
Sep 10 19:21:42.229947 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:21:42.229534    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-55577c8988-dvdgp" podUID="a7f402db-3f90-4dd0-b5de-f75d6aef23bf" containerName="authorino" containerID="cri-o://a48b4551297e614a2bdb04e2fdd4f7333b1750cc0edf129b1fa9d02c8a929bf6" gracePeriod=30
Sep 10 19:21:42.486365 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:21:42.486288    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-55577c8988-dvdgp"
Sep 10 19:21:42.521394 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:21:42.521365    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a7f402db-3f90-4dd0-b5de-f75d6aef23bf-tls-cert\") pod \"a7f402db-3f90-4dd0-b5de-f75d6aef23bf\" (UID: \"a7f402db-3f90-4dd0-b5de-f75d6aef23bf\") "
Sep 10 19:21:42.531706 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:21:42.531664    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7f402db-3f90-4dd0-b5de-f75d6aef23bf-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "a7f402db-3f90-4dd0-b5de-f75d6aef23bf" (UID: "a7f402db-3f90-4dd0-b5de-f75d6aef23bf"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 19:21:42.622530 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:21:42.622497    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pptf8\" (UniqueName: \"kubernetes.io/projected/a7f402db-3f90-4dd0-b5de-f75d6aef23bf-kube-api-access-pptf8\") pod \"a7f402db-3f90-4dd0-b5de-f75d6aef23bf\" (UID: \"a7f402db-3f90-4dd0-b5de-f75d6aef23bf\") "
Sep 10 19:21:42.622691 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:21:42.622681    2546 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a7f402db-3f90-4dd0-b5de-f75d6aef23bf-tls-cert\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:21:42.624502 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:21:42.624478    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7f402db-3f90-4dd0-b5de-f75d6aef23bf-kube-api-access-pptf8" (OuterVolumeSpecName: "kube-api-access-pptf8") pod "a7f402db-3f90-4dd0-b5de-f75d6aef23bf" (UID: "a7f402db-3f90-4dd0-b5de-f75d6aef23bf"). InnerVolumeSpecName "kube-api-access-pptf8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 19:21:42.723782 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:21:42.723747    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pptf8\" (UniqueName: \"kubernetes.io/projected/a7f402db-3f90-4dd0-b5de-f75d6aef23bf-kube-api-access-pptf8\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:21:43.095000 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:21:43.094958    2546 generic.go:356] "Generic (PLEG): container finished" podID="a7f402db-3f90-4dd0-b5de-f75d6aef23bf" containerID="a48b4551297e614a2bdb04e2fdd4f7333b1750cc0edf129b1fa9d02c8a929bf6" exitCode=0
Sep 10 19:21:43.095000 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:21:43.094996    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-55577c8988-dvdgp"
Sep 10 19:21:43.095256 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:21:43.095039    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-55577c8988-dvdgp" event={"ID":"a7f402db-3f90-4dd0-b5de-f75d6aef23bf","Type":"ContainerDied","Data":"a48b4551297e614a2bdb04e2fdd4f7333b1750cc0edf129b1fa9d02c8a929bf6"}
Sep 10 19:21:43.095256 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:21:43.095084    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-55577c8988-dvdgp" event={"ID":"a7f402db-3f90-4dd0-b5de-f75d6aef23bf","Type":"ContainerDied","Data":"855880ba23b367e4980dd447732d4a7cb1403a37505fa2221b56ab647c60eeb2"}
Sep 10 19:21:43.095256 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:21:43.095104    2546 scope.go:117] "RemoveContainer" containerID="a48b4551297e614a2bdb04e2fdd4f7333b1750cc0edf129b1fa9d02c8a929bf6"
Sep 10 19:21:43.103671 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:21:43.103654    2546 scope.go:117] "RemoveContainer" containerID="a48b4551297e614a2bdb04e2fdd4f7333b1750cc0edf129b1fa9d02c8a929bf6"
Sep 10 19:21:43.103932 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:21:43.103913    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a48b4551297e614a2bdb04e2fdd4f7333b1750cc0edf129b1fa9d02c8a929bf6\": container with ID starting with a48b4551297e614a2bdb04e2fdd4f7333b1750cc0edf129b1fa9d02c8a929bf6 not found: ID does not exist" containerID="a48b4551297e614a2bdb04e2fdd4f7333b1750cc0edf129b1fa9d02c8a929bf6"
Sep 10 19:21:43.103978 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:21:43.103941    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a48b4551297e614a2bdb04e2fdd4f7333b1750cc0edf129b1fa9d02c8a929bf6"} err="failed to get container status \"a48b4551297e614a2bdb04e2fdd4f7333b1750cc0edf129b1fa9d02c8a929bf6\": rpc error: code = NotFound desc = could not find container \"a48b4551297e614a2bdb04e2fdd4f7333b1750cc0edf129b1fa9d02c8a929bf6\": container with ID starting with a48b4551297e614a2bdb04e2fdd4f7333b1750cc0edf129b1fa9d02c8a929bf6 not found: ID does not exist"
Sep 10 19:21:43.118507 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:21:43.118480    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-55577c8988-dvdgp"]
Sep 10 19:21:43.124558 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:21:43.124538    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-55577c8988-dvdgp"]
Sep 10 19:21:43.280898 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:21:43.280859    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7f402db-3f90-4dd0-b5de-f75d6aef23bf" path="/var/lib/kubelet/pods/a7f402db-3f90-4dd0-b5de-f75d6aef23bf/volumes"
Sep 10 19:22:00.134393 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:00.134360    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-6b562c-openshift-default-854c68595d-lrkhb"]
Sep 10 19:22:00.134897 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:00.134876    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7f402db-3f90-4dd0-b5de-f75d6aef23bf" containerName="authorino"
Sep 10 19:22:00.134975 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:00.134900    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7f402db-3f90-4dd0-b5de-f75d6aef23bf" containerName="authorino"
Sep 10 19:22:00.135026 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:00.134993    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7f402db-3f90-4dd0-b5de-f75d6aef23bf" containerName="authorino"
Sep 10 19:22:00.139177 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:00.139154    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6b562c-openshift-default-854c68595d-lrkhb"
Sep 10 19:22:00.143306 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:00.143282    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-6b562c-openshift-default-dockercfg-htzsx\""
Sep 10 19:22:00.152893 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:00.152863    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6b562c-openshift-default-854c68595d-lrkhb"]
Sep 10 19:22:00.157029 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:00.157001    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/919bfe97-f0e4-4fcf-8d71-58d7b93357f4-istio-token\") pod \"e2e-inf-a-6b562c-openshift-default-854c68595d-lrkhb\" (UID: \"919bfe97-f0e4-4fcf-8d71-58d7b93357f4\") " pod="openshift-ingress/e2e-inf-a-6b562c-openshift-default-854c68595d-lrkhb"
Sep 10 19:22:00.157127 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:00.157052    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/919bfe97-f0e4-4fcf-8d71-58d7b93357f4-istiod-ca-cert\") pod \"e2e-inf-a-6b562c-openshift-default-854c68595d-lrkhb\" (UID: \"919bfe97-f0e4-4fcf-8d71-58d7b93357f4\") " pod="openshift-ingress/e2e-inf-a-6b562c-openshift-default-854c68595d-lrkhb"
Sep 10 19:22:00.157254 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:00.157127    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/919bfe97-f0e4-4fcf-8d71-58d7b93357f4-istio-podinfo\") pod \"e2e-inf-a-6b562c-openshift-default-854c68595d-lrkhb\" (UID: \"919bfe97-f0e4-4fcf-8d71-58d7b93357f4\") " pod="openshift-ingress/e2e-inf-a-6b562c-openshift-default-854c68595d-lrkhb"
Sep 10 19:22:00.157320 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:00.157255    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/919bfe97-f0e4-4fcf-8d71-58d7b93357f4-workload-socket\") pod \"e2e-inf-a-6b562c-openshift-default-854c68595d-lrkhb\" (UID: \"919bfe97-f0e4-4fcf-8d71-58d7b93357f4\") " pod="openshift-ingress/e2e-inf-a-6b562c-openshift-default-854c68595d-lrkhb"
Sep 10 19:22:00.157791 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:00.157763    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97tfj\" (UniqueName: \"kubernetes.io/projected/919bfe97-f0e4-4fcf-8d71-58d7b93357f4-kube-api-access-97tfj\") pod \"e2e-inf-a-6b562c-openshift-default-854c68595d-lrkhb\" (UID: \"919bfe97-f0e4-4fcf-8d71-58d7b93357f4\") " pod="openshift-ingress/e2e-inf-a-6b562c-openshift-default-854c68595d-lrkhb"
Sep 10 19:22:00.157889 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:00.157834    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/919bfe97-f0e4-4fcf-8d71-58d7b93357f4-credential-socket\") pod \"e2e-inf-a-6b562c-openshift-default-854c68595d-lrkhb\" (UID: \"919bfe97-f0e4-4fcf-8d71-58d7b93357f4\") " pod="openshift-ingress/e2e-inf-a-6b562c-openshift-default-854c68595d-lrkhb"
Sep 10 19:22:00.157889 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:00.157861    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/919bfe97-f0e4-4fcf-8d71-58d7b93357f4-workload-certs\") pod \"e2e-inf-a-6b562c-openshift-default-854c68595d-lrkhb\" (UID: \"919bfe97-f0e4-4fcf-8d71-58d7b93357f4\") " pod="openshift-ingress/e2e-inf-a-6b562c-openshift-default-854c68595d-lrkhb"
Sep 10 19:22:00.157889 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:00.157883    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/919bfe97-f0e4-4fcf-8d71-58d7b93357f4-istio-envoy\") pod \"e2e-inf-a-6b562c-openshift-default-854c68595d-lrkhb\" (UID: \"919bfe97-f0e4-4fcf-8d71-58d7b93357f4\") " pod="openshift-ingress/e2e-inf-a-6b562c-openshift-default-854c68595d-lrkhb"
Sep 10 19:22:00.158024 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:00.157907    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/919bfe97-f0e4-4fcf-8d71-58d7b93357f4-istio-data\") pod \"e2e-inf-a-6b562c-openshift-default-854c68595d-lrkhb\" (UID: \"919bfe97-f0e4-4fcf-8d71-58d7b93357f4\") " pod="openshift-ingress/e2e-inf-a-6b562c-openshift-default-854c68595d-lrkhb"
Sep 10 19:22:00.258804 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:00.258757    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/919bfe97-f0e4-4fcf-8d71-58d7b93357f4-credential-socket\") pod \"e2e-inf-a-6b562c-openshift-default-854c68595d-lrkhb\" (UID: \"919bfe97-f0e4-4fcf-8d71-58d7b93357f4\") " pod="openshift-ingress/e2e-inf-a-6b562c-openshift-default-854c68595d-lrkhb"
Sep 10 19:22:00.258804 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:00.258801    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/919bfe97-f0e4-4fcf-8d71-58d7b93357f4-workload-certs\") pod \"e2e-inf-a-6b562c-openshift-default-854c68595d-lrkhb\" (UID: \"919bfe97-f0e4-4fcf-8d71-58d7b93357f4\") " pod="openshift-ingress/e2e-inf-a-6b562c-openshift-default-854c68595d-lrkhb"
Sep 10 19:22:00.259053 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:00.258824    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/919bfe97-f0e4-4fcf-8d71-58d7b93357f4-istio-envoy\") pod \"e2e-inf-a-6b562c-openshift-default-854c68595d-lrkhb\" (UID: \"919bfe97-f0e4-4fcf-8d71-58d7b93357f4\") " pod="openshift-ingress/e2e-inf-a-6b562c-openshift-default-854c68595d-lrkhb"
Sep 10 19:22:00.259053 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:00.258849    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/919bfe97-f0e4-4fcf-8d71-58d7b93357f4-istio-data\") pod \"e2e-inf-a-6b562c-openshift-default-854c68595d-lrkhb\" (UID: \"919bfe97-f0e4-4fcf-8d71-58d7b93357f4\") " pod="openshift-ingress/e2e-inf-a-6b562c-openshift-default-854c68595d-lrkhb"
Sep 10 19:22:00.259053 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:00.258892    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/919bfe97-f0e4-4fcf-8d71-58d7b93357f4-istio-token\") pod \"e2e-inf-a-6b562c-openshift-default-854c68595d-lrkhb\" (UID: \"919bfe97-f0e4-4fcf-8d71-58d7b93357f4\") " pod="openshift-ingress/e2e-inf-a-6b562c-openshift-default-854c68595d-lrkhb"
Sep 10 19:22:00.259053 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:00.258920    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/919bfe97-f0e4-4fcf-8d71-58d7b93357f4-istiod-ca-cert\") pod \"e2e-inf-a-6b562c-openshift-default-854c68595d-lrkhb\" (UID: \"919bfe97-f0e4-4fcf-8d71-58d7b93357f4\") " pod="openshift-ingress/e2e-inf-a-6b562c-openshift-default-854c68595d-lrkhb"
Sep 10 19:22:00.259053 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:00.258967    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/919bfe97-f0e4-4fcf-8d71-58d7b93357f4-istio-podinfo\") pod \"e2e-inf-a-6b562c-openshift-default-854c68595d-lrkhb\" (UID: \"919bfe97-f0e4-4fcf-8d71-58d7b93357f4\") " pod="openshift-ingress/e2e-inf-a-6b562c-openshift-default-854c68595d-lrkhb"
Sep 10 19:22:00.259053 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:00.259048    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/919bfe97-f0e4-4fcf-8d71-58d7b93357f4-workload-socket\") pod \"e2e-inf-a-6b562c-openshift-default-854c68595d-lrkhb\" (UID: \"919bfe97-f0e4-4fcf-8d71-58d7b93357f4\") " pod="openshift-ingress/e2e-inf-a-6b562c-openshift-default-854c68595d-lrkhb"
Sep 10 19:22:00.259391 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:00.259073    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97tfj\" (UniqueName: \"kubernetes.io/projected/919bfe97-f0e4-4fcf-8d71-58d7b93357f4-kube-api-access-97tfj\") pod \"e2e-inf-a-6b562c-openshift-default-854c68595d-lrkhb\" (UID: \"919bfe97-f0e4-4fcf-8d71-58d7b93357f4\") " pod="openshift-ingress/e2e-inf-a-6b562c-openshift-default-854c68595d-lrkhb"
Sep 10 19:22:00.259391 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:00.259248    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/919bfe97-f0e4-4fcf-8d71-58d7b93357f4-credential-socket\") pod \"e2e-inf-a-6b562c-openshift-default-854c68595d-lrkhb\" (UID: \"919bfe97-f0e4-4fcf-8d71-58d7b93357f4\") " pod="openshift-ingress/e2e-inf-a-6b562c-openshift-default-854c68595d-lrkhb"
Sep 10 19:22:00.259678 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:00.259650    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/919bfe97-f0e4-4fcf-8d71-58d7b93357f4-workload-socket\") pod \"e2e-inf-a-6b562c-openshift-default-854c68595d-lrkhb\" (UID: \"919bfe97-f0e4-4fcf-8d71-58d7b93357f4\") " pod="openshift-ingress/e2e-inf-a-6b562c-openshift-default-854c68595d-lrkhb"
Sep 10 19:22:00.259805 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:00.259692    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/919bfe97-f0e4-4fcf-8d71-58d7b93357f4-istio-data\") pod \"e2e-inf-a-6b562c-openshift-default-854c68595d-lrkhb\" (UID: \"919bfe97-f0e4-4fcf-8d71-58d7b93357f4\") " pod="openshift-ingress/e2e-inf-a-6b562c-openshift-default-854c68595d-lrkhb"
Sep 10 19:22:00.259805 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:00.259771    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/919bfe97-f0e4-4fcf-8d71-58d7b93357f4-workload-certs\") pod \"e2e-inf-a-6b562c-openshift-default-854c68595d-lrkhb\" (UID: \"919bfe97-f0e4-4fcf-8d71-58d7b93357f4\") " pod="openshift-ingress/e2e-inf-a-6b562c-openshift-default-854c68595d-lrkhb"
Sep 10 19:22:00.259975 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:00.259879    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/919bfe97-f0e4-4fcf-8d71-58d7b93357f4-istiod-ca-cert\") pod \"e2e-inf-a-6b562c-openshift-default-854c68595d-lrkhb\" (UID: \"919bfe97-f0e4-4fcf-8d71-58d7b93357f4\") " pod="openshift-ingress/e2e-inf-a-6b562c-openshift-default-854c68595d-lrkhb"
Sep 10 19:22:00.261588 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:00.261564    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/919bfe97-f0e4-4fcf-8d71-58d7b93357f4-istio-envoy\") pod \"e2e-inf-a-6b562c-openshift-default-854c68595d-lrkhb\" (UID: \"919bfe97-f0e4-4fcf-8d71-58d7b93357f4\") " pod="openshift-ingress/e2e-inf-a-6b562c-openshift-default-854c68595d-lrkhb"
Sep 10 19:22:00.261930 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:00.261909    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/919bfe97-f0e4-4fcf-8d71-58d7b93357f4-istio-podinfo\") pod \"e2e-inf-a-6b562c-openshift-default-854c68595d-lrkhb\" (UID: \"919bfe97-f0e4-4fcf-8d71-58d7b93357f4\") " pod="openshift-ingress/e2e-inf-a-6b562c-openshift-default-854c68595d-lrkhb"
Sep 10 19:22:00.268548 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:00.268523    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-97tfj\" (UniqueName: \"kubernetes.io/projected/919bfe97-f0e4-4fcf-8d71-58d7b93357f4-kube-api-access-97tfj\") pod \"e2e-inf-a-6b562c-openshift-default-854c68595d-lrkhb\" (UID: \"919bfe97-f0e4-4fcf-8d71-58d7b93357f4\") " pod="openshift-ingress/e2e-inf-a-6b562c-openshift-default-854c68595d-lrkhb"
Sep 10 19:22:00.268910 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:00.268891    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/919bfe97-f0e4-4fcf-8d71-58d7b93357f4-istio-token\") pod \"e2e-inf-a-6b562c-openshift-default-854c68595d-lrkhb\" (UID: \"919bfe97-f0e4-4fcf-8d71-58d7b93357f4\") " pod="openshift-ingress/e2e-inf-a-6b562c-openshift-default-854c68595d-lrkhb"
Sep 10 19:22:00.451776 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:00.451700    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6b562c-openshift-default-854c68595d-lrkhb"
Sep 10 19:22:00.730310 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:00.730283    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6b562c-openshift-default-854c68595d-lrkhb"]
Sep 10 19:22:00.732830 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:22:00.732800    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod919bfe97_f0e4_4fcf_8d71_58d7b93357f4.slice/crio-f9722c31e986f03dce279a587d2cf99bed325d56868998be02e8537d86051926 WatchSource:0}: Error finding container f9722c31e986f03dce279a587d2cf99bed325d56868998be02e8537d86051926: Status 404 returned error can't find the container with id f9722c31e986f03dce279a587d2cf99bed325d56868998be02e8537d86051926
Sep 10 19:22:00.734929 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:00.734894    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 19:22:00.735004 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:00.734960    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 19:22:00.735004 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:00.734988    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 19:22:01.186514 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:01.186482    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6b562c-openshift-default-854c68595d-lrkhb" event={"ID":"919bfe97-f0e4-4fcf-8d71-58d7b93357f4","Type":"ContainerStarted","Data":"891a86758496e92e8a45c37e81dbcdd6a961bf206c09bb9d3da18da26ba9225d"}
Sep 10 19:22:01.186891 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:01.186534    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6b562c-openshift-default-854c68595d-lrkhb" event={"ID":"919bfe97-f0e4-4fcf-8d71-58d7b93357f4","Type":"ContainerStarted","Data":"f9722c31e986f03dce279a587d2cf99bed325d56868998be02e8537d86051926"}
Sep 10 19:22:01.208939 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:01.208890    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-6b562c-openshift-default-854c68595d-lrkhb" podStartSLOduration=1.2088740709999999 podStartE2EDuration="1.208874071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:22:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:22:01.207642329 +0000 UTC m=+684.436692353" watchObservedRunningTime="2026-09-10 19:22:01.208874071 +0000 UTC m=+684.437924095"
Sep 10 19:22:01.452182 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:01.452033    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-6b562c-openshift-default-854c68595d-lrkhb"
Sep 10 19:22:01.457157 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:01.457129    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-6b562c-openshift-default-854c68595d-lrkhb"
Sep 10 19:22:02.191110 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:02.191077    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-6b562c-openshift-default-854c68595d-lrkhb"
Sep 10 19:22:02.191983 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:02.191965    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-6b562c-openshift-default-854c68595d-lrkhb"
Sep 10 19:22:03.629116 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:03.629080    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-88n8f"]
Sep 10 19:22:04.494056 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:04.494023    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6b562c-774f4867fc-bmk4k"]
Sep 10 19:22:04.497642 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:04.497583    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6b562c-774f4867fc-bmk4k"
Sep 10 19:22:04.504764 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:04.504739    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-6b562c-serving-cert\""
Sep 10 19:22:04.511732 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:04.511702    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6b562c-774f4867fc-bmk4k"]
Sep 10 19:22:04.597198 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:04.597159    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7e6dc57-f6fe-408b-bf57-0bf15512ade4-openshift-service-ca\") pod \"maas-api-e2e-inf-a-6b562c-774f4867fc-bmk4k\" (UID: \"f7e6dc57-f6fe-408b-bf57-0bf15512ade4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6b562c-774f4867fc-bmk4k"
Sep 10 19:22:04.597198 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:04.597200    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7e6dc57-f6fe-408b-bf57-0bf15512ade4-maas-api-tls\") pod \"maas-api-e2e-inf-a-6b562c-774f4867fc-bmk4k\" (UID: \"f7e6dc57-f6fe-408b-bf57-0bf15512ade4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6b562c-774f4867fc-bmk4k"
Sep 10 19:22:04.597446 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:04.597232    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7e6dc57-f6fe-408b-bf57-0bf15512ade4-metrics-certs\") pod \"maas-api-e2e-inf-a-6b562c-774f4867fc-bmk4k\" (UID: \"f7e6dc57-f6fe-408b-bf57-0bf15512ade4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6b562c-774f4867fc-bmk4k"
Sep 10 19:22:04.597446 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:04.597274    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n77xr\" (UniqueName: \"kubernetes.io/projected/f7e6dc57-f6fe-408b-bf57-0bf15512ade4-kube-api-access-n77xr\") pod \"maas-api-e2e-inf-a-6b562c-774f4867fc-bmk4k\" (UID: \"f7e6dc57-f6fe-408b-bf57-0bf15512ade4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6b562c-774f4867fc-bmk4k"
Sep 10 19:22:04.698457 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:04.698409    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7e6dc57-f6fe-408b-bf57-0bf15512ade4-metrics-certs\") pod \"maas-api-e2e-inf-a-6b562c-774f4867fc-bmk4k\" (UID: \"f7e6dc57-f6fe-408b-bf57-0bf15512ade4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6b562c-774f4867fc-bmk4k"
Sep 10 19:22:04.698457 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:04.698455    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n77xr\" (UniqueName: \"kubernetes.io/projected/f7e6dc57-f6fe-408b-bf57-0bf15512ade4-kube-api-access-n77xr\") pod \"maas-api-e2e-inf-a-6b562c-774f4867fc-bmk4k\" (UID: \"f7e6dc57-f6fe-408b-bf57-0bf15512ade4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6b562c-774f4867fc-bmk4k"
Sep 10 19:22:04.698993 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:04.698550    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7e6dc57-f6fe-408b-bf57-0bf15512ade4-openshift-service-ca\") pod \"maas-api-e2e-inf-a-6b562c-774f4867fc-bmk4k\" (UID: \"f7e6dc57-f6fe-408b-bf57-0bf15512ade4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6b562c-774f4867fc-bmk4k"
Sep 10 19:22:04.698993 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:04.698597    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7e6dc57-f6fe-408b-bf57-0bf15512ade4-maas-api-tls\") pod \"maas-api-e2e-inf-a-6b562c-774f4867fc-bmk4k\" (UID: \"f7e6dc57-f6fe-408b-bf57-0bf15512ade4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6b562c-774f4867fc-bmk4k"
Sep 10 19:22:04.698993 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:22:04.698726    2546 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-6b562c-serving-cert: secret "maas-api-e2e-inf-a-6b562c-serving-cert" not found
Sep 10 19:22:04.698993 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:22:04.698796    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7e6dc57-f6fe-408b-bf57-0bf15512ade4-maas-api-tls podName:f7e6dc57-f6fe-408b-bf57-0bf15512ade4 nodeName:}" failed. No retries permitted until 2026-09-10 19:22:05.19877323 +0000 UTC m=+688.427823245 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f7e6dc57-f6fe-408b-bf57-0bf15512ade4-maas-api-tls") pod "maas-api-e2e-inf-a-6b562c-774f4867fc-bmk4k" (UID: "f7e6dc57-f6fe-408b-bf57-0bf15512ade4") : secret "maas-api-e2e-inf-a-6b562c-serving-cert" not found
Sep 10 19:22:04.699550 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:04.699523    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7e6dc57-f6fe-408b-bf57-0bf15512ade4-openshift-service-ca\") pod \"maas-api-e2e-inf-a-6b562c-774f4867fc-bmk4k\" (UID: \"f7e6dc57-f6fe-408b-bf57-0bf15512ade4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6b562c-774f4867fc-bmk4k"
Sep 10 19:22:04.701771 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:04.701727    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7e6dc57-f6fe-408b-bf57-0bf15512ade4-metrics-certs\") pod \"maas-api-e2e-inf-a-6b562c-774f4867fc-bmk4k\" (UID: \"f7e6dc57-f6fe-408b-bf57-0bf15512ade4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6b562c-774f4867fc-bmk4k"
Sep 10 19:22:04.708333 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:04.708306    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n77xr\" (UniqueName: \"kubernetes.io/projected/f7e6dc57-f6fe-408b-bf57-0bf15512ade4-kube-api-access-n77xr\") pod \"maas-api-e2e-inf-a-6b562c-774f4867fc-bmk4k\" (UID: \"f7e6dc57-f6fe-408b-bf57-0bf15512ade4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6b562c-774f4867fc-bmk4k"
Sep 10 19:22:05.205030 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:05.204997    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7e6dc57-f6fe-408b-bf57-0bf15512ade4-maas-api-tls\") pod \"maas-api-e2e-inf-a-6b562c-774f4867fc-bmk4k\" (UID: \"f7e6dc57-f6fe-408b-bf57-0bf15512ade4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6b562c-774f4867fc-bmk4k"
Sep 10 19:22:05.207546 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:05.207518    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7e6dc57-f6fe-408b-bf57-0bf15512ade4-maas-api-tls\") pod \"maas-api-e2e-inf-a-6b562c-774f4867fc-bmk4k\" (UID: \"f7e6dc57-f6fe-408b-bf57-0bf15512ade4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6b562c-774f4867fc-bmk4k"
Sep 10 19:22:05.410442 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:05.410396    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6b562c-774f4867fc-bmk4k"
Sep 10 19:22:05.689922 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:05.689897    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6b562c-774f4867fc-bmk4k"]
Sep 10 19:22:05.691911 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:22:05.691885    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7e6dc57_f6fe_408b_bf57_0bf15512ade4.slice/crio-9ae313e6a32a05a915c11633534a8cc8d6d1e8a0a7fb7243a5ab5ec22360be92 WatchSource:0}: Error finding container 9ae313e6a32a05a915c11633534a8cc8d6d1e8a0a7fb7243a5ab5ec22360be92: Status 404 returned error can't find the container with id 9ae313e6a32a05a915c11633534a8cc8d6d1e8a0a7fb7243a5ab5ec22360be92
Sep 10 19:22:06.215787 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:06.215748    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6b562c-774f4867fc-bmk4k" event={"ID":"f7e6dc57-f6fe-408b-bf57-0bf15512ade4","Type":"ContainerStarted","Data":"9ae313e6a32a05a915c11633534a8cc8d6d1e8a0a7fb7243a5ab5ec22360be92"}
Sep 10 19:22:07.221957 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:07.221919    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6b562c-774f4867fc-bmk4k" event={"ID":"f7e6dc57-f6fe-408b-bf57-0bf15512ade4","Type":"ContainerStarted","Data":"46aef71e4b6cefc22010f9f9100d4aaf0e68245514c884fa83f8ce6e70082417"}
Sep 10 19:22:07.222335 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:07.222063    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6b562c-774f4867fc-bmk4k"
Sep 10 19:22:07.242682 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:07.242634    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6b562c-774f4867fc-bmk4k" podStartSLOduration=2.6294567989999997 podStartE2EDuration="3.242618862s" totalImagesPullingTime="613.162063ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:22:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:22:07.240150448 +0000 UTC m=+690.469200474" watchObservedRunningTime="2026-09-10 19:22:07.242618862 +0000 UTC m=+690.471668886"
Sep 10 19:22:08.739981 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:08.739947    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-3c7a5e-openshift-default-94b6bdd6b-6mg46"]
Sep 10 19:22:08.743424 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:08.743392    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-3c7a5e-openshift-default-94b6bdd6b-6mg46"
Sep 10 19:22:08.745950 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:08.745925    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-3c7a5e-openshift-default-dockercfg-dr2t6\""
Sep 10 19:22:08.761194 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:08.761167    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-3c7a5e-openshift-default-94b6bdd6b-6mg46"]
Sep 10 19:22:08.839406 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:08.839334    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdt5c\" (UniqueName: \"kubernetes.io/projected/cbc91d4d-adea-4eac-a841-691897e6c095-kube-api-access-vdt5c\") pod \"e2e-autores-3c7a5e-openshift-default-94b6bdd6b-6mg46\" (UID: \"cbc91d4d-adea-4eac-a841-691897e6c095\") " pod="openshift-ingress/e2e-autores-3c7a5e-openshift-default-94b6bdd6b-6mg46"
Sep 10 19:22:08.839595 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:08.839493    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbc91d4d-adea-4eac-a841-691897e6c095-istio-data\") pod \"e2e-autores-3c7a5e-openshift-default-94b6bdd6b-6mg46\" (UID: \"cbc91d4d-adea-4eac-a841-691897e6c095\") " pod="openshift-ingress/e2e-autores-3c7a5e-openshift-default-94b6bdd6b-6mg46"
Sep 10 19:22:08.839595 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:08.839540    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbc91d4d-adea-4eac-a841-691897e6c095-istio-envoy\") pod \"e2e-autores-3c7a5e-openshift-default-94b6bdd6b-6mg46\" (UID: \"cbc91d4d-adea-4eac-a841-691897e6c095\") " pod="openshift-ingress/e2e-autores-3c7a5e-openshift-default-94b6bdd6b-6mg46"
Sep 10 19:22:08.839595 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:08.839568    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbc91d4d-adea-4eac-a841-691897e6c095-istio-podinfo\") pod \"e2e-autores-3c7a5e-openshift-default-94b6bdd6b-6mg46\" (UID: \"cbc91d4d-adea-4eac-a841-691897e6c095\") " pod="openshift-ingress/e2e-autores-3c7a5e-openshift-default-94b6bdd6b-6mg46"
Sep 10 19:22:08.839719 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:08.839602    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbc91d4d-adea-4eac-a841-691897e6c095-istiod-ca-cert\") pod \"e2e-autores-3c7a5e-openshift-default-94b6bdd6b-6mg46\" (UID: \"cbc91d4d-adea-4eac-a841-691897e6c095\") " pod="openshift-ingress/e2e-autores-3c7a5e-openshift-default-94b6bdd6b-6mg46"
Sep 10 19:22:08.839719 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:08.839703    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbc91d4d-adea-4eac-a841-691897e6c095-workload-certs\") pod \"e2e-autores-3c7a5e-openshift-default-94b6bdd6b-6mg46\" (UID: \"cbc91d4d-adea-4eac-a841-691897e6c095\") " pod="openshift-ingress/e2e-autores-3c7a5e-openshift-default-94b6bdd6b-6mg46"
Sep 10 19:22:08.839789 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:08.839747    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbc91d4d-adea-4eac-a841-691897e6c095-credential-socket\") pod \"e2e-autores-3c7a5e-openshift-default-94b6bdd6b-6mg46\" (UID: \"cbc91d4d-adea-4eac-a841-691897e6c095\") " pod="openshift-ingress/e2e-autores-3c7a5e-openshift-default-94b6bdd6b-6mg46"
Sep 10 19:22:08.839789 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:08.839778    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbc91d4d-adea-4eac-a841-691897e6c095-workload-socket\") pod \"e2e-autores-3c7a5e-openshift-default-94b6bdd6b-6mg46\" (UID: \"cbc91d4d-adea-4eac-a841-691897e6c095\") " pod="openshift-ingress/e2e-autores-3c7a5e-openshift-default-94b6bdd6b-6mg46"
Sep 10 19:22:08.839896 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:08.839876    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbc91d4d-adea-4eac-a841-691897e6c095-istio-token\") pod \"e2e-autores-3c7a5e-openshift-default-94b6bdd6b-6mg46\" (UID: \"cbc91d4d-adea-4eac-a841-691897e6c095\") " pod="openshift-ingress/e2e-autores-3c7a5e-openshift-default-94b6bdd6b-6mg46"
Sep 10 19:22:08.940747 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:08.940705    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbc91d4d-adea-4eac-a841-691897e6c095-credential-socket\") pod \"e2e-autores-3c7a5e-openshift-default-94b6bdd6b-6mg46\" (UID: \"cbc91d4d-adea-4eac-a841-691897e6c095\") " pod="openshift-ingress/e2e-autores-3c7a5e-openshift-default-94b6bdd6b-6mg46"
Sep 10 19:22:08.940936 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:08.940760    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbc91d4d-adea-4eac-a841-691897e6c095-workload-socket\") pod \"e2e-autores-3c7a5e-openshift-default-94b6bdd6b-6mg46\" (UID: \"cbc91d4d-adea-4eac-a841-691897e6c095\") " pod="openshift-ingress/e2e-autores-3c7a5e-openshift-default-94b6bdd6b-6mg46"
Sep 10 19:22:08.940936 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:08.940820    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbc91d4d-adea-4eac-a841-691897e6c095-istio-token\") pod \"e2e-autores-3c7a5e-openshift-default-94b6bdd6b-6mg46\" (UID: \"cbc91d4d-adea-4eac-a841-691897e6c095\") " pod="openshift-ingress/e2e-autores-3c7a5e-openshift-default-94b6bdd6b-6mg46"
Sep 10 19:22:08.940936 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:08.940870    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdt5c\" (UniqueName: \"kubernetes.io/projected/cbc91d4d-adea-4eac-a841-691897e6c095-kube-api-access-vdt5c\") pod \"e2e-autores-3c7a5e-openshift-default-94b6bdd6b-6mg46\" (UID: \"cbc91d4d-adea-4eac-a841-691897e6c095\") " pod="openshift-ingress/e2e-autores-3c7a5e-openshift-default-94b6bdd6b-6mg46"
Sep 10 19:22:08.940936 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:08.940932    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbc91d4d-adea-4eac-a841-691897e6c095-istio-data\") pod \"e2e-autores-3c7a5e-openshift-default-94b6bdd6b-6mg46\" (UID: \"cbc91d4d-adea-4eac-a841-691897e6c095\") " pod="openshift-ingress/e2e-autores-3c7a5e-openshift-default-94b6bdd6b-6mg46"
Sep 10 19:22:08.941144 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:08.940960    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbc91d4d-adea-4eac-a841-691897e6c095-istio-envoy\") pod \"e2e-autores-3c7a5e-openshift-default-94b6bdd6b-6mg46\" (UID: \"cbc91d4d-adea-4eac-a841-691897e6c095\") " pod="openshift-ingress/e2e-autores-3c7a5e-openshift-default-94b6bdd6b-6mg46"
Sep 10 19:22:08.941144 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:08.940985    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbc91d4d-adea-4eac-a841-691897e6c095-istio-podinfo\") pod \"e2e-autores-3c7a5e-openshift-default-94b6bdd6b-6mg46\" (UID: \"cbc91d4d-adea-4eac-a841-691897e6c095\") " pod="openshift-ingress/e2e-autores-3c7a5e-openshift-default-94b6bdd6b-6mg46"
Sep 10 19:22:08.941144 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:08.941011    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbc91d4d-adea-4eac-a841-691897e6c095-istiod-ca-cert\") pod \"e2e-autores-3c7a5e-openshift-default-94b6bdd6b-6mg46\" (UID: \"cbc91d4d-adea-4eac-a841-691897e6c095\") " pod="openshift-ingress/e2e-autores-3c7a5e-openshift-default-94b6bdd6b-6mg46"
Sep 10 19:22:08.941144 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:08.941059    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbc91d4d-adea-4eac-a841-691897e6c095-workload-certs\") pod \"e2e-autores-3c7a5e-openshift-default-94b6bdd6b-6mg46\" (UID: \"cbc91d4d-adea-4eac-a841-691897e6c095\") " pod="openshift-ingress/e2e-autores-3c7a5e-openshift-default-94b6bdd6b-6mg46"
Sep 10 19:22:08.941391 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:08.941156    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbc91d4d-adea-4eac-a841-691897e6c095-workload-socket\") pod \"e2e-autores-3c7a5e-openshift-default-94b6bdd6b-6mg46\" (UID: \"cbc91d4d-adea-4eac-a841-691897e6c095\") " pod="openshift-ingress/e2e-autores-3c7a5e-openshift-default-94b6bdd6b-6mg46"
Sep 10 19:22:08.941471 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:08.941404    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbc91d4d-adea-4eac-a841-691897e6c095-istio-data\") pod \"e2e-autores-3c7a5e-openshift-default-94b6bdd6b-6mg46\" (UID: \"cbc91d4d-adea-4eac-a841-691897e6c095\") " pod="openshift-ingress/e2e-autores-3c7a5e-openshift-default-94b6bdd6b-6mg46"
Sep 10 19:22:08.941746 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:08.941716    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbc91d4d-adea-4eac-a841-691897e6c095-credential-socket\") pod \"e2e-autores-3c7a5e-openshift-default-94b6bdd6b-6mg46\" (UID: \"cbc91d4d-adea-4eac-a841-691897e6c095\") " pod="openshift-ingress/e2e-autores-3c7a5e-openshift-default-94b6bdd6b-6mg46"
Sep 10 19:22:08.941746 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:08.941740    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbc91d4d-adea-4eac-a841-691897e6c095-workload-certs\") pod \"e2e-autores-3c7a5e-openshift-default-94b6bdd6b-6mg46\" (UID: \"cbc91d4d-adea-4eac-a841-691897e6c095\") " pod="openshift-ingress/e2e-autores-3c7a5e-openshift-default-94b6bdd6b-6mg46"
Sep 10 19:22:08.941910 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:08.941849    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbc91d4d-adea-4eac-a841-691897e6c095-istiod-ca-cert\") pod \"e2e-autores-3c7a5e-openshift-default-94b6bdd6b-6mg46\" (UID: \"cbc91d4d-adea-4eac-a841-691897e6c095\") " pod="openshift-ingress/e2e-autores-3c7a5e-openshift-default-94b6bdd6b-6mg46"
Sep 10 19:22:08.943599 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:08.943577    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbc91d4d-adea-4eac-a841-691897e6c095-istio-podinfo\") pod \"e2e-autores-3c7a5e-openshift-default-94b6bdd6b-6mg46\" (UID: \"cbc91d4d-adea-4eac-a841-691897e6c095\") " pod="openshift-ingress/e2e-autores-3c7a5e-openshift-default-94b6bdd6b-6mg46"
Sep 10 19:22:08.943772 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:08.943753    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbc91d4d-adea-4eac-a841-691897e6c095-istio-envoy\") pod \"e2e-autores-3c7a5e-openshift-default-94b6bdd6b-6mg46\" (UID: \"cbc91d4d-adea-4eac-a841-691897e6c095\") " pod="openshift-ingress/e2e-autores-3c7a5e-openshift-default-94b6bdd6b-6mg46"
Sep 10 19:22:08.950636 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:08.950615    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdt5c\" (UniqueName: \"kubernetes.io/projected/cbc91d4d-adea-4eac-a841-691897e6c095-kube-api-access-vdt5c\") pod \"e2e-autores-3c7a5e-openshift-default-94b6bdd6b-6mg46\" (UID: \"cbc91d4d-adea-4eac-a841-691897e6c095\") " pod="openshift-ingress/e2e-autores-3c7a5e-openshift-default-94b6bdd6b-6mg46"
Sep 10 19:22:08.952777 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:08.952756    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbc91d4d-adea-4eac-a841-691897e6c095-istio-token\") pod \"e2e-autores-3c7a5e-openshift-default-94b6bdd6b-6mg46\" (UID: \"cbc91d4d-adea-4eac-a841-691897e6c095\") " pod="openshift-ingress/e2e-autores-3c7a5e-openshift-default-94b6bdd6b-6mg46"
Sep 10 19:22:09.054618 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:09.054528    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-3c7a5e-openshift-default-94b6bdd6b-6mg46"
Sep 10 19:22:09.336563 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:09.336538    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-3c7a5e-openshift-default-94b6bdd6b-6mg46"]
Sep 10 19:22:09.338652 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:22:09.338621    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbc91d4d_adea_4eac_a841_691897e6c095.slice/crio-26c6f242be4a50db84646155aa475d5b618cac93b73e9fdd824d659933451a48 WatchSource:0}: Error finding container 26c6f242be4a50db84646155aa475d5b618cac93b73e9fdd824d659933451a48: Status 404 returned error can't find the container with id 26c6f242be4a50db84646155aa475d5b618cac93b73e9fdd824d659933451a48
Sep 10 19:22:09.340962 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:09.340929    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 19:22:09.341039 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:09.340994    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 19:22:09.341039 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:09.341025    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 19:22:10.237187 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:10.237136    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-3c7a5e-openshift-default-94b6bdd6b-6mg46" event={"ID":"cbc91d4d-adea-4eac-a841-691897e6c095","Type":"ContainerStarted","Data":"19cb18b95c32205d955306781edf3ea1b47b5ff57fec709efb7edde8492cdf17"}
Sep 10 19:22:10.237667 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:10.237229    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-3c7a5e-openshift-default-94b6bdd6b-6mg46" event={"ID":"cbc91d4d-adea-4eac-a841-691897e6c095","Type":"ContainerStarted","Data":"26c6f242be4a50db84646155aa475d5b618cac93b73e9fdd824d659933451a48"}
Sep 10 19:22:10.258042 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:10.257980    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-3c7a5e-openshift-default-94b6bdd6b-6mg46" podStartSLOduration=2.257959594 podStartE2EDuration="2.257959594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:22:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:22:10.257383271 +0000 UTC m=+693.486433301" watchObservedRunningTime="2026-09-10 19:22:10.257959594 +0000 UTC m=+693.487009621"
Sep 10 19:22:11.055048 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:11.055009    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-3c7a5e-openshift-default-94b6bdd6b-6mg46"
Sep 10 19:22:11.061476 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:11.061446    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-3c7a5e-openshift-default-94b6bdd6b-6mg46"
Sep 10 19:22:11.241611 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:11.241574    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-3c7a5e-openshift-default-94b6bdd6b-6mg46"
Sep 10 19:22:11.242714 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:11.242690    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-3c7a5e-openshift-default-94b6bdd6b-6mg46"
Sep 10 19:22:13.231576 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:13.231550    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6b562c-774f4867fc-bmk4k"
Sep 10 19:22:13.695172 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:13.695138    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-3c7a5e-59b6d88ff5-5xrmw"]
Sep 10 19:22:13.698746 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:13.698722    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-3c7a5e-59b6d88ff5-5xrmw"
Sep 10 19:22:13.701334 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:13.701316    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-3c7a5e-serving-cert\""
Sep 10 19:22:13.712675 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:13.712637    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-3c7a5e-59b6d88ff5-5xrmw"]
Sep 10 19:22:13.796571 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:13.796531    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/362d18ff-1ad6-4d25-8014-22b9e065d8fb-metrics-certs\") pod \"maas-api-e2e-autores-3c7a5e-59b6d88ff5-5xrmw\" (UID: \"362d18ff-1ad6-4d25-8014-22b9e065d8fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-3c7a5e-59b6d88ff5-5xrmw"
Sep 10 19:22:13.796745 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:13.796593    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsrrn\" (UniqueName: \"kubernetes.io/projected/362d18ff-1ad6-4d25-8014-22b9e065d8fb-kube-api-access-jsrrn\") pod \"maas-api-e2e-autores-3c7a5e-59b6d88ff5-5xrmw\" (UID: \"362d18ff-1ad6-4d25-8014-22b9e065d8fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-3c7a5e-59b6d88ff5-5xrmw"
Sep 10 19:22:13.796745 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:13.796614    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/362d18ff-1ad6-4d25-8014-22b9e065d8fb-maas-api-tls\") pod \"maas-api-e2e-autores-3c7a5e-59b6d88ff5-5xrmw\" (UID: \"362d18ff-1ad6-4d25-8014-22b9e065d8fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-3c7a5e-59b6d88ff5-5xrmw"
Sep 10 19:22:13.796745 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:13.796682    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/362d18ff-1ad6-4d25-8014-22b9e065d8fb-openshift-service-ca\") pod \"maas-api-e2e-autores-3c7a5e-59b6d88ff5-5xrmw\" (UID: \"362d18ff-1ad6-4d25-8014-22b9e065d8fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-3c7a5e-59b6d88ff5-5xrmw"
Sep 10 19:22:13.897416 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:13.897379    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/362d18ff-1ad6-4d25-8014-22b9e065d8fb-metrics-certs\") pod \"maas-api-e2e-autores-3c7a5e-59b6d88ff5-5xrmw\" (UID: \"362d18ff-1ad6-4d25-8014-22b9e065d8fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-3c7a5e-59b6d88ff5-5xrmw"
Sep 10 19:22:13.897585 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:13.897449    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsrrn\" (UniqueName: \"kubernetes.io/projected/362d18ff-1ad6-4d25-8014-22b9e065d8fb-kube-api-access-jsrrn\") pod \"maas-api-e2e-autores-3c7a5e-59b6d88ff5-5xrmw\" (UID: \"362d18ff-1ad6-4d25-8014-22b9e065d8fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-3c7a5e-59b6d88ff5-5xrmw"
Sep 10 19:22:13.897585 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:13.897474    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/362d18ff-1ad6-4d25-8014-22b9e065d8fb-maas-api-tls\") pod \"maas-api-e2e-autores-3c7a5e-59b6d88ff5-5xrmw\" (UID: \"362d18ff-1ad6-4d25-8014-22b9e065d8fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-3c7a5e-59b6d88ff5-5xrmw"
Sep 10 19:22:13.897585 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:13.897496    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/362d18ff-1ad6-4d25-8014-22b9e065d8fb-openshift-service-ca\") pod \"maas-api-e2e-autores-3c7a5e-59b6d88ff5-5xrmw\" (UID: \"362d18ff-1ad6-4d25-8014-22b9e065d8fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-3c7a5e-59b6d88ff5-5xrmw"
Sep 10 19:22:13.897692 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:22:13.897620    2546 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-3c7a5e-serving-cert: secret "maas-api-e2e-autores-3c7a5e-serving-cert" not found
Sep 10 19:22:13.897725 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:22:13.897711    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/362d18ff-1ad6-4d25-8014-22b9e065d8fb-maas-api-tls podName:362d18ff-1ad6-4d25-8014-22b9e065d8fb nodeName:}" failed. No retries permitted until 2026-09-10 19:22:14.397688986 +0000 UTC m=+697.626739002 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/362d18ff-1ad6-4d25-8014-22b9e065d8fb-maas-api-tls") pod "maas-api-e2e-autores-3c7a5e-59b6d88ff5-5xrmw" (UID: "362d18ff-1ad6-4d25-8014-22b9e065d8fb") : secret "maas-api-e2e-autores-3c7a5e-serving-cert" not found
Sep 10 19:22:13.898194 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:13.898171    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/362d18ff-1ad6-4d25-8014-22b9e065d8fb-openshift-service-ca\") pod \"maas-api-e2e-autores-3c7a5e-59b6d88ff5-5xrmw\" (UID: \"362d18ff-1ad6-4d25-8014-22b9e065d8fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-3c7a5e-59b6d88ff5-5xrmw"
Sep 10 19:22:13.900007 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:13.899983    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/362d18ff-1ad6-4d25-8014-22b9e065d8fb-metrics-certs\") pod \"maas-api-e2e-autores-3c7a5e-59b6d88ff5-5xrmw\" (UID: \"362d18ff-1ad6-4d25-8014-22b9e065d8fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-3c7a5e-59b6d88ff5-5xrmw"
Sep 10 19:22:13.906632 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:13.906611    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsrrn\" (UniqueName: \"kubernetes.io/projected/362d18ff-1ad6-4d25-8014-22b9e065d8fb-kube-api-access-jsrrn\") pod \"maas-api-e2e-autores-3c7a5e-59b6d88ff5-5xrmw\" (UID: \"362d18ff-1ad6-4d25-8014-22b9e065d8fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-3c7a5e-59b6d88ff5-5xrmw"
Sep 10 19:22:14.402722 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:14.402561    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/362d18ff-1ad6-4d25-8014-22b9e065d8fb-maas-api-tls\") pod \"maas-api-e2e-autores-3c7a5e-59b6d88ff5-5xrmw\" (UID: \"362d18ff-1ad6-4d25-8014-22b9e065d8fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-3c7a5e-59b6d88ff5-5xrmw"
Sep 10 19:22:14.403216 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:22:14.402776    2546 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-3c7a5e-serving-cert: secret "maas-api-e2e-autores-3c7a5e-serving-cert" not found
Sep 10 19:22:14.403216 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:22:14.402853    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/362d18ff-1ad6-4d25-8014-22b9e065d8fb-maas-api-tls podName:362d18ff-1ad6-4d25-8014-22b9e065d8fb nodeName:}" failed. No retries permitted until 2026-09-10 19:22:15.40282946 +0000 UTC m=+698.631879463 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/362d18ff-1ad6-4d25-8014-22b9e065d8fb-maas-api-tls") pod "maas-api-e2e-autores-3c7a5e-59b6d88ff5-5xrmw" (UID: "362d18ff-1ad6-4d25-8014-22b9e065d8fb") : secret "maas-api-e2e-autores-3c7a5e-serving-cert" not found
Sep 10 19:22:15.415359 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:15.415312    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/362d18ff-1ad6-4d25-8014-22b9e065d8fb-maas-api-tls\") pod \"maas-api-e2e-autores-3c7a5e-59b6d88ff5-5xrmw\" (UID: \"362d18ff-1ad6-4d25-8014-22b9e065d8fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-3c7a5e-59b6d88ff5-5xrmw"
Sep 10 19:22:15.417735 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:15.417709    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/362d18ff-1ad6-4d25-8014-22b9e065d8fb-maas-api-tls\") pod \"maas-api-e2e-autores-3c7a5e-59b6d88ff5-5xrmw\" (UID: \"362d18ff-1ad6-4d25-8014-22b9e065d8fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-3c7a5e-59b6d88ff5-5xrmw"
Sep 10 19:22:15.513920 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:15.513883    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-3c7a5e-59b6d88ff5-5xrmw"
Sep 10 19:22:15.813079 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:15.812918    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-3c7a5e-59b6d88ff5-5xrmw"]
Sep 10 19:22:15.815455 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:22:15.815417    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod362d18ff_1ad6_4d25_8014_22b9e065d8fb.slice/crio-faf2c5253734cf48ed610564d21cbb6c39e2111feae7aa507138b0743b31db7c WatchSource:0}: Error finding container faf2c5253734cf48ed610564d21cbb6c39e2111feae7aa507138b0743b31db7c: Status 404 returned error can't find the container with id faf2c5253734cf48ed610564d21cbb6c39e2111feae7aa507138b0743b31db7c
Sep 10 19:22:16.309170 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:16.309126    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-3c7a5e-59b6d88ff5-5xrmw" event={"ID":"362d18ff-1ad6-4d25-8014-22b9e065d8fb","Type":"ContainerStarted","Data":"faf2c5253734cf48ed610564d21cbb6c39e2111feae7aa507138b0743b31db7c"}
Sep 10 19:22:17.316389 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:17.316328    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-3c7a5e-59b6d88ff5-5xrmw" event={"ID":"362d18ff-1ad6-4d25-8014-22b9e065d8fb","Type":"ContainerStarted","Data":"039297f75ee84b28b6f601eeaccf77651ffbf3abe863cf835c51b9278bba8e6d"}
Sep 10 19:22:17.316753 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:17.316417    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-3c7a5e-59b6d88ff5-5xrmw"
Sep 10 19:22:17.336722 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:17.336652    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-3c7a5e-59b6d88ff5-5xrmw" podStartSLOduration=3.505306097 podStartE2EDuration="4.336634312s" totalImagesPullingTime="831.328215ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:22:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:22:17.334326223 +0000 UTC m=+700.563376249" watchObservedRunningTime="2026-09-10 19:22:17.336634312 +0000 UTC m=+700.565684337"
Sep 10 19:22:19.621371 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:19.621328    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-88n8f"]
Sep 10 19:22:21.336853 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:21.336820    2546 generic.go:356] "Generic (PLEG): container finished" podID="919bfe97-f0e4-4fcf-8d71-58d7b93357f4" containerID="891a86758496e92e8a45c37e81dbcdd6a961bf206c09bb9d3da18da26ba9225d" exitCode=0
Sep 10 19:22:21.337297 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:21.336893    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6b562c-openshift-default-854c68595d-lrkhb" event={"ID":"919bfe97-f0e4-4fcf-8d71-58d7b93357f4","Type":"ContainerDied","Data":"891a86758496e92e8a45c37e81dbcdd6a961bf206c09bb9d3da18da26ba9225d"}
Sep 10 19:22:21.337297 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:21.337264    2546 scope.go:117] "RemoveContainer" containerID="891a86758496e92e8a45c37e81dbcdd6a961bf206c09bb9d3da18da26ba9225d"
Sep 10 19:22:21.338371 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:21.338317    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 19:22:21.338495 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:21.338417    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 19:22:21.338495 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:21.338475    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 19:22:22.343027 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:22.342993    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6b562c-openshift-default-854c68595d-lrkhb" event={"ID":"919bfe97-f0e4-4fcf-8d71-58d7b93357f4","Type":"ContainerStarted","Data":"d44cbc2f89a78d27f24bb4548dd4b273a11279c2b0cf6bd4abaf49510355094c"}
Sep 10 19:22:22.451835 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:22.451799    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-6b562c-openshift-default-854c68595d-lrkhb"
Sep 10 19:22:22.452124 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:22.452105    2546 patch_prober.go:28] interesting pod/e2e-inf-a-6b562c-openshift-default-854c68595d-lrkhb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Sep 10 19:22:22.452178 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:22.452161    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6b562c-openshift-default-854c68595d-lrkhb" podUID="919bfe97-f0e4-4fcf-8d71-58d7b93357f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Sep 10 19:22:23.326059 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:23.326032    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-3c7a5e-59b6d88ff5-5xrmw"
Sep 10 19:22:23.452180 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:23.452145    2546 patch_prober.go:28] interesting pod/e2e-inf-a-6b562c-openshift-default-854c68595d-lrkhb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Sep 10 19:22:23.452554 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:23.452206    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6b562c-openshift-default-854c68595d-lrkhb" podUID="919bfe97-f0e4-4fcf-8d71-58d7b93357f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Sep 10 19:22:24.452417 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:24.452383    2546 patch_prober.go:28] interesting pod/e2e-inf-a-6b562c-openshift-default-854c68595d-lrkhb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Sep 10 19:22:24.452779 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:24.452443    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6b562c-openshift-default-854c68595d-lrkhb" podUID="919bfe97-f0e4-4fcf-8d71-58d7b93357f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Sep 10 19:22:25.453062 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:25.453022    2546 patch_prober.go:28] interesting pod/e2e-inf-a-6b562c-openshift-default-854c68595d-lrkhb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Sep 10 19:22:25.453491 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:25.453079    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6b562c-openshift-default-854c68595d-lrkhb" podUID="919bfe97-f0e4-4fcf-8d71-58d7b93357f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Sep 10 19:22:26.453025 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:26.452981    2546 patch_prober.go:28] interesting pod/e2e-inf-a-6b562c-openshift-default-854c68595d-lrkhb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Sep 10 19:22:26.453207 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:26.453050    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6b562c-openshift-default-854c68595d-lrkhb" podUID="919bfe97-f0e4-4fcf-8d71-58d7b93357f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Sep 10 19:22:27.452372 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:27.452322    2546 patch_prober.go:28] interesting pod/e2e-inf-a-6b562c-openshift-default-854c68595d-lrkhb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Sep 10 19:22:27.452559 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:27.452395    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6b562c-openshift-default-854c68595d-lrkhb" podUID="919bfe97-f0e4-4fcf-8d71-58d7b93357f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Sep 10 19:22:28.456260 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:28.456228    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-6b562c-openshift-default-854c68595d-lrkhb"
Sep 10 19:22:28.456728 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:28.456713    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-6b562c-openshift-default-854c68595d-lrkhb"
Sep 10 19:22:28.457485 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:28.457463    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-6b562c-openshift-default-854c68595d-lrkhb"
Sep 10 19:22:30.392303 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:30.392261    2546 generic.go:356] "Generic (PLEG): container finished" podID="cbc91d4d-adea-4eac-a841-691897e6c095" containerID="19cb18b95c32205d955306781edf3ea1b47b5ff57fec709efb7edde8492cdf17" exitCode=0
Sep 10 19:22:30.392303 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:30.392292    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-3c7a5e-openshift-default-94b6bdd6b-6mg46" event={"ID":"cbc91d4d-adea-4eac-a841-691897e6c095","Type":"ContainerDied","Data":"19cb18b95c32205d955306781edf3ea1b47b5ff57fec709efb7edde8492cdf17"}
Sep 10 19:22:30.392918 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:30.392744    2546 scope.go:117] "RemoveContainer" containerID="19cb18b95c32205d955306781edf3ea1b47b5ff57fec709efb7edde8492cdf17"
Sep 10 19:22:30.394129 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:30.394089    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 19:22:30.394226 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:30.394183    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 19:22:30.394264 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:30.394227    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 19:22:31.399378 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:31.399329    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-3c7a5e-openshift-default-94b6bdd6b-6mg46" event={"ID":"cbc91d4d-adea-4eac-a841-691897e6c095","Type":"ContainerStarted","Data":"57046ff5721443c014639bfd2129f083df4ba010a7958c631b2795da77fbb817"}
Sep 10 19:22:32.055412 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:32.055374    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-3c7a5e-openshift-default-94b6bdd6b-6mg46"
Sep 10 19:22:32.055639 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:32.055617    2546 patch_prober.go:28] interesting pod/e2e-autores-3c7a5e-openshift-default-94b6bdd6b-6mg46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Sep 10 19:22:32.055688 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:32.055664    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-3c7a5e-openshift-default-94b6bdd6b-6mg46" podUID="cbc91d4d-adea-4eac-a841-691897e6c095" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Sep 10 19:22:33.055360 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:33.055304    2546 patch_prober.go:28] interesting pod/e2e-autores-3c7a5e-openshift-default-94b6bdd6b-6mg46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Sep 10 19:22:33.055795 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:33.055385    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-3c7a5e-openshift-default-94b6bdd6b-6mg46" podUID="cbc91d4d-adea-4eac-a841-691897e6c095" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Sep 10 19:22:34.055491 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:34.055459    2546 patch_prober.go:28] interesting pod/e2e-autores-3c7a5e-openshift-default-94b6bdd6b-6mg46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Sep 10 19:22:34.055883 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:34.055514    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-3c7a5e-openshift-default-94b6bdd6b-6mg46" podUID="cbc91d4d-adea-4eac-a841-691897e6c095" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Sep 10 19:22:34.132788 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:34.132754    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b960a2-7f955bb764-ftlgf"]
Sep 10 19:22:34.136170 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:34.136150    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b960a2-7f955bb764-ftlgf"
Sep 10 19:22:34.139036 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:34.139012    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-b960a2-serving-cert\""
Sep 10 19:22:34.152219 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:34.152193    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b960a2-7f955bb764-ftlgf"]
Sep 10 19:22:34.310976 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:34.310886    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4k49t\" (UniqueName: \"kubernetes.io/projected/1bd47a6e-dd0f-47ae-b0a3-d25207d9b209-kube-api-access-4k49t\") pod \"maas-api-e2e-inf-b-b960a2-7f955bb764-ftlgf\" (UID: \"1bd47a6e-dd0f-47ae-b0a3-d25207d9b209\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b960a2-7f955bb764-ftlgf"
Sep 10 19:22:34.310976 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:34.310948    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bd47a6e-dd0f-47ae-b0a3-d25207d9b209-openshift-service-ca\") pod \"maas-api-e2e-inf-b-b960a2-7f955bb764-ftlgf\" (UID: \"1bd47a6e-dd0f-47ae-b0a3-d25207d9b209\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b960a2-7f955bb764-ftlgf"
Sep 10 19:22:34.311163 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:34.311039    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1bd47a6e-dd0f-47ae-b0a3-d25207d9b209-maas-api-tls\") pod \"maas-api-e2e-inf-b-b960a2-7f955bb764-ftlgf\" (UID: \"1bd47a6e-dd0f-47ae-b0a3-d25207d9b209\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b960a2-7f955bb764-ftlgf"
Sep 10 19:22:34.311163 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:34.311069    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1bd47a6e-dd0f-47ae-b0a3-d25207d9b209-metrics-certs\") pod \"maas-api-e2e-inf-b-b960a2-7f955bb764-ftlgf\" (UID: \"1bd47a6e-dd0f-47ae-b0a3-d25207d9b209\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b960a2-7f955bb764-ftlgf"
Sep 10 19:22:34.411940 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:34.411909    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bd47a6e-dd0f-47ae-b0a3-d25207d9b209-openshift-service-ca\") pod \"maas-api-e2e-inf-b-b960a2-7f955bb764-ftlgf\" (UID: \"1bd47a6e-dd0f-47ae-b0a3-d25207d9b209\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b960a2-7f955bb764-ftlgf"
Sep 10 19:22:34.412142 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:34.411978    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1bd47a6e-dd0f-47ae-b0a3-d25207d9b209-maas-api-tls\") pod \"maas-api-e2e-inf-b-b960a2-7f955bb764-ftlgf\" (UID: \"1bd47a6e-dd0f-47ae-b0a3-d25207d9b209\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b960a2-7f955bb764-ftlgf"
Sep 10 19:22:34.412142 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:34.412008    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1bd47a6e-dd0f-47ae-b0a3-d25207d9b209-metrics-certs\") pod \"maas-api-e2e-inf-b-b960a2-7f955bb764-ftlgf\" (UID: \"1bd47a6e-dd0f-47ae-b0a3-d25207d9b209\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b960a2-7f955bb764-ftlgf"
Sep 10 19:22:34.412142 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:34.412113    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4k49t\" (UniqueName: \"kubernetes.io/projected/1bd47a6e-dd0f-47ae-b0a3-d25207d9b209-kube-api-access-4k49t\") pod \"maas-api-e2e-inf-b-b960a2-7f955bb764-ftlgf\" (UID: \"1bd47a6e-dd0f-47ae-b0a3-d25207d9b209\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b960a2-7f955bb764-ftlgf"
Sep 10 19:22:34.412311 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:22:34.412149    2546 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-b960a2-serving-cert: secret "maas-api-e2e-inf-b-b960a2-serving-cert" not found
Sep 10 19:22:34.412311 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:22:34.412220    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1bd47a6e-dd0f-47ae-b0a3-d25207d9b209-maas-api-tls podName:1bd47a6e-dd0f-47ae-b0a3-d25207d9b209 nodeName:}" failed. No retries permitted until 2026-09-10 19:22:34.912197041 +0000 UTC m=+718.141247058 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1bd47a6e-dd0f-47ae-b0a3-d25207d9b209-maas-api-tls") pod "maas-api-e2e-inf-b-b960a2-7f955bb764-ftlgf" (UID: "1bd47a6e-dd0f-47ae-b0a3-d25207d9b209") : secret "maas-api-e2e-inf-b-b960a2-serving-cert" not found
Sep 10 19:22:34.412743 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:34.412709    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bd47a6e-dd0f-47ae-b0a3-d25207d9b209-openshift-service-ca\") pod \"maas-api-e2e-inf-b-b960a2-7f955bb764-ftlgf\" (UID: \"1bd47a6e-dd0f-47ae-b0a3-d25207d9b209\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b960a2-7f955bb764-ftlgf"
Sep 10 19:22:34.414983 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:34.414954    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1bd47a6e-dd0f-47ae-b0a3-d25207d9b209-metrics-certs\") pod \"maas-api-e2e-inf-b-b960a2-7f955bb764-ftlgf\" (UID: \"1bd47a6e-dd0f-47ae-b0a3-d25207d9b209\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b960a2-7f955bb764-ftlgf"
Sep 10 19:22:34.422350 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:34.422317    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4k49t\" (UniqueName: \"kubernetes.io/projected/1bd47a6e-dd0f-47ae-b0a3-d25207d9b209-kube-api-access-4k49t\") pod \"maas-api-e2e-inf-b-b960a2-7f955bb764-ftlgf\" (UID: \"1bd47a6e-dd0f-47ae-b0a3-d25207d9b209\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b960a2-7f955bb764-ftlgf"
Sep 10 19:22:34.917032 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:34.916990    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1bd47a6e-dd0f-47ae-b0a3-d25207d9b209-maas-api-tls\") pod \"maas-api-e2e-inf-b-b960a2-7f955bb764-ftlgf\" (UID: \"1bd47a6e-dd0f-47ae-b0a3-d25207d9b209\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b960a2-7f955bb764-ftlgf"
Sep 10 19:22:34.919480 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:34.919456    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1bd47a6e-dd0f-47ae-b0a3-d25207d9b209-maas-api-tls\") pod \"maas-api-e2e-inf-b-b960a2-7f955bb764-ftlgf\" (UID: \"1bd47a6e-dd0f-47ae-b0a3-d25207d9b209\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b960a2-7f955bb764-ftlgf"
Sep 10 19:22:35.050765 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:35.050731    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b960a2-7f955bb764-ftlgf"
Sep 10 19:22:35.055005 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:35.054976    2546 patch_prober.go:28] interesting pod/e2e-autores-3c7a5e-openshift-default-94b6bdd6b-6mg46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Sep 10 19:22:35.055124 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:35.055037    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-3c7a5e-openshift-default-94b6bdd6b-6mg46" podUID="cbc91d4d-adea-4eac-a841-691897e6c095" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Sep 10 19:22:35.390367 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:35.390268    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b960a2-7f955bb764-ftlgf"]
Sep 10 19:22:35.397703 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:22:35.397673    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bd47a6e_dd0f_47ae_b0a3_d25207d9b209.slice/crio-88019d4cf3dbce72822273808aa202da180dadbe5bd81ec639eb168ba8020bf7 WatchSource:0}: Error finding container 88019d4cf3dbce72822273808aa202da180dadbe5bd81ec639eb168ba8020bf7: Status 404 returned error can't find the container with id 88019d4cf3dbce72822273808aa202da180dadbe5bd81ec639eb168ba8020bf7
Sep 10 19:22:35.421452 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:35.421413    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b960a2-7f955bb764-ftlgf" event={"ID":"1bd47a6e-dd0f-47ae-b0a3-d25207d9b209","Type":"ContainerStarted","Data":"88019d4cf3dbce72822273808aa202da180dadbe5bd81ec639eb168ba8020bf7"}
Sep 10 19:22:36.055476 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:36.055435    2546 patch_prober.go:28] interesting pod/e2e-autores-3c7a5e-openshift-default-94b6bdd6b-6mg46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Sep 10 19:22:36.055656 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:36.055494    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-3c7a5e-openshift-default-94b6bdd6b-6mg46" podUID="cbc91d4d-adea-4eac-a841-691897e6c095" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Sep 10 19:22:36.428225 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:36.428194    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b960a2-7f955bb764-ftlgf" event={"ID":"1bd47a6e-dd0f-47ae-b0a3-d25207d9b209","Type":"ContainerStarted","Data":"bb0f607bb948d989abfb6cf3226a9f521b7e92f29eae34c8f74b5815fc8d213c"}
Sep 10 19:22:36.428628 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:36.428351    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b960a2-7f955bb764-ftlgf"
Sep 10 19:22:36.448696 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:36.448648    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b960a2-7f955bb764-ftlgf" podStartSLOduration=1.936409955 podStartE2EDuration="2.448631535s" totalImagesPullingTime="512.22158ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:22:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:22:36.447918316 +0000 UTC m=+719.676968413" watchObservedRunningTime="2026-09-10 19:22:36.448631535 +0000 UTC m=+719.677681560"
Sep 10 19:22:37.059225 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:37.059191    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-3c7a5e-openshift-default-94b6bdd6b-6mg46"
Sep 10 19:22:37.059486 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:37.059463    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-3c7a5e-openshift-default-94b6bdd6b-6mg46"
Sep 10 19:22:37.060301 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:37.060279    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-3c7a5e-openshift-default-94b6bdd6b-6mg46"
Sep 10 19:22:38.524785 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:38.524748    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-88n8f"]
Sep 10 19:22:42.439231 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:42.439202    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b960a2-7f955bb764-ftlgf"
Sep 10 19:22:51.519372 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:22:51.519323    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-88n8f"]
Sep 10 19:23:07.917709 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:23:07.917675    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-a-fcaf1295-kserve-59cf48bfd6-dpt8r"]
Sep 10 19:23:07.921149 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:23:07.921132    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-fcaf1295-kserve-59cf48bfd6-dpt8r"
Sep 10 19:23:07.925505 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:23:07.925476    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-a-fcaf1295-kserve-self-signed-certs\""
Sep 10 19:23:07.939181 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:23:07.939152    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-fcaf1295-kserve-59cf48bfd6-dpt8r"]
Sep 10 19:23:08.024991 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:23:08.024958    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/758e803e-ce78-4b7e-ae13-7d07f9ad7598-tls-certs\") pod \"e2e-conflict-a-fcaf1295-kserve-59cf48bfd6-dpt8r\" (UID: \"758e803e-ce78-4b7e-ae13-7d07f9ad7598\") " pod="llm/e2e-conflict-a-fcaf1295-kserve-59cf48bfd6-dpt8r"
Sep 10 19:23:08.025175 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:23:08.025007    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/758e803e-ce78-4b7e-ae13-7d07f9ad7598-home\") pod \"e2e-conflict-a-fcaf1295-kserve-59cf48bfd6-dpt8r\" (UID: \"758e803e-ce78-4b7e-ae13-7d07f9ad7598\") " pod="llm/e2e-conflict-a-fcaf1295-kserve-59cf48bfd6-dpt8r"
Sep 10 19:23:08.025175 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:23:08.025072    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/758e803e-ce78-4b7e-ae13-7d07f9ad7598-tmp-dir\") pod \"e2e-conflict-a-fcaf1295-kserve-59cf48bfd6-dpt8r\" (UID: \"758e803e-ce78-4b7e-ae13-7d07f9ad7598\") " pod="llm/e2e-conflict-a-fcaf1295-kserve-59cf48bfd6-dpt8r"
Sep 10 19:23:08.025175 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:23:08.025166    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsvf5\" (UniqueName: \"kubernetes.io/projected/758e803e-ce78-4b7e-ae13-7d07f9ad7598-kube-api-access-qsvf5\") pod \"e2e-conflict-a-fcaf1295-kserve-59cf48bfd6-dpt8r\" (UID: \"758e803e-ce78-4b7e-ae13-7d07f9ad7598\") " pod="llm/e2e-conflict-a-fcaf1295-kserve-59cf48bfd6-dpt8r"
Sep 10 19:23:08.025396 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:23:08.025199    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/758e803e-ce78-4b7e-ae13-7d07f9ad7598-model-cache\") pod \"e2e-conflict-a-fcaf1295-kserve-59cf48bfd6-dpt8r\" (UID: \"758e803e-ce78-4b7e-ae13-7d07f9ad7598\") " pod="llm/e2e-conflict-a-fcaf1295-kserve-59cf48bfd6-dpt8r"
Sep 10 19:23:08.025396 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:23:08.025237    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/758e803e-ce78-4b7e-ae13-7d07f9ad7598-dshm\") pod \"e2e-conflict-a-fcaf1295-kserve-59cf48bfd6-dpt8r\" (UID: \"758e803e-ce78-4b7e-ae13-7d07f9ad7598\") " pod="llm/e2e-conflict-a-fcaf1295-kserve-59cf48bfd6-dpt8r"
Sep 10 19:23:08.126539 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:23:08.126499    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/758e803e-ce78-4b7e-ae13-7d07f9ad7598-dshm\") pod \"e2e-conflict-a-fcaf1295-kserve-59cf48bfd6-dpt8r\" (UID: \"758e803e-ce78-4b7e-ae13-7d07f9ad7598\") " pod="llm/e2e-conflict-a-fcaf1295-kserve-59cf48bfd6-dpt8r"
Sep 10 19:23:08.126731 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:23:08.126580    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/758e803e-ce78-4b7e-ae13-7d07f9ad7598-tls-certs\") pod \"e2e-conflict-a-fcaf1295-kserve-59cf48bfd6-dpt8r\" (UID: \"758e803e-ce78-4b7e-ae13-7d07f9ad7598\") " pod="llm/e2e-conflict-a-fcaf1295-kserve-59cf48bfd6-dpt8r"
Sep 10 19:23:08.126731 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:23:08.126632    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/758e803e-ce78-4b7e-ae13-7d07f9ad7598-home\") pod \"e2e-conflict-a-fcaf1295-kserve-59cf48bfd6-dpt8r\" (UID: \"758e803e-ce78-4b7e-ae13-7d07f9ad7598\") " pod="llm/e2e-conflict-a-fcaf1295-kserve-59cf48bfd6-dpt8r"
Sep 10 19:23:08.126731 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:23:08.126661    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/758e803e-ce78-4b7e-ae13-7d07f9ad7598-tmp-dir\") pod \"e2e-conflict-a-fcaf1295-kserve-59cf48bfd6-dpt8r\" (UID: \"758e803e-ce78-4b7e-ae13-7d07f9ad7598\") " pod="llm/e2e-conflict-a-fcaf1295-kserve-59cf48bfd6-dpt8r"
Sep 10 19:23:08.126731 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:23:08.126710    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsvf5\" (UniqueName: \"kubernetes.io/projected/758e803e-ce78-4b7e-ae13-7d07f9ad7598-kube-api-access-qsvf5\") pod \"e2e-conflict-a-fcaf1295-kserve-59cf48bfd6-dpt8r\" (UID: \"758e803e-ce78-4b7e-ae13-7d07f9ad7598\") " pod="llm/e2e-conflict-a-fcaf1295-kserve-59cf48bfd6-dpt8r"
Sep 10 19:23:08.126937 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:23:08.126742    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/758e803e-ce78-4b7e-ae13-7d07f9ad7598-model-cache\") pod \"e2e-conflict-a-fcaf1295-kserve-59cf48bfd6-dpt8r\" (UID: \"758e803e-ce78-4b7e-ae13-7d07f9ad7598\") " pod="llm/e2e-conflict-a-fcaf1295-kserve-59cf48bfd6-dpt8r"
Sep 10 19:23:08.127068 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:23:08.127041    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/758e803e-ce78-4b7e-ae13-7d07f9ad7598-home\") pod \"e2e-conflict-a-fcaf1295-kserve-59cf48bfd6-dpt8r\" (UID: \"758e803e-ce78-4b7e-ae13-7d07f9ad7598\") " pod="llm/e2e-conflict-a-fcaf1295-kserve-59cf48bfd6-dpt8r"
Sep 10 19:23:08.127144 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:23:08.127041    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/758e803e-ce78-4b7e-ae13-7d07f9ad7598-tmp-dir\") pod \"e2e-conflict-a-fcaf1295-kserve-59cf48bfd6-dpt8r\" (UID: \"758e803e-ce78-4b7e-ae13-7d07f9ad7598\") " pod="llm/e2e-conflict-a-fcaf1295-kserve-59cf48bfd6-dpt8r"
Sep 10 19:23:08.127144 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:23:08.127092    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/758e803e-ce78-4b7e-ae13-7d07f9ad7598-model-cache\") pod \"e2e-conflict-a-fcaf1295-kserve-59cf48bfd6-dpt8r\" (UID: \"758e803e-ce78-4b7e-ae13-7d07f9ad7598\") " pod="llm/e2e-conflict-a-fcaf1295-kserve-59cf48bfd6-dpt8r"
Sep 10 19:23:08.128854 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:23:08.128832    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/758e803e-ce78-4b7e-ae13-7d07f9ad7598-dshm\") pod \"e2e-conflict-a-fcaf1295-kserve-59cf48bfd6-dpt8r\" (UID: \"758e803e-ce78-4b7e-ae13-7d07f9ad7598\") " pod="llm/e2e-conflict-a-fcaf1295-kserve-59cf48bfd6-dpt8r"
Sep 10 19:23:08.129052 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:23:08.129034    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/758e803e-ce78-4b7e-ae13-7d07f9ad7598-tls-certs\") pod \"e2e-conflict-a-fcaf1295-kserve-59cf48bfd6-dpt8r\" (UID: \"758e803e-ce78-4b7e-ae13-7d07f9ad7598\") " pod="llm/e2e-conflict-a-fcaf1295-kserve-59cf48bfd6-dpt8r"
Sep 10 19:23:08.135057 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:23:08.135035    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsvf5\" (UniqueName: \"kubernetes.io/projected/758e803e-ce78-4b7e-ae13-7d07f9ad7598-kube-api-access-qsvf5\") pod \"e2e-conflict-a-fcaf1295-kserve-59cf48bfd6-dpt8r\" (UID: \"758e803e-ce78-4b7e-ae13-7d07f9ad7598\") " pod="llm/e2e-conflict-a-fcaf1295-kserve-59cf48bfd6-dpt8r"
Sep 10 19:23:08.229788 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:23:08.229696    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-fcaf1295-kserve-59cf48bfd6-dpt8r"
Sep 10 19:23:08.510736 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:23:08.510710    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-fcaf1295-kserve-59cf48bfd6-dpt8r"]
Sep 10 19:23:08.512829 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:23:08.512800    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod758e803e_ce78_4b7e_ae13_7d07f9ad7598.slice/crio-328801f63f256a9d8c36d60e99c2e2e853470dc78b16b2b0ec823819cc67cb20 WatchSource:0}: Error finding container 328801f63f256a9d8c36d60e99c2e2e853470dc78b16b2b0ec823819cc67cb20: Status 404 returned error can't find the container with id 328801f63f256a9d8c36d60e99c2e2e853470dc78b16b2b0ec823819cc67cb20
Sep 10 19:23:08.590318 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:23:08.590287    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-fcaf1295-kserve-59cf48bfd6-dpt8r" event={"ID":"758e803e-ce78-4b7e-ae13-7d07f9ad7598","Type":"ContainerStarted","Data":"9001934ace05d005d353eb36f2e6e27c9aa74dc73df26e2a9348062a1eb69afb"}
Sep 10 19:23:08.590481 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:23:08.590332    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-fcaf1295-kserve-59cf48bfd6-dpt8r" event={"ID":"758e803e-ce78-4b7e-ae13-7d07f9ad7598","Type":"ContainerStarted","Data":"328801f63f256a9d8c36d60e99c2e2e853470dc78b16b2b0ec823819cc67cb20"}
Sep 10 19:23:08.609930 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:23:08.609878    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-a-fcaf1295-kserve-59cf48bfd6-dpt8r" podStartSLOduration=1.60986048 podStartE2EDuration="1.60986048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:23:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:23:08.608530446 +0000 UTC m=+751.837580461" watchObservedRunningTime="2026-09-10 19:23:08.60986048 +0000 UTC m=+751.838910505"
Sep 10 19:23:08.943905 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:23:08.943868    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-88n8f"]
Sep 10 19:23:18.230168 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:23:18.230127    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-a-fcaf1295-kserve-59cf48bfd6-dpt8r"
Sep 10 19:23:18.230619 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:23:18.230189    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-a-fcaf1295-kserve-59cf48bfd6-dpt8r"
Sep 10 19:23:18.242737 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:23:18.242711    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-a-fcaf1295-kserve-59cf48bfd6-dpt8r"
Sep 10 19:23:18.648074 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:23:18.648037    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-a-fcaf1295-kserve-59cf48bfd6-dpt8r"
Sep 10 19:23:26.628153 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:23:26.628112    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-88n8f"]
Sep 10 19:23:44.122422 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:23:44.122376    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-88n8f"]
Sep 10 19:24:02.421011 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:24:02.420910    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-88n8f"]
Sep 10 19:24:17.636890 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:24:17.636853    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-88n8f"]
Sep 10 19:24:21.887644 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:24:21.887610    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-3c7a5e-59b6d88ff5-5xrmw"]
Sep 10 19:24:21.888048 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:24:21.887829    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-3c7a5e-59b6d88ff5-5xrmw" podUID="362d18ff-1ad6-4d25-8014-22b9e065d8fb" containerName="maas-api" containerID="cri-o://039297f75ee84b28b6f601eeaccf77651ffbf3abe863cf835c51b9278bba8e6d" gracePeriod=30
Sep 10 19:24:22.139450 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:24:22.139385    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-3c7a5e-59b6d88ff5-5xrmw"
Sep 10 19:24:22.220234 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:24:22.220203    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/362d18ff-1ad6-4d25-8014-22b9e065d8fb-metrics-certs\") pod \"362d18ff-1ad6-4d25-8014-22b9e065d8fb\" (UID: \"362d18ff-1ad6-4d25-8014-22b9e065d8fb\") "
Sep 10 19:24:22.220435 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:24:22.220252    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/362d18ff-1ad6-4d25-8014-22b9e065d8fb-openshift-service-ca\") pod \"362d18ff-1ad6-4d25-8014-22b9e065d8fb\" (UID: \"362d18ff-1ad6-4d25-8014-22b9e065d8fb\") "
Sep 10 19:24:22.220435 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:24:22.220276    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/362d18ff-1ad6-4d25-8014-22b9e065d8fb-maas-api-tls\") pod \"362d18ff-1ad6-4d25-8014-22b9e065d8fb\" (UID: \"362d18ff-1ad6-4d25-8014-22b9e065d8fb\") "
Sep 10 19:24:22.220435 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:24:22.220365    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jsrrn\" (UniqueName: \"kubernetes.io/projected/362d18ff-1ad6-4d25-8014-22b9e065d8fb-kube-api-access-jsrrn\") pod \"362d18ff-1ad6-4d25-8014-22b9e065d8fb\" (UID: \"362d18ff-1ad6-4d25-8014-22b9e065d8fb\") "
Sep 10 19:24:22.220695 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:24:22.220660    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/362d18ff-1ad6-4d25-8014-22b9e065d8fb-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "362d18ff-1ad6-4d25-8014-22b9e065d8fb" (UID: "362d18ff-1ad6-4d25-8014-22b9e065d8fb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 19:24:22.222661 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:24:22.222637    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/362d18ff-1ad6-4d25-8014-22b9e065d8fb-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "362d18ff-1ad6-4d25-8014-22b9e065d8fb" (UID: "362d18ff-1ad6-4d25-8014-22b9e065d8fb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 19:24:22.222765 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:24:22.222731    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/362d18ff-1ad6-4d25-8014-22b9e065d8fb-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "362d18ff-1ad6-4d25-8014-22b9e065d8fb" (UID: "362d18ff-1ad6-4d25-8014-22b9e065d8fb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 19:24:22.222828 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:24:22.222754    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/362d18ff-1ad6-4d25-8014-22b9e065d8fb-kube-api-access-jsrrn" (OuterVolumeSpecName: "kube-api-access-jsrrn") pod "362d18ff-1ad6-4d25-8014-22b9e065d8fb" (UID: "362d18ff-1ad6-4d25-8014-22b9e065d8fb"). InnerVolumeSpecName "kube-api-access-jsrrn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 19:24:22.321865 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:24:22.321823    2546 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/362d18ff-1ad6-4d25-8014-22b9e065d8fb-metrics-certs\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:24:22.321865 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:24:22.321856    2546 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/362d18ff-1ad6-4d25-8014-22b9e065d8fb-openshift-service-ca\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:24:22.321865 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:24:22.321866    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/362d18ff-1ad6-4d25-8014-22b9e065d8fb-maas-api-tls\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:24:22.321865 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:24:22.321875    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jsrrn\" (UniqueName: \"kubernetes.io/projected/362d18ff-1ad6-4d25-8014-22b9e065d8fb-kube-api-access-jsrrn\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:24:22.966739 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:24:22.966701    2546 generic.go:356] "Generic (PLEG): container finished" podID="362d18ff-1ad6-4d25-8014-22b9e065d8fb" containerID="039297f75ee84b28b6f601eeaccf77651ffbf3abe863cf835c51b9278bba8e6d" exitCode=0
Sep 10 19:24:22.967177 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:24:22.966755    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-3c7a5e-59b6d88ff5-5xrmw"
Sep 10 19:24:22.967177 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:24:22.966787    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-3c7a5e-59b6d88ff5-5xrmw" event={"ID":"362d18ff-1ad6-4d25-8014-22b9e065d8fb","Type":"ContainerDied","Data":"039297f75ee84b28b6f601eeaccf77651ffbf3abe863cf835c51b9278bba8e6d"}
Sep 10 19:24:22.967177 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:24:22.966880    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-3c7a5e-59b6d88ff5-5xrmw" event={"ID":"362d18ff-1ad6-4d25-8014-22b9e065d8fb","Type":"ContainerDied","Data":"faf2c5253734cf48ed610564d21cbb6c39e2111feae7aa507138b0743b31db7c"}
Sep 10 19:24:22.967177 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:24:22.966909    2546 scope.go:117] "RemoveContainer" containerID="039297f75ee84b28b6f601eeaccf77651ffbf3abe863cf835c51b9278bba8e6d"
Sep 10 19:24:22.976530 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:24:22.976515    2546 scope.go:117] "RemoveContainer" containerID="039297f75ee84b28b6f601eeaccf77651ffbf3abe863cf835c51b9278bba8e6d"
Sep 10 19:24:22.976780 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:24:22.976760    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"039297f75ee84b28b6f601eeaccf77651ffbf3abe863cf835c51b9278bba8e6d\": container with ID starting with 039297f75ee84b28b6f601eeaccf77651ffbf3abe863cf835c51b9278bba8e6d not found: ID does not exist" containerID="039297f75ee84b28b6f601eeaccf77651ffbf3abe863cf835c51b9278bba8e6d"
Sep 10 19:24:22.976835 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:24:22.976786    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"039297f75ee84b28b6f601eeaccf77651ffbf3abe863cf835c51b9278bba8e6d"} err="failed to get container status \"039297f75ee84b28b6f601eeaccf77651ffbf3abe863cf835c51b9278bba8e6d\": rpc error: code = NotFound desc = could not find container \"039297f75ee84b28b6f601eeaccf77651ffbf3abe863cf835c51b9278bba8e6d\": container with ID starting with 039297f75ee84b28b6f601eeaccf77651ffbf3abe863cf835c51b9278bba8e6d not found: ID does not exist"
Sep 10 19:24:23.004697 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:24:23.004675    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-3c7a5e-59b6d88ff5-5xrmw"]
Sep 10 19:24:23.018102 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:24:23.018081    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-3c7a5e-59b6d88ff5-5xrmw"]
Sep 10 19:24:23.281796 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:24:23.281724    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="362d18ff-1ad6-4d25-8014-22b9e065d8fb" path="/var/lib/kubelet/pods/362d18ff-1ad6-4d25-8014-22b9e065d8fb/volumes"
Sep 10 19:24:28.634536 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:24:28.634490    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-88n8f"]
Sep 10 19:24:34.018845 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:24:34.018810    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-6b562c/test-model-6b562c-kserve-56cd4564d9-b5vr6"]
Sep 10 19:24:34.019675 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:24:34.019419    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="362d18ff-1ad6-4d25-8014-22b9e065d8fb" containerName="maas-api"
Sep 10 19:24:34.019675 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:24:34.019439    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="362d18ff-1ad6-4d25-8014-22b9e065d8fb" containerName="maas-api"
Sep 10 19:24:34.019675 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:24:34.019536    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="362d18ff-1ad6-4d25-8014-22b9e065d8fb" containerName="maas-api"
Sep 10 19:24:34.028529 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:24:34.028467    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-6b562c/test-model-6b562c-kserve-56cd4564d9-b5vr6"
Sep 10 19:24:34.031692 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:24:34.031661    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-6b562c\"/\"openshift-service-ca.crt\""
Sep 10 19:24:34.032715 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:24:34.032688    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-6b562c\"/\"test-model-6b562c-kserve-self-signed-certs\""
Sep 10 19:24:34.032841 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:24:34.032737    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-6b562c\"/\"kube-root-ca.crt\""
Sep 10 19:24:34.033086 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:24:34.033070    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-6b562c\"/\"default-dockercfg-rgr6v\""
Sep 10 19:24:34.039694 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:24:34.039668    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-6b562c/test-model-6b562c-kserve-56cd4564d9-b5vr6"]
Sep 10 19:24:34.133687 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:24:34.133653    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24e6afb8-845c-4c89-8cea-5236276c5725-home\") pod \"test-model-6b562c-kserve-56cd4564d9-b5vr6\" (UID: \"24e6afb8-845c-4c89-8cea-5236276c5725\") " pod="ai-tenant-e2e-inf-a-6b562c/test-model-6b562c-kserve-56cd4564d9-b5vr6"
Sep 10 19:24:34.133687 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:24:34.133685    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24e6afb8-845c-4c89-8cea-5236276c5725-dshm\") pod \"test-model-6b562c-kserve-56cd4564d9-b5vr6\" (UID: \"24e6afb8-845c-4c89-8cea-5236276c5725\") " pod="ai-tenant-e2e-inf-a-6b562c/test-model-6b562c-kserve-56cd4564d9-b5vr6"
Sep 10 19:24:34.133917 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:24:34.133703    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45mm8\" (UniqueName: \"kubernetes.io/projected/24e6afb8-845c-4c89-8cea-5236276c5725-kube-api-access-45mm8\") pod \"test-model-6b562c-kserve-56cd4564d9-b5vr6\" (UID: \"24e6afb8-845c-4c89-8cea-5236276c5725\") " pod="ai-tenant-e2e-inf-a-6b562c/test-model-6b562c-kserve-56cd4564d9-b5vr6"
Sep 10 19:24:34.133917 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:24:34.133824    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24e6afb8-845c-4c89-8cea-5236276c5725-tls-certs\") pod \"test-model-6b562c-kserve-56cd4564d9-b5vr6\" (UID: \"24e6afb8-845c-4c89-8cea-5236276c5725\") " pod="ai-tenant-e2e-inf-a-6b562c/test-model-6b562c-kserve-56cd4564d9-b5vr6"
Sep 10 19:24:34.133917 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:24:34.133907    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24e6afb8-845c-4c89-8cea-5236276c5725-tmp-dir\") pod \"test-model-6b562c-kserve-56cd4564d9-b5vr6\" (UID: \"24e6afb8-845c-4c89-8cea-5236276c5725\") " pod="ai-tenant-e2e-inf-a-6b562c/test-model-6b562c-kserve-56cd4564d9-b5vr6"
Sep 10 19:24:34.134035 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:24:34.133935    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24e6afb8-845c-4c89-8cea-5236276c5725-model-cache\") pod \"test-model-6b562c-kserve-56cd4564d9-b5vr6\" (UID: \"24e6afb8-845c-4c89-8cea-5236276c5725\") " pod="ai-tenant-e2e-inf-a-6b562c/test-model-6b562c-kserve-56cd4564d9-b5vr6"
Sep 10 19:24:34.234738 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:24:34.234704    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24e6afb8-845c-4c89-8cea-5236276c5725-tmp-dir\") pod \"test-model-6b562c-kserve-56cd4564d9-b5vr6\" (UID: \"24e6afb8-845c-4c89-8cea-5236276c5725\") " pod="ai-tenant-e2e-inf-a-6b562c/test-model-6b562c-kserve-56cd4564d9-b5vr6"
Sep 10 19:24:34.234937 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:24:34.234746    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24e6afb8-845c-4c89-8cea-5236276c5725-model-cache\") pod \"test-model-6b562c-kserve-56cd4564d9-b5vr6\" (UID: \"24e6afb8-845c-4c89-8cea-5236276c5725\") " pod="ai-tenant-e2e-inf-a-6b562c/test-model-6b562c-kserve-56cd4564d9-b5vr6"
Sep 10 19:24:34.234937 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:24:34.234767    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24e6afb8-845c-4c89-8cea-5236276c5725-home\") pod \"test-model-6b562c-kserve-56cd4564d9-b5vr6\" (UID: \"24e6afb8-845c-4c89-8cea-5236276c5725\") " pod="ai-tenant-e2e-inf-a-6b562c/test-model-6b562c-kserve-56cd4564d9-b5vr6"
Sep 10 19:24:34.234937 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:24:34.234781    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24e6afb8-845c-4c89-8cea-5236276c5725-dshm\") pod \"test-model-6b562c-kserve-56cd4564d9-b5vr6\" (UID: \"24e6afb8-845c-4c89-8cea-5236276c5725\") " pod="ai-tenant-e2e-inf-a-6b562c/test-model-6b562c-kserve-56cd4564d9-b5vr6"
Sep 10 19:24:34.234937 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:24:34.234849    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45mm8\" (UniqueName: \"kubernetes.io/projected/24e6afb8-845c-4c89-8cea-5236276c5725-kube-api-access-45mm8\") pod \"test-model-6b562c-kserve-56cd4564d9-b5vr6\" (UID: \"24e6afb8-845c-4c89-8cea-5236276c5725\") " pod="ai-tenant-e2e-inf-a-6b562c/test-model-6b562c-kserve-56cd4564d9-b5vr6"
Sep 10 19:24:34.235142 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:24:34.234965    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24e6afb8-845c-4c89-8cea-5236276c5725-tls-certs\") pod \"test-model-6b562c-kserve-56cd4564d9-b5vr6\" (UID: \"24e6afb8-845c-4c89-8cea-5236276c5725\") " pod="ai-tenant-e2e-inf-a-6b562c/test-model-6b562c-kserve-56cd4564d9-b5vr6"
Sep 10 19:24:34.235211 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:24:34.235192    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24e6afb8-845c-4c89-8cea-5236276c5725-model-cache\") pod \"test-model-6b562c-kserve-56cd4564d9-b5vr6\" (UID: \"24e6afb8-845c-4c89-8cea-5236276c5725\") " pod="ai-tenant-e2e-inf-a-6b562c/test-model-6b562c-kserve-56cd4564d9-b5vr6"
Sep 10 19:24:34.235281 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:24:34.235261    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24e6afb8-845c-4c89-8cea-5236276c5725-tmp-dir\") pod \"test-model-6b562c-kserve-56cd4564d9-b5vr6\" (UID: \"24e6afb8-845c-4c89-8cea-5236276c5725\") " pod="ai-tenant-e2e-inf-a-6b562c/test-model-6b562c-kserve-56cd4564d9-b5vr6"
Sep 10 19:24:34.235364 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:24:34.235296    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24e6afb8-845c-4c89-8cea-5236276c5725-home\") pod \"test-model-6b562c-kserve-56cd4564d9-b5vr6\" (UID: \"24e6afb8-845c-4c89-8cea-5236276c5725\") " pod="ai-tenant-e2e-inf-a-6b562c/test-model-6b562c-kserve-56cd4564d9-b5vr6"
Sep 10 19:24:34.237143 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:24:34.237120    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24e6afb8-845c-4c89-8cea-5236276c5725-dshm\") pod \"test-model-6b562c-kserve-56cd4564d9-b5vr6\" (UID: \"24e6afb8-845c-4c89-8cea-5236276c5725\") " pod="ai-tenant-e2e-inf-a-6b562c/test-model-6b562c-kserve-56cd4564d9-b5vr6"
Sep 10 19:24:34.237465 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:24:34.237446    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24e6afb8-845c-4c89-8cea-5236276c5725-tls-certs\") pod \"test-model-6b562c-kserve-56cd4564d9-b5vr6\" (UID: \"24e6afb8-845c-4c89-8cea-5236276c5725\") " pod="ai-tenant-e2e-inf-a-6b562c/test-model-6b562c-kserve-56cd4564d9-b5vr6"
Sep 10 19:24:34.243223 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:24:34.243202    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-45mm8\" (UniqueName: \"kubernetes.io/projected/24e6afb8-845c-4c89-8cea-5236276c5725-kube-api-access-45mm8\") pod \"test-model-6b562c-kserve-56cd4564d9-b5vr6\" (UID: \"24e6afb8-845c-4c89-8cea-5236276c5725\") " pod="ai-tenant-e2e-inf-a-6b562c/test-model-6b562c-kserve-56cd4564d9-b5vr6"
Sep 10 19:24:34.340263 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:24:34.340176    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-6b562c/test-model-6b562c-kserve-56cd4564d9-b5vr6"
Sep 10 19:24:34.616831 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:24:34.616806    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-6b562c/test-model-6b562c-kserve-56cd4564d9-b5vr6"]
Sep 10 19:24:34.618207 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:24:34.618180    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod24e6afb8_845c_4c89_8cea_5236276c5725.slice/crio-f283e0eee5ada272fb2f18c2ed9b8727fa7fd671d86c8b4d6321622e114a620b WatchSource:0}: Error finding container f283e0eee5ada272fb2f18c2ed9b8727fa7fd671d86c8b4d6321622e114a620b: Status 404 returned error can't find the container with id f283e0eee5ada272fb2f18c2ed9b8727fa7fd671d86c8b4d6321622e114a620b
Sep 10 19:24:35.040922 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:24:35.040837    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-6b562c/test-model-6b562c-kserve-56cd4564d9-b5vr6" event={"ID":"24e6afb8-845c-4c89-8cea-5236276c5725","Type":"ContainerStarted","Data":"174c86bdb0c6085e487d1fd4d130f1a19054be8e43b2c93ae34cd3d5cd8656a1"}
Sep 10 19:24:35.040922 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:24:35.040879    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-6b562c/test-model-6b562c-kserve-56cd4564d9-b5vr6" event={"ID":"24e6afb8-845c-4c89-8cea-5236276c5725","Type":"ContainerStarted","Data":"f283e0eee5ada272fb2f18c2ed9b8727fa7fd671d86c8b4d6321622e114a620b"}
Sep 10 19:24:35.062656 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:24:35.062596    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-6b562c/test-model-6b562c-kserve-56cd4564d9-b5vr6" podStartSLOduration=2.062574825 podStartE2EDuration="2.062574825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:24:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:24:35.061213994 +0000 UTC m=+838.290264018" watchObservedRunningTime="2026-09-10 19:24:35.062574825 +0000 UTC m=+838.291624851"
Sep 10 19:24:39.355882 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:24:39.355850    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-88n8f"]
Sep 10 19:24:44.340548 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:24:44.340506    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-6b562c/test-model-6b562c-kserve-56cd4564d9-b5vr6"
Sep 10 19:24:44.340978 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:24:44.340659    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-6b562c/test-model-6b562c-kserve-56cd4564d9-b5vr6"
Sep 10 19:24:44.353416 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:24:44.353382    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-6b562c/test-model-6b562c-kserve-56cd4564d9-b5vr6"
Sep 10 19:24:45.104853 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:24:45.104824    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-6b562c/test-model-6b562c-kserve-56cd4564d9-b5vr6"
Sep 10 19:24:51.734878 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:24:51.734841    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-88n8f"]
Sep 10 19:25:08.423671 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:25:08.423633    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-88n8f"]
Sep 10 19:25:11.221040 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:25:11.221004    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-3c7a5e-openshift-default-94b6bdd6b-6mg46"]
Sep 10 19:25:11.221494 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:25:11.221255    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-3c7a5e-openshift-default-94b6bdd6b-6mg46" podUID="cbc91d4d-adea-4eac-a841-691897e6c095" containerName="istio-proxy" containerID="cri-o://57046ff5721443c014639bfd2129f083df4ba010a7958c631b2795da77fbb817" gracePeriod=30
Sep 10 19:25:16.473061 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:25:16.473033    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-3c7a5e-openshift-default-94b6bdd6b-6mg46"
Sep 10 19:25:16.535393 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:25:16.535276    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbc91d4d-adea-4eac-a841-691897e6c095-istio-token\") pod \"cbc91d4d-adea-4eac-a841-691897e6c095\" (UID: \"cbc91d4d-adea-4eac-a841-691897e6c095\") "
Sep 10 19:25:16.535393 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:25:16.535331    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbc91d4d-adea-4eac-a841-691897e6c095-istio-podinfo\") pod \"cbc91d4d-adea-4eac-a841-691897e6c095\" (UID: \"cbc91d4d-adea-4eac-a841-691897e6c095\") "
Sep 10 19:25:16.535393 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:25:16.535375    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbc91d4d-adea-4eac-a841-691897e6c095-workload-socket\") pod \"cbc91d4d-adea-4eac-a841-691897e6c095\" (UID: \"cbc91d4d-adea-4eac-a841-691897e6c095\") "
Sep 10 19:25:16.535639 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:25:16.535477    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vdt5c\" (UniqueName: \"kubernetes.io/projected/cbc91d4d-adea-4eac-a841-691897e6c095-kube-api-access-vdt5c\") pod \"cbc91d4d-adea-4eac-a841-691897e6c095\" (UID: \"cbc91d4d-adea-4eac-a841-691897e6c095\") "
Sep 10 19:25:16.535639 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:25:16.535535    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbc91d4d-adea-4eac-a841-691897e6c095-credential-socket\") pod \"cbc91d4d-adea-4eac-a841-691897e6c095\" (UID: \"cbc91d4d-adea-4eac-a841-691897e6c095\") "
Sep 10 19:25:16.535639 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:25:16.535597    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbc91d4d-adea-4eac-a841-691897e6c095-istio-data\") pod \"cbc91d4d-adea-4eac-a841-691897e6c095\" (UID: \"cbc91d4d-adea-4eac-a841-691897e6c095\") "
Sep 10 19:25:16.535639 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:25:16.535623    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbc91d4d-adea-4eac-a841-691897e6c095-istio-envoy\") pod \"cbc91d4d-adea-4eac-a841-691897e6c095\" (UID: \"cbc91d4d-adea-4eac-a841-691897e6c095\") "
Sep 10 19:25:16.535639 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:25:16.535621    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbc91d4d-adea-4eac-a841-691897e6c095-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "cbc91d4d-adea-4eac-a841-691897e6c095" (UID: "cbc91d4d-adea-4eac-a841-691897e6c095"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:25:16.535902 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:25:16.535702    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbc91d4d-adea-4eac-a841-691897e6c095-workload-certs\") pod \"cbc91d4d-adea-4eac-a841-691897e6c095\" (UID: \"cbc91d4d-adea-4eac-a841-691897e6c095\") "
Sep 10 19:25:16.535902 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:25:16.535740    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbc91d4d-adea-4eac-a841-691897e6c095-istiod-ca-cert\") pod \"cbc91d4d-adea-4eac-a841-691897e6c095\" (UID: \"cbc91d4d-adea-4eac-a841-691897e6c095\") "
Sep 10 19:25:16.535902 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:25:16.535792    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbc91d4d-adea-4eac-a841-691897e6c095-istio-data" (OuterVolumeSpecName: "istio-data") pod "cbc91d4d-adea-4eac-a841-691897e6c095" (UID: "cbc91d4d-adea-4eac-a841-691897e6c095"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:25:16.535902 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:25:16.535819    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbc91d4d-adea-4eac-a841-691897e6c095-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "cbc91d4d-adea-4eac-a841-691897e6c095" (UID: "cbc91d4d-adea-4eac-a841-691897e6c095"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:25:16.536114 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:25:16.536002    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbc91d4d-adea-4eac-a841-691897e6c095-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "cbc91d4d-adea-4eac-a841-691897e6c095" (UID: "cbc91d4d-adea-4eac-a841-691897e6c095"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:25:16.536173 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:25:16.536116    2546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbc91d4d-adea-4eac-a841-691897e6c095-workload-socket\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:25:16.536173 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:25:16.536138    2546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbc91d4d-adea-4eac-a841-691897e6c095-credential-socket\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:25:16.536173 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:25:16.536151    2546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbc91d4d-adea-4eac-a841-691897e6c095-istio-data\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:25:16.536173 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:25:16.536163    2546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbc91d4d-adea-4eac-a841-691897e6c095-workload-certs\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:25:16.536401 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:25:16.536198    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbc91d4d-adea-4eac-a841-691897e6c095-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "cbc91d4d-adea-4eac-a841-691897e6c095" (UID: "cbc91d4d-adea-4eac-a841-691897e6c095"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 19:25:16.537791 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:25:16.537763    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cbc91d4d-adea-4eac-a841-691897e6c095-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "cbc91d4d-adea-4eac-a841-691897e6c095" (UID: "cbc91d4d-adea-4eac-a841-691897e6c095"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 19:25:16.537791 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:25:16.537777    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbc91d4d-adea-4eac-a841-691897e6c095-kube-api-access-vdt5c" (OuterVolumeSpecName: "kube-api-access-vdt5c") pod "cbc91d4d-adea-4eac-a841-691897e6c095" (UID: "cbc91d4d-adea-4eac-a841-691897e6c095"). InnerVolumeSpecName "kube-api-access-vdt5c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 19:25:16.537938 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:25:16.537768    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbc91d4d-adea-4eac-a841-691897e6c095-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "cbc91d4d-adea-4eac-a841-691897e6c095" (UID: "cbc91d4d-adea-4eac-a841-691897e6c095"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:25:16.537938 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:25:16.537800    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbc91d4d-adea-4eac-a841-691897e6c095-istio-token" (OuterVolumeSpecName: "istio-token") pod "cbc91d4d-adea-4eac-a841-691897e6c095" (UID: "cbc91d4d-adea-4eac-a841-691897e6c095"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 19:25:16.637175 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:25:16.637140    2546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbc91d4d-adea-4eac-a841-691897e6c095-istio-envoy\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:25:16.637175 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:25:16.637171    2546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbc91d4d-adea-4eac-a841-691897e6c095-istiod-ca-cert\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:25:16.637175 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:25:16.637180    2546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbc91d4d-adea-4eac-a841-691897e6c095-istio-token\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:25:16.637175 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:25:16.637188    2546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbc91d4d-adea-4eac-a841-691897e6c095-istio-podinfo\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:25:16.637540 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:25:16.637197    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdt5c\" (UniqueName: \"kubernetes.io/projected/cbc91d4d-adea-4eac-a841-691897e6c095-kube-api-access-vdt5c\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:25:17.252426 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:25:17.252387    2546 generic.go:356] "Generic (PLEG): container finished" podID="cbc91d4d-adea-4eac-a841-691897e6c095" containerID="57046ff5721443c014639bfd2129f083df4ba010a7958c631b2795da77fbb817" exitCode=0
Sep 10 19:25:17.252608 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:25:17.252440    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-3c7a5e-openshift-default-94b6bdd6b-6mg46"
Sep 10 19:25:17.252608 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:25:17.252475    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-3c7a5e-openshift-default-94b6bdd6b-6mg46" event={"ID":"cbc91d4d-adea-4eac-a841-691897e6c095","Type":"ContainerDied","Data":"57046ff5721443c014639bfd2129f083df4ba010a7958c631b2795da77fbb817"}
Sep 10 19:25:17.252608 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:25:17.252539    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-3c7a5e-openshift-default-94b6bdd6b-6mg46" event={"ID":"cbc91d4d-adea-4eac-a841-691897e6c095","Type":"ContainerDied","Data":"26c6f242be4a50db84646155aa475d5b618cac93b73e9fdd824d659933451a48"}
Sep 10 19:25:17.252608 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:25:17.252564    2546 scope.go:117] "RemoveContainer" containerID="57046ff5721443c014639bfd2129f083df4ba010a7958c631b2795da77fbb817"
Sep 10 19:25:17.264305 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:25:17.264284    2546 scope.go:117] "RemoveContainer" containerID="19cb18b95c32205d955306781edf3ea1b47b5ff57fec709efb7edde8492cdf17"
Sep 10 19:25:17.273273 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:25:17.273242    2546 scope.go:117] "RemoveContainer" containerID="57046ff5721443c014639bfd2129f083df4ba010a7958c631b2795da77fbb817"
Sep 10 19:25:17.273607 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:25:17.273585    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57046ff5721443c014639bfd2129f083df4ba010a7958c631b2795da77fbb817\": container with ID starting with 57046ff5721443c014639bfd2129f083df4ba010a7958c631b2795da77fbb817 not found: ID does not exist" containerID="57046ff5721443c014639bfd2129f083df4ba010a7958c631b2795da77fbb817"
Sep 10 19:25:17.273675 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:25:17.273618    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57046ff5721443c014639bfd2129f083df4ba010a7958c631b2795da77fbb817"} err="failed to get container status \"57046ff5721443c014639bfd2129f083df4ba010a7958c631b2795da77fbb817\": rpc error: code = NotFound desc = could not find container \"57046ff5721443c014639bfd2129f083df4ba010a7958c631b2795da77fbb817\": container with ID starting with 57046ff5721443c014639bfd2129f083df4ba010a7958c631b2795da77fbb817 not found: ID does not exist"
Sep 10 19:25:17.273675 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:25:17.273638    2546 scope.go:117] "RemoveContainer" containerID="19cb18b95c32205d955306781edf3ea1b47b5ff57fec709efb7edde8492cdf17"
Sep 10 19:25:17.273929 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:25:17.273913    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19cb18b95c32205d955306781edf3ea1b47b5ff57fec709efb7edde8492cdf17\": container with ID starting with 19cb18b95c32205d955306781edf3ea1b47b5ff57fec709efb7edde8492cdf17 not found: ID does not exist" containerID="19cb18b95c32205d955306781edf3ea1b47b5ff57fec709efb7edde8492cdf17"
Sep 10 19:25:17.273984 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:25:17.273933    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19cb18b95c32205d955306781edf3ea1b47b5ff57fec709efb7edde8492cdf17"} err="failed to get container status \"19cb18b95c32205d955306781edf3ea1b47b5ff57fec709efb7edde8492cdf17\": rpc error: code = NotFound desc = could not find container \"19cb18b95c32205d955306781edf3ea1b47b5ff57fec709efb7edde8492cdf17\": container with ID starting with 19cb18b95c32205d955306781edf3ea1b47b5ff57fec709efb7edde8492cdf17 not found: ID does not exist"
Sep 10 19:25:17.288325 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:25:17.288298    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-3c7a5e-openshift-default-94b6bdd6b-6mg46"]
Sep 10 19:25:17.298735 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:25:17.298705    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-3c7a5e-openshift-default-94b6bdd6b-6mg46"]
Sep 10 19:25:19.281001 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:25:19.280954    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cbc91d4d-adea-4eac-a841-691897e6c095" path="/var/lib/kubelet/pods/cbc91d4d-adea-4eac-a841-691897e6c095/volumes"
Sep 10 19:25:23.231118 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:25:23.231079    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-88n8f"]
Sep 10 19:25:26.525310 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:25:26.525274    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-fcaf1295-kserve-59cf48bfd6-dpt8r"]
Sep 10 19:25:38.620103 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:25:38.620068    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-88n8f"]
Sep 10 19:25:41.546231 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:25:41.546188    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-a-fcaf1295-kserve-59cf48bfd6-dpt8r" podUID="758e803e-ce78-4b7e-ae13-7d07f9ad7598" containerName="main" containerID="cri-o://9001934ace05d005d353eb36f2e6e27c9aa74dc73df26e2a9348062a1eb69afb" gracePeriod=45
Sep 10 19:25:41.807657 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:25:41.807596    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-fcaf1295-kserve-59cf48bfd6-dpt8r"
Sep 10 19:25:41.866729 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:25:41.866699    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/758e803e-ce78-4b7e-ae13-7d07f9ad7598-model-cache\") pod \"758e803e-ce78-4b7e-ae13-7d07f9ad7598\" (UID: \"758e803e-ce78-4b7e-ae13-7d07f9ad7598\") "
Sep 10 19:25:41.866922 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:25:41.866767    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qsvf5\" (UniqueName: \"kubernetes.io/projected/758e803e-ce78-4b7e-ae13-7d07f9ad7598-kube-api-access-qsvf5\") pod \"758e803e-ce78-4b7e-ae13-7d07f9ad7598\" (UID: \"758e803e-ce78-4b7e-ae13-7d07f9ad7598\") "
Sep 10 19:25:41.866922 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:25:41.866815    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/758e803e-ce78-4b7e-ae13-7d07f9ad7598-tmp-dir\") pod \"758e803e-ce78-4b7e-ae13-7d07f9ad7598\" (UID: \"758e803e-ce78-4b7e-ae13-7d07f9ad7598\") "
Sep 10 19:25:41.866922 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:25:41.866849    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/758e803e-ce78-4b7e-ae13-7d07f9ad7598-tls-certs\") pod \"758e803e-ce78-4b7e-ae13-7d07f9ad7598\" (UID: \"758e803e-ce78-4b7e-ae13-7d07f9ad7598\") "
Sep 10 19:25:41.866922 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:25:41.866872    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/758e803e-ce78-4b7e-ae13-7d07f9ad7598-dshm\") pod \"758e803e-ce78-4b7e-ae13-7d07f9ad7598\" (UID: \"758e803e-ce78-4b7e-ae13-7d07f9ad7598\") "
Sep 10 19:25:41.866922 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:25:41.866897    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/758e803e-ce78-4b7e-ae13-7d07f9ad7598-home\") pod \"758e803e-ce78-4b7e-ae13-7d07f9ad7598\" (UID: \"758e803e-ce78-4b7e-ae13-7d07f9ad7598\") "
Sep 10 19:25:41.867182 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:25:41.866980    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/758e803e-ce78-4b7e-ae13-7d07f9ad7598-model-cache" (OuterVolumeSpecName: "model-cache") pod "758e803e-ce78-4b7e-ae13-7d07f9ad7598" (UID: "758e803e-ce78-4b7e-ae13-7d07f9ad7598"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:25:41.867182 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:25:41.867069    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/758e803e-ce78-4b7e-ae13-7d07f9ad7598-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "758e803e-ce78-4b7e-ae13-7d07f9ad7598" (UID: "758e803e-ce78-4b7e-ae13-7d07f9ad7598"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:25:41.867182 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:25:41.867151    2546 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/758e803e-ce78-4b7e-ae13-7d07f9ad7598-model-cache\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:25:41.867182 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:25:41.867166    2546 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/758e803e-ce78-4b7e-ae13-7d07f9ad7598-tmp-dir\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:25:41.867468 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:25:41.867430    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/758e803e-ce78-4b7e-ae13-7d07f9ad7598-home" (OuterVolumeSpecName: "home") pod "758e803e-ce78-4b7e-ae13-7d07f9ad7598" (UID: "758e803e-ce78-4b7e-ae13-7d07f9ad7598"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:25:41.869165 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:25:41.869138    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/758e803e-ce78-4b7e-ae13-7d07f9ad7598-kube-api-access-qsvf5" (OuterVolumeSpecName: "kube-api-access-qsvf5") pod "758e803e-ce78-4b7e-ae13-7d07f9ad7598" (UID: "758e803e-ce78-4b7e-ae13-7d07f9ad7598"). InnerVolumeSpecName "kube-api-access-qsvf5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 19:25:41.869279 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:25:41.869165    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/758e803e-ce78-4b7e-ae13-7d07f9ad7598-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "758e803e-ce78-4b7e-ae13-7d07f9ad7598" (UID: "758e803e-ce78-4b7e-ae13-7d07f9ad7598"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 19:25:41.869279 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:25:41.869238    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/758e803e-ce78-4b7e-ae13-7d07f9ad7598-dshm" (OuterVolumeSpecName: "dshm") pod "758e803e-ce78-4b7e-ae13-7d07f9ad7598" (UID: "758e803e-ce78-4b7e-ae13-7d07f9ad7598"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:25:41.967954 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:25:41.967911    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qsvf5\" (UniqueName: \"kubernetes.io/projected/758e803e-ce78-4b7e-ae13-7d07f9ad7598-kube-api-access-qsvf5\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:25:41.967954 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:25:41.967948    2546 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/758e803e-ce78-4b7e-ae13-7d07f9ad7598-tls-certs\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:25:41.967954 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:25:41.967961    2546 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/758e803e-ce78-4b7e-ae13-7d07f9ad7598-dshm\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:25:41.968198 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:25:41.967972    2546 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/758e803e-ce78-4b7e-ae13-7d07f9ad7598-home\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:25:42.377167 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:25:42.377126    2546 generic.go:356] "Generic (PLEG): container finished" podID="758e803e-ce78-4b7e-ae13-7d07f9ad7598" containerID="9001934ace05d005d353eb36f2e6e27c9aa74dc73df26e2a9348062a1eb69afb" exitCode=0
Sep 10 19:25:42.377360 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:25:42.377174    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-fcaf1295-kserve-59cf48bfd6-dpt8r"
Sep 10 19:25:42.377360 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:25:42.377212    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-fcaf1295-kserve-59cf48bfd6-dpt8r" event={"ID":"758e803e-ce78-4b7e-ae13-7d07f9ad7598","Type":"ContainerDied","Data":"9001934ace05d005d353eb36f2e6e27c9aa74dc73df26e2a9348062a1eb69afb"}
Sep 10 19:25:42.377360 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:25:42.377256    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-fcaf1295-kserve-59cf48bfd6-dpt8r" event={"ID":"758e803e-ce78-4b7e-ae13-7d07f9ad7598","Type":"ContainerDied","Data":"328801f63f256a9d8c36d60e99c2e2e853470dc78b16b2b0ec823819cc67cb20"}
Sep 10 19:25:42.377360 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:25:42.377276    2546 scope.go:117] "RemoveContainer" containerID="9001934ace05d005d353eb36f2e6e27c9aa74dc73df26e2a9348062a1eb69afb"
Sep 10 19:25:42.386445 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:25:42.386415    2546 scope.go:117] "RemoveContainer" containerID="9001934ace05d005d353eb36f2e6e27c9aa74dc73df26e2a9348062a1eb69afb"
Sep 10 19:25:42.386710 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:25:42.386691    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9001934ace05d005d353eb36f2e6e27c9aa74dc73df26e2a9348062a1eb69afb\": container with ID starting with 9001934ace05d005d353eb36f2e6e27c9aa74dc73df26e2a9348062a1eb69afb not found: ID does not exist" containerID="9001934ace05d005d353eb36f2e6e27c9aa74dc73df26e2a9348062a1eb69afb"
Sep 10 19:25:42.386772 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:25:42.386723    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9001934ace05d005d353eb36f2e6e27c9aa74dc73df26e2a9348062a1eb69afb"} err="failed to get container status \"9001934ace05d005d353eb36f2e6e27c9aa74dc73df26e2a9348062a1eb69afb\": rpc error: code = NotFound desc = could not find container \"9001934ace05d005d353eb36f2e6e27c9aa74dc73df26e2a9348062a1eb69afb\": container with ID starting with 9001934ace05d005d353eb36f2e6e27c9aa74dc73df26e2a9348062a1eb69afb not found: ID does not exist"
Sep 10 19:25:42.401022 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:25:42.400995    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-fcaf1295-kserve-59cf48bfd6-dpt8r"]
Sep 10 19:25:42.411421 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:25:42.409581    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-a-fcaf1295-kserve-59cf48bfd6-dpt8r"]
Sep 10 19:25:43.281520 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:25:43.281483    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="758e803e-ce78-4b7e-ae13-7d07f9ad7598" path="/var/lib/kubelet/pods/758e803e-ce78-4b7e-ae13-7d07f9ad7598/volumes"
Sep 10 19:25:52.345567 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:25:52.345530    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-88n8f"]
Sep 10 19:26:05.016868 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:05.016832    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-717161/explicit-model-717161-kserve-777c7cddf9-dggmj"]
Sep 10 19:26:05.017356 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:05.017234    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="758e803e-ce78-4b7e-ae13-7d07f9ad7598" containerName="main"
Sep 10 19:26:05.017356 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:05.017247    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="758e803e-ce78-4b7e-ae13-7d07f9ad7598" containerName="main"
Sep 10 19:26:05.017356 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:05.017258    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbc91d4d-adea-4eac-a841-691897e6c095" containerName="istio-proxy"
Sep 10 19:26:05.017356 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:05.017264    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbc91d4d-adea-4eac-a841-691897e6c095" containerName="istio-proxy"
Sep 10 19:26:05.017356 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:05.017277    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbc91d4d-adea-4eac-a841-691897e6c095" containerName="istio-proxy"
Sep 10 19:26:05.017356 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:05.017285    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbc91d4d-adea-4eac-a841-691897e6c095" containerName="istio-proxy"
Sep 10 19:26:05.017552 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:05.017387    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="758e803e-ce78-4b7e-ae13-7d07f9ad7598" containerName="main"
Sep 10 19:26:05.017552 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:05.017399    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="cbc91d4d-adea-4eac-a841-691897e6c095" containerName="istio-proxy"
Sep 10 19:26:05.017552 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:05.017406    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="cbc91d4d-adea-4eac-a841-691897e6c095" containerName="istio-proxy"
Sep 10 19:26:05.019383 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:05.019334    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-717161/explicit-model-717161-kserve-777c7cddf9-dggmj"
Sep 10 19:26:05.021958 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:05.021923    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-717161\"/\"openshift-service-ca.crt\""
Sep 10 19:26:05.021958 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:05.021923    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-717161\"/\"explicit-model-717161-kserve-self-signed-certs\""
Sep 10 19:26:05.022961 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:05.022934    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-717161\"/\"kube-root-ca.crt\""
Sep 10 19:26:05.023069 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:05.022986    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-717161\"/\"default-dockercfg-4tr6t\""
Sep 10 19:26:05.031055 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:05.031026    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-717161/explicit-model-717161-kserve-777c7cddf9-dggmj"]
Sep 10 19:26:05.180172 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:05.180133    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39788d92-6b78-413b-b9f4-75fc646c00a3-tmp-dir\") pod \"explicit-model-717161-kserve-777c7cddf9-dggmj\" (UID: \"39788d92-6b78-413b-b9f4-75fc646c00a3\") " pod="ai-tenant-e2e-explicit-717161/explicit-model-717161-kserve-777c7cddf9-dggmj"
Sep 10 19:26:05.180385 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:05.180182    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jkxg\" (UniqueName: \"kubernetes.io/projected/39788d92-6b78-413b-b9f4-75fc646c00a3-kube-api-access-6jkxg\") pod \"explicit-model-717161-kserve-777c7cddf9-dggmj\" (UID: \"39788d92-6b78-413b-b9f4-75fc646c00a3\") " pod="ai-tenant-e2e-explicit-717161/explicit-model-717161-kserve-777c7cddf9-dggmj"
Sep 10 19:26:05.180385 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:05.180229    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39788d92-6b78-413b-b9f4-75fc646c00a3-home\") pod \"explicit-model-717161-kserve-777c7cddf9-dggmj\" (UID: \"39788d92-6b78-413b-b9f4-75fc646c00a3\") " pod="ai-tenant-e2e-explicit-717161/explicit-model-717161-kserve-777c7cddf9-dggmj"
Sep 10 19:26:05.180385 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:05.180328    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39788d92-6b78-413b-b9f4-75fc646c00a3-tls-certs\") pod \"explicit-model-717161-kserve-777c7cddf9-dggmj\" (UID: \"39788d92-6b78-413b-b9f4-75fc646c00a3\") " pod="ai-tenant-e2e-explicit-717161/explicit-model-717161-kserve-777c7cddf9-dggmj"
Sep 10 19:26:05.180537 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:05.180402    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39788d92-6b78-413b-b9f4-75fc646c00a3-dshm\") pod \"explicit-model-717161-kserve-777c7cddf9-dggmj\" (UID: \"39788d92-6b78-413b-b9f4-75fc646c00a3\") " pod="ai-tenant-e2e-explicit-717161/explicit-model-717161-kserve-777c7cddf9-dggmj"
Sep 10 19:26:05.180537 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:05.180460    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39788d92-6b78-413b-b9f4-75fc646c00a3-model-cache\") pod \"explicit-model-717161-kserve-777c7cddf9-dggmj\" (UID: \"39788d92-6b78-413b-b9f4-75fc646c00a3\") " pod="ai-tenant-e2e-explicit-717161/explicit-model-717161-kserve-777c7cddf9-dggmj"
Sep 10 19:26:05.281906 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:05.281827    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39788d92-6b78-413b-b9f4-75fc646c00a3-model-cache\") pod \"explicit-model-717161-kserve-777c7cddf9-dggmj\" (UID: \"39788d92-6b78-413b-b9f4-75fc646c00a3\") " pod="ai-tenant-e2e-explicit-717161/explicit-model-717161-kserve-777c7cddf9-dggmj"
Sep 10 19:26:05.281906 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:05.281884    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39788d92-6b78-413b-b9f4-75fc646c00a3-tmp-dir\") pod \"explicit-model-717161-kserve-777c7cddf9-dggmj\" (UID: \"39788d92-6b78-413b-b9f4-75fc646c00a3\") " pod="ai-tenant-e2e-explicit-717161/explicit-model-717161-kserve-777c7cddf9-dggmj"
Sep 10 19:26:05.281906 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:05.281907    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jkxg\" (UniqueName: \"kubernetes.io/projected/39788d92-6b78-413b-b9f4-75fc646c00a3-kube-api-access-6jkxg\") pod \"explicit-model-717161-kserve-777c7cddf9-dggmj\" (UID: \"39788d92-6b78-413b-b9f4-75fc646c00a3\") " pod="ai-tenant-e2e-explicit-717161/explicit-model-717161-kserve-777c7cddf9-dggmj"
Sep 10 19:26:05.282198 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:05.281926    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39788d92-6b78-413b-b9f4-75fc646c00a3-home\") pod \"explicit-model-717161-kserve-777c7cddf9-dggmj\" (UID: \"39788d92-6b78-413b-b9f4-75fc646c00a3\") " pod="ai-tenant-e2e-explicit-717161/explicit-model-717161-kserve-777c7cddf9-dggmj"
Sep 10 19:26:05.282198 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:05.281948    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39788d92-6b78-413b-b9f4-75fc646c00a3-tls-certs\") pod \"explicit-model-717161-kserve-777c7cddf9-dggmj\" (UID: \"39788d92-6b78-413b-b9f4-75fc646c00a3\") " pod="ai-tenant-e2e-explicit-717161/explicit-model-717161-kserve-777c7cddf9-dggmj"
Sep 10 19:26:05.282198 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:05.281972    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39788d92-6b78-413b-b9f4-75fc646c00a3-dshm\") pod \"explicit-model-717161-kserve-777c7cddf9-dggmj\" (UID: \"39788d92-6b78-413b-b9f4-75fc646c00a3\") " pod="ai-tenant-e2e-explicit-717161/explicit-model-717161-kserve-777c7cddf9-dggmj"
Sep 10 19:26:05.282366 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:05.282293    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39788d92-6b78-413b-b9f4-75fc646c00a3-model-cache\") pod \"explicit-model-717161-kserve-777c7cddf9-dggmj\" (UID: \"39788d92-6b78-413b-b9f4-75fc646c00a3\") " pod="ai-tenant-e2e-explicit-717161/explicit-model-717161-kserve-777c7cddf9-dggmj"
Sep 10 19:26:05.282366 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:05.282326    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39788d92-6b78-413b-b9f4-75fc646c00a3-tmp-dir\") pod \"explicit-model-717161-kserve-777c7cddf9-dggmj\" (UID: \"39788d92-6b78-413b-b9f4-75fc646c00a3\") " pod="ai-tenant-e2e-explicit-717161/explicit-model-717161-kserve-777c7cddf9-dggmj"
Sep 10 19:26:05.282512 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:05.282405    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39788d92-6b78-413b-b9f4-75fc646c00a3-home\") pod \"explicit-model-717161-kserve-777c7cddf9-dggmj\" (UID: \"39788d92-6b78-413b-b9f4-75fc646c00a3\") " pod="ai-tenant-e2e-explicit-717161/explicit-model-717161-kserve-777c7cddf9-dggmj"
Sep 10 19:26:05.284246 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:05.284224    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39788d92-6b78-413b-b9f4-75fc646c00a3-dshm\") pod \"explicit-model-717161-kserve-777c7cddf9-dggmj\" (UID: \"39788d92-6b78-413b-b9f4-75fc646c00a3\") " pod="ai-tenant-e2e-explicit-717161/explicit-model-717161-kserve-777c7cddf9-dggmj"
Sep 10 19:26:05.284513 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:05.284497    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39788d92-6b78-413b-b9f4-75fc646c00a3-tls-certs\") pod \"explicit-model-717161-kserve-777c7cddf9-dggmj\" (UID: \"39788d92-6b78-413b-b9f4-75fc646c00a3\") " pod="ai-tenant-e2e-explicit-717161/explicit-model-717161-kserve-777c7cddf9-dggmj"
Sep 10 19:26:05.290272 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:05.290251    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jkxg\" (UniqueName: \"kubernetes.io/projected/39788d92-6b78-413b-b9f4-75fc646c00a3-kube-api-access-6jkxg\") pod \"explicit-model-717161-kserve-777c7cddf9-dggmj\" (UID: \"39788d92-6b78-413b-b9f4-75fc646c00a3\") " pod="ai-tenant-e2e-explicit-717161/explicit-model-717161-kserve-777c7cddf9-dggmj"
Sep 10 19:26:05.329479 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:05.329443    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-717161/explicit-model-717161-kserve-777c7cddf9-dggmj"
Sep 10 19:26:05.605050 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:05.605023    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-717161/explicit-model-717161-kserve-777c7cddf9-dggmj"]
Sep 10 19:26:05.606865 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:26:05.606840    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod39788d92_6b78_413b_b9f4_75fc646c00a3.slice/crio-161ace4ed2cd040b206a4fc1fcad82da292dc0897b0fdfc7f6ab1cf41683dd93 WatchSource:0}: Error finding container 161ace4ed2cd040b206a4fc1fcad82da292dc0897b0fdfc7f6ab1cf41683dd93: Status 404 returned error can't find the container with id 161ace4ed2cd040b206a4fc1fcad82da292dc0897b0fdfc7f6ab1cf41683dd93
Sep 10 19:26:06.494679 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:06.494630    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-717161/explicit-model-717161-kserve-777c7cddf9-dggmj" event={"ID":"39788d92-6b78-413b-b9f4-75fc646c00a3","Type":"ContainerStarted","Data":"6fd095f6ab9a5994e48f458b3782095baaa832a469ffaf161a6269b2a8956816"}
Sep 10 19:26:06.495273 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:06.494689    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-717161/explicit-model-717161-kserve-777c7cddf9-dggmj" event={"ID":"39788d92-6b78-413b-b9f4-75fc646c00a3","Type":"ContainerStarted","Data":"161ace4ed2cd040b206a4fc1fcad82da292dc0897b0fdfc7f6ab1cf41683dd93"}
Sep 10 19:26:06.514432 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:06.514384    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-717161/explicit-model-717161-kserve-777c7cddf9-dggmj" podStartSLOduration=2.514369385 podStartE2EDuration="2.514369385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:26:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:26:06.513535526 +0000 UTC m=+929.742585548" watchObservedRunningTime="2026-09-10 19:26:06.514369385 +0000 UTC m=+929.743419443"
Sep 10 19:26:09.526414 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:09.526379    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-88n8f"]
Sep 10 19:26:15.330565 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:15.330514    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-717161/explicit-model-717161-kserve-777c7cddf9-dggmj"
Sep 10 19:26:15.330968 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:15.330587    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-717161/explicit-model-717161-kserve-777c7cddf9-dggmj"
Sep 10 19:26:15.343299 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:15.343269    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-717161/explicit-model-717161-kserve-777c7cddf9-dggmj"
Sep 10 19:26:15.552884 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:15.552849    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-717161/explicit-model-717161-kserve-777c7cddf9-dggmj"
Sep 10 19:26:16.557888 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:16.557853    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-717161/explicit-model-717161-kserve-777c7cddf9-dggmj"]
Sep 10 19:26:16.685105 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:26:16.685069    2546 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-717161/explicit-model-717161-kserve-self-signed-certs: secret "explicit-model-717161-kserve-self-signed-certs" not found
Sep 10 19:26:16.685277 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:26:16.685143    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39788d92-6b78-413b-b9f4-75fc646c00a3-tls-certs podName:39788d92-6b78-413b-b9f4-75fc646c00a3 nodeName:}" failed. No retries permitted until 2026-09-10 19:26:17.185126384 +0000 UTC m=+940.414176388 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/39788d92-6b78-413b-b9f4-75fc646c00a3-tls-certs") pod "explicit-model-717161-kserve-777c7cddf9-dggmj" (UID: "39788d92-6b78-413b-b9f4-75fc646c00a3") : secret "explicit-model-717161-kserve-self-signed-certs" not found
Sep 10 19:26:17.191190 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:26:17.191150    2546 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-717161/explicit-model-717161-kserve-self-signed-certs: secret "explicit-model-717161-kserve-self-signed-certs" not found
Sep 10 19:26:17.191446 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:26:17.191243    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39788d92-6b78-413b-b9f4-75fc646c00a3-tls-certs podName:39788d92-6b78-413b-b9f4-75fc646c00a3 nodeName:}" failed. No retries permitted until 2026-09-10 19:26:18.191215505 +0000 UTC m=+941.420265509 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/39788d92-6b78-413b-b9f4-75fc646c00a3-tls-certs") pod "explicit-model-717161-kserve-777c7cddf9-dggmj" (UID: "39788d92-6b78-413b-b9f4-75fc646c00a3") : secret "explicit-model-717161-kserve-self-signed-certs" not found
Sep 10 19:26:18.200869 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:26:18.200827    2546 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-717161/explicit-model-717161-kserve-self-signed-certs: secret "explicit-model-717161-kserve-self-signed-certs" not found
Sep 10 19:26:18.201271 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:26:18.200907    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39788d92-6b78-413b-b9f4-75fc646c00a3-tls-certs podName:39788d92-6b78-413b-b9f4-75fc646c00a3 nodeName:}" failed. No retries permitted until 2026-09-10 19:26:20.200889191 +0000 UTC m=+943.429939195 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/39788d92-6b78-413b-b9f4-75fc646c00a3-tls-certs") pod "explicit-model-717161-kserve-777c7cddf9-dggmj" (UID: "39788d92-6b78-413b-b9f4-75fc646c00a3") : secret "explicit-model-717161-kserve-self-signed-certs" not found
Sep 10 19:26:18.585430 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:18.585327    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-c4487c09-openshift-default-7c76fd77dd-8j85g"]
Sep 10 19:26:18.588369 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:18.588332    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c4487c09-openshift-default-7c76fd77dd-8j85g"
Sep 10 19:26:18.591270 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:18.591246    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-c4487c09-openshift-default-dockercfg-hmm7d\""
Sep 10 19:26:18.601123 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:18.601097    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c4487c09-openshift-default-7c76fd77dd-8j85g"]
Sep 10 19:26:18.705329 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:18.705287    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc-istio-data\") pod \"e2e-mt-c4487c09-openshift-default-7c76fd77dd-8j85g\" (UID: \"2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc\") " pod="openshift-ingress/e2e-mt-c4487c09-openshift-default-7c76fd77dd-8j85g"
Sep 10 19:26:18.705561 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:18.705426    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc-istio-envoy\") pod \"e2e-mt-c4487c09-openshift-default-7c76fd77dd-8j85g\" (UID: \"2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc\") " pod="openshift-ingress/e2e-mt-c4487c09-openshift-default-7c76fd77dd-8j85g"
Sep 10 19:26:18.705561 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:18.705511    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc-istio-podinfo\") pod \"e2e-mt-c4487c09-openshift-default-7c76fd77dd-8j85g\" (UID: \"2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc\") " pod="openshift-ingress/e2e-mt-c4487c09-openshift-default-7c76fd77dd-8j85g"
Sep 10 19:26:18.705561 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:18.705547    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc-workload-certs\") pod \"e2e-mt-c4487c09-openshift-default-7c76fd77dd-8j85g\" (UID: \"2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc\") " pod="openshift-ingress/e2e-mt-c4487c09-openshift-default-7c76fd77dd-8j85g"
Sep 10 19:26:18.705711 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:18.705614    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7fc7\" (UniqueName: \"kubernetes.io/projected/2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc-kube-api-access-l7fc7\") pod \"e2e-mt-c4487c09-openshift-default-7c76fd77dd-8j85g\" (UID: \"2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc\") " pod="openshift-ingress/e2e-mt-c4487c09-openshift-default-7c76fd77dd-8j85g"
Sep 10 19:26:18.705711 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:18.705653    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc-istiod-ca-cert\") pod \"e2e-mt-c4487c09-openshift-default-7c76fd77dd-8j85g\" (UID: \"2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc\") " pod="openshift-ingress/e2e-mt-c4487c09-openshift-default-7c76fd77dd-8j85g"
Sep 10 19:26:18.705775 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:18.705720    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc-workload-socket\") pod \"e2e-mt-c4487c09-openshift-default-7c76fd77dd-8j85g\" (UID: \"2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc\") " pod="openshift-ingress/e2e-mt-c4487c09-openshift-default-7c76fd77dd-8j85g"
Sep 10 19:26:18.705775 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:18.705762    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc-credential-socket\") pod \"e2e-mt-c4487c09-openshift-default-7c76fd77dd-8j85g\" (UID: \"2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc\") " pod="openshift-ingress/e2e-mt-c4487c09-openshift-default-7c76fd77dd-8j85g"
Sep 10 19:26:18.705836 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:18.705795    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc-istio-token\") pod \"e2e-mt-c4487c09-openshift-default-7c76fd77dd-8j85g\" (UID: \"2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc\") " pod="openshift-ingress/e2e-mt-c4487c09-openshift-default-7c76fd77dd-8j85g"
Sep 10 19:26:18.806490 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:18.806430    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc-istio-data\") pod \"e2e-mt-c4487c09-openshift-default-7c76fd77dd-8j85g\" (UID: \"2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc\") " pod="openshift-ingress/e2e-mt-c4487c09-openshift-default-7c76fd77dd-8j85g"
Sep 10 19:26:18.806698 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:18.806522    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc-istio-envoy\") pod \"e2e-mt-c4487c09-openshift-default-7c76fd77dd-8j85g\" (UID: \"2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc\") " pod="openshift-ingress/e2e-mt-c4487c09-openshift-default-7c76fd77dd-8j85g"
Sep 10 19:26:18.806698 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:18.806584    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc-istio-podinfo\") pod \"e2e-mt-c4487c09-openshift-default-7c76fd77dd-8j85g\" (UID: \"2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc\") " pod="openshift-ingress/e2e-mt-c4487c09-openshift-default-7c76fd77dd-8j85g"
Sep 10 19:26:18.806698 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:18.806612    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc-workload-certs\") pod \"e2e-mt-c4487c09-openshift-default-7c76fd77dd-8j85g\" (UID: \"2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc\") " pod="openshift-ingress/e2e-mt-c4487c09-openshift-default-7c76fd77dd-8j85g"
Sep 10 19:26:18.806698 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:18.806651    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7fc7\" (UniqueName: \"kubernetes.io/projected/2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc-kube-api-access-l7fc7\") pod \"e2e-mt-c4487c09-openshift-default-7c76fd77dd-8j85g\" (UID: \"2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc\") " pod="openshift-ingress/e2e-mt-c4487c09-openshift-default-7c76fd77dd-8j85g"
Sep 10 19:26:18.806698 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:18.806677    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc-istiod-ca-cert\") pod \"e2e-mt-c4487c09-openshift-default-7c76fd77dd-8j85g\" (UID: \"2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc\") " pod="openshift-ingress/e2e-mt-c4487c09-openshift-default-7c76fd77dd-8j85g"
Sep 10 19:26:18.807275 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:18.806883    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc-istio-data\") pod \"e2e-mt-c4487c09-openshift-default-7c76fd77dd-8j85g\" (UID: \"2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc\") " pod="openshift-ingress/e2e-mt-c4487c09-openshift-default-7c76fd77dd-8j85g"
Sep 10 19:26:18.807275 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:18.807010    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc-workload-socket\") pod \"e2e-mt-c4487c09-openshift-default-7c76fd77dd-8j85g\" (UID: \"2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc\") " pod="openshift-ingress/e2e-mt-c4487c09-openshift-default-7c76fd77dd-8j85g"
Sep 10 19:26:18.807275 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:18.807065    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc-credential-socket\") pod \"e2e-mt-c4487c09-openshift-default-7c76fd77dd-8j85g\" (UID: \"2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc\") " pod="openshift-ingress/e2e-mt-c4487c09-openshift-default-7c76fd77dd-8j85g"
Sep 10 19:26:18.807275 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:18.807091    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc-istio-token\") pod \"e2e-mt-c4487c09-openshift-default-7c76fd77dd-8j85g\" (UID: \"2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc\") " pod="openshift-ingress/e2e-mt-c4487c09-openshift-default-7c76fd77dd-8j85g"
Sep 10 19:26:18.807275 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:18.807117    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc-workload-certs\") pod \"e2e-mt-c4487c09-openshift-default-7c76fd77dd-8j85g\" (UID: \"2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc\") " pod="openshift-ingress/e2e-mt-c4487c09-openshift-default-7c76fd77dd-8j85g"
Sep 10 19:26:18.807584 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:18.807363    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc-workload-socket\") pod \"e2e-mt-c4487c09-openshift-default-7c76fd77dd-8j85g\" (UID: \"2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc\") " pod="openshift-ingress/e2e-mt-c4487c09-openshift-default-7c76fd77dd-8j85g"
Sep 10 19:26:18.807584 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:18.807403    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc-credential-socket\") pod \"e2e-mt-c4487c09-openshift-default-7c76fd77dd-8j85g\" (UID: \"2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc\") " pod="openshift-ingress/e2e-mt-c4487c09-openshift-default-7c76fd77dd-8j85g"
Sep 10 19:26:18.807912 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:18.807891    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc-istiod-ca-cert\") pod \"e2e-mt-c4487c09-openshift-default-7c76fd77dd-8j85g\" (UID: \"2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc\") " pod="openshift-ingress/e2e-mt-c4487c09-openshift-default-7c76fd77dd-8j85g"
Sep 10 19:26:18.808929 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:18.808908    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc-istio-envoy\") pod \"e2e-mt-c4487c09-openshift-default-7c76fd77dd-8j85g\" (UID: \"2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc\") " pod="openshift-ingress/e2e-mt-c4487c09-openshift-default-7c76fd77dd-8j85g"
Sep 10 19:26:18.809649 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:18.809624    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc-istio-podinfo\") pod \"e2e-mt-c4487c09-openshift-default-7c76fd77dd-8j85g\" (UID: \"2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc\") " pod="openshift-ingress/e2e-mt-c4487c09-openshift-default-7c76fd77dd-8j85g"
Sep 10 19:26:18.816972 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:18.816936    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc-istio-token\") pod \"e2e-mt-c4487c09-openshift-default-7c76fd77dd-8j85g\" (UID: \"2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc\") " pod="openshift-ingress/e2e-mt-c4487c09-openshift-default-7c76fd77dd-8j85g"
Sep 10 19:26:18.817105 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:18.817076    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7fc7\" (UniqueName: \"kubernetes.io/projected/2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc-kube-api-access-l7fc7\") pod \"e2e-mt-c4487c09-openshift-default-7c76fd77dd-8j85g\" (UID: \"2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc\") " pod="openshift-ingress/e2e-mt-c4487c09-openshift-default-7c76fd77dd-8j85g"
Sep 10 19:26:18.900662 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:18.900617    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c4487c09-openshift-default-7c76fd77dd-8j85g"
Sep 10 19:26:19.193555 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:19.193524    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c4487c09-openshift-default-7c76fd77dd-8j85g"]
Sep 10 19:26:19.195633 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:26:19.195593    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e741ef0_d7d4_47f1_b41a_b3795a5b9fdc.slice/crio-6a8f91d89b5aca53f6479447ca9f7befe6e4fe3b07c544f7ea4aa82974812be5 WatchSource:0}: Error finding container 6a8f91d89b5aca53f6479447ca9f7befe6e4fe3b07c544f7ea4aa82974812be5: Status 404 returned error can't find the container with id 6a8f91d89b5aca53f6479447ca9f7befe6e4fe3b07c544f7ea4aa82974812be5
Sep 10 19:26:19.197929 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:19.197893    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 19:26:19.198027 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:19.197973    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 19:26:19.198027 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:19.198022    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 19:26:19.567030 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:19.566928    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c4487c09-openshift-default-7c76fd77dd-8j85g" event={"ID":"2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc","Type":"ContainerStarted","Data":"3a0b56aac008ccaec717cb3177f20a7a7fbc8222706bad336b23ac9efe9f65ed"}
Sep 10 19:26:19.567030 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:19.566983    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c4487c09-openshift-default-7c76fd77dd-8j85g" event={"ID":"2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc","Type":"ContainerStarted","Data":"6a8f91d89b5aca53f6479447ca9f7befe6e4fe3b07c544f7ea4aa82974812be5"}
Sep 10 19:26:19.595945 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:19.595891    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-c4487c09-openshift-default-7c76fd77dd-8j85g" podStartSLOduration=1.595874347 podStartE2EDuration="1.595874347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:26:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:26:19.594398137 +0000 UTC m=+942.823448163" watchObservedRunningTime="2026-09-10 19:26:19.595874347 +0000 UTC m=+942.824924372"
Sep 10 19:26:19.901724 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:19.901682    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c4487c09-openshift-default-7c76fd77dd-8j85g"
Sep 10 19:26:20.222012 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:26:20.221916    2546 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-717161/explicit-model-717161-kserve-self-signed-certs: secret "explicit-model-717161-kserve-self-signed-certs" not found
Sep 10 19:26:20.222012 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:26:20.221994    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39788d92-6b78-413b-b9f4-75fc646c00a3-tls-certs podName:39788d92-6b78-413b-b9f4-75fc646c00a3 nodeName:}" failed. No retries permitted until 2026-09-10 19:26:24.221977614 +0000 UTC m=+947.451027617 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/39788d92-6b78-413b-b9f4-75fc646c00a3-tls-certs") pod "explicit-model-717161-kserve-777c7cddf9-dggmj" (UID: "39788d92-6b78-413b-b9f4-75fc646c00a3") : secret "explicit-model-717161-kserve-self-signed-certs" not found
Sep 10 19:26:20.906725 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:20.906680    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c4487c09-openshift-default-7c76fd77dd-8j85g"
Sep 10 19:26:21.577939 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:21.577908    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c4487c09-openshift-default-7c76fd77dd-8j85g"
Sep 10 19:26:21.578869 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:21.578846    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c4487c09-openshift-default-7c76fd77dd-8j85g"
Sep 10 19:26:24.259213 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:26:24.259177    2546 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-717161/explicit-model-717161-kserve-self-signed-certs: secret "explicit-model-717161-kserve-self-signed-certs" not found
Sep 10 19:26:24.259724 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:26:24.259257    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39788d92-6b78-413b-b9f4-75fc646c00a3-tls-certs podName:39788d92-6b78-413b-b9f4-75fc646c00a3 nodeName:}" failed. No retries permitted until 2026-09-10 19:26:32.259239073 +0000 UTC m=+955.488289077 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/39788d92-6b78-413b-b9f4-75fc646c00a3-tls-certs") pod "explicit-model-717161-kserve-777c7cddf9-dggmj" (UID: "39788d92-6b78-413b-b9f4-75fc646c00a3") : secret "explicit-model-717161-kserve-self-signed-certs" not found
Sep 10 19:26:26.724243 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:26.724201    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-88n8f"]
Sep 10 19:26:32.341790 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:26:32.341738    2546 secret.go:189] Couldn't get secret ai-tenant-e2e-explicit-717161/explicit-model-717161-kserve-self-signed-certs: secret "explicit-model-717161-kserve-self-signed-certs" not found
Sep 10 19:26:32.342190 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:26:32.341852    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39788d92-6b78-413b-b9f4-75fc646c00a3-tls-certs podName:39788d92-6b78-413b-b9f4-75fc646c00a3 nodeName:}" failed. No retries permitted until 2026-09-10 19:26:48.341828814 +0000 UTC m=+971.570878819 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/39788d92-6b78-413b-b9f4-75fc646c00a3-tls-certs") pod "explicit-model-717161-kserve-777c7cddf9-dggmj" (UID: "39788d92-6b78-413b-b9f4-75fc646c00a3") : secret "explicit-model-717161-kserve-self-signed-certs" not found
Sep 10 19:26:32.575082 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:32.575027    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-717161/explicit-model-717161-kserve-777c7cddf9-dggmj" podUID="39788d92-6b78-413b-b9f4-75fc646c00a3" containerName="main" containerID="cri-o://6fd095f6ab9a5994e48f458b3782095baaa832a469ffaf161a6269b2a8956816" gracePeriod=45
Sep 10 19:26:32.855651 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:32.855587    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-717161/explicit-model-717161-kserve-777c7cddf9-dggmj"
Sep 10 19:26:32.948054 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:32.948019    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39788d92-6b78-413b-b9f4-75fc646c00a3-model-cache\") pod \"39788d92-6b78-413b-b9f4-75fc646c00a3\" (UID: \"39788d92-6b78-413b-b9f4-75fc646c00a3\") "
Sep 10 19:26:32.948054 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:32.948063    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39788d92-6b78-413b-b9f4-75fc646c00a3-home\") pod \"39788d92-6b78-413b-b9f4-75fc646c00a3\" (UID: \"39788d92-6b78-413b-b9f4-75fc646c00a3\") "
Sep 10 19:26:32.948408 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:32.948099    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39788d92-6b78-413b-b9f4-75fc646c00a3-tmp-dir\") pod \"39788d92-6b78-413b-b9f4-75fc646c00a3\" (UID: \"39788d92-6b78-413b-b9f4-75fc646c00a3\") "
Sep 10 19:26:32.948408 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:32.948126    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39788d92-6b78-413b-b9f4-75fc646c00a3-tls-certs\") pod \"39788d92-6b78-413b-b9f4-75fc646c00a3\" (UID: \"39788d92-6b78-413b-b9f4-75fc646c00a3\") "
Sep 10 19:26:32.948408 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:32.948162    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6jkxg\" (UniqueName: \"kubernetes.io/projected/39788d92-6b78-413b-b9f4-75fc646c00a3-kube-api-access-6jkxg\") pod \"39788d92-6b78-413b-b9f4-75fc646c00a3\" (UID: \"39788d92-6b78-413b-b9f4-75fc646c00a3\") "
Sep 10 19:26:32.948408 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:32.948260    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39788d92-6b78-413b-b9f4-75fc646c00a3-dshm\") pod \"39788d92-6b78-413b-b9f4-75fc646c00a3\" (UID: \"39788d92-6b78-413b-b9f4-75fc646c00a3\") "
Sep 10 19:26:32.948408 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:32.948329    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39788d92-6b78-413b-b9f4-75fc646c00a3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "39788d92-6b78-413b-b9f4-75fc646c00a3" (UID: "39788d92-6b78-413b-b9f4-75fc646c00a3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:26:32.948408 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:32.948356    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39788d92-6b78-413b-b9f4-75fc646c00a3-model-cache" (OuterVolumeSpecName: "model-cache") pod "39788d92-6b78-413b-b9f4-75fc646c00a3" (UID: "39788d92-6b78-413b-b9f4-75fc646c00a3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:26:32.948408 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:32.948403    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39788d92-6b78-413b-b9f4-75fc646c00a3-home" (OuterVolumeSpecName: "home") pod "39788d92-6b78-413b-b9f4-75fc646c00a3" (UID: "39788d92-6b78-413b-b9f4-75fc646c00a3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:26:32.948762 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:32.948605    2546 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39788d92-6b78-413b-b9f4-75fc646c00a3-model-cache\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:26:32.948762 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:32.948619    2546 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39788d92-6b78-413b-b9f4-75fc646c00a3-home\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:26:32.948762 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:32.948628    2546 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39788d92-6b78-413b-b9f4-75fc646c00a3-tmp-dir\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:26:32.950559 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:32.950529    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39788d92-6b78-413b-b9f4-75fc646c00a3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "39788d92-6b78-413b-b9f4-75fc646c00a3" (UID: "39788d92-6b78-413b-b9f4-75fc646c00a3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 19:26:32.950559 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:32.950548    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39788d92-6b78-413b-b9f4-75fc646c00a3-kube-api-access-6jkxg" (OuterVolumeSpecName: "kube-api-access-6jkxg") pod "39788d92-6b78-413b-b9f4-75fc646c00a3" (UID: "39788d92-6b78-413b-b9f4-75fc646c00a3"). InnerVolumeSpecName "kube-api-access-6jkxg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 19:26:32.951212 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:32.951191    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39788d92-6b78-413b-b9f4-75fc646c00a3-dshm" (OuterVolumeSpecName: "dshm") pod "39788d92-6b78-413b-b9f4-75fc646c00a3" (UID: "39788d92-6b78-413b-b9f4-75fc646c00a3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:26:33.049891 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:33.049846    2546 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39788d92-6b78-413b-b9f4-75fc646c00a3-dshm\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:26:33.049891 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:33.049881    2546 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39788d92-6b78-413b-b9f4-75fc646c00a3-tls-certs\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:26:33.049891 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:33.049897    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jkxg\" (UniqueName: \"kubernetes.io/projected/39788d92-6b78-413b-b9f4-75fc646c00a3-kube-api-access-6jkxg\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:26:33.644047 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:33.644015    2546 generic.go:356] "Generic (PLEG): container finished" podID="39788d92-6b78-413b-b9f4-75fc646c00a3" containerID="6fd095f6ab9a5994e48f458b3782095baaa832a469ffaf161a6269b2a8956816" exitCode=0
Sep 10 19:26:33.644514 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:33.644061    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-717161/explicit-model-717161-kserve-777c7cddf9-dggmj"
Sep 10 19:26:33.644514 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:33.644107    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-717161/explicit-model-717161-kserve-777c7cddf9-dggmj" event={"ID":"39788d92-6b78-413b-b9f4-75fc646c00a3","Type":"ContainerDied","Data":"6fd095f6ab9a5994e48f458b3782095baaa832a469ffaf161a6269b2a8956816"}
Sep 10 19:26:33.644514 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:33.644152    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-717161/explicit-model-717161-kserve-777c7cddf9-dggmj" event={"ID":"39788d92-6b78-413b-b9f4-75fc646c00a3","Type":"ContainerDied","Data":"161ace4ed2cd040b206a4fc1fcad82da292dc0897b0fdfc7f6ab1cf41683dd93"}
Sep 10 19:26:33.644514 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:33.644178    2546 scope.go:117] "RemoveContainer" containerID="6fd095f6ab9a5994e48f458b3782095baaa832a469ffaf161a6269b2a8956816"
Sep 10 19:26:33.653048 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:33.653030    2546 scope.go:117] "RemoveContainer" containerID="6fd095f6ab9a5994e48f458b3782095baaa832a469ffaf161a6269b2a8956816"
Sep 10 19:26:33.653366 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:26:33.653328    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fd095f6ab9a5994e48f458b3782095baaa832a469ffaf161a6269b2a8956816\": container with ID starting with 6fd095f6ab9a5994e48f458b3782095baaa832a469ffaf161a6269b2a8956816 not found: ID does not exist" containerID="6fd095f6ab9a5994e48f458b3782095baaa832a469ffaf161a6269b2a8956816"
Sep 10 19:26:33.653474 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:33.653374    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fd095f6ab9a5994e48f458b3782095baaa832a469ffaf161a6269b2a8956816"} err="failed to get container status \"6fd095f6ab9a5994e48f458b3782095baaa832a469ffaf161a6269b2a8956816\": rpc error: code = NotFound desc = could not find container \"6fd095f6ab9a5994e48f458b3782095baaa832a469ffaf161a6269b2a8956816\": container with ID starting with 6fd095f6ab9a5994e48f458b3782095baaa832a469ffaf161a6269b2a8956816 not found: ID does not exist"
Sep 10 19:26:33.661450 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:33.661414    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-717161/explicit-model-717161-kserve-777c7cddf9-dggmj"]
Sep 10 19:26:33.663318 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:33.663292    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-717161/explicit-model-717161-kserve-777c7cddf9-dggmj"]
Sep 10 19:26:35.282995 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:35.282958    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39788d92-6b78-413b-b9f4-75fc646c00a3" path="/var/lib/kubelet/pods/39788d92-6b78-413b-b9f4-75fc646c00a3/volumes"
Sep 10 19:26:35.971037 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:35.971001    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-70a2c9-openshift-default-68d7cc875b-m84w5"]
Sep 10 19:26:35.972143 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:35.971602    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39788d92-6b78-413b-b9f4-75fc646c00a3" containerName="main"
Sep 10 19:26:35.972143 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:35.971621    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="39788d92-6b78-413b-b9f4-75fc646c00a3" containerName="main"
Sep 10 19:26:35.972143 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:35.971722    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="39788d92-6b78-413b-b9f4-75fc646c00a3" containerName="main"
Sep 10 19:26:35.985726 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:35.984956    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-70a2c9-openshift-default-68d7cc875b-m84w5"
Sep 10 19:26:35.988204 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:35.988078    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-70a2c9-openshift-default-dockercfg-cwk8w\""
Sep 10 19:26:35.989857 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:35.989829    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-70a2c9-openshift-default-68d7cc875b-m84w5"]
Sep 10 19:26:36.084233 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:36.084198    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a9ed50b-ee46-40aa-a4d2-22c964b7f129-workload-socket\") pod \"e2e-noten-70a2c9-openshift-default-68d7cc875b-m84w5\" (UID: \"8a9ed50b-ee46-40aa-a4d2-22c964b7f129\") " pod="openshift-ingress/e2e-noten-70a2c9-openshift-default-68d7cc875b-m84w5"
Sep 10 19:26:36.084233 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:36.084235    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a9ed50b-ee46-40aa-a4d2-22c964b7f129-istio-token\") pod \"e2e-noten-70a2c9-openshift-default-68d7cc875b-m84w5\" (UID: \"8a9ed50b-ee46-40aa-a4d2-22c964b7f129\") " pod="openshift-ingress/e2e-noten-70a2c9-openshift-default-68d7cc875b-m84w5"
Sep 10 19:26:36.084461 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:36.084256    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a9ed50b-ee46-40aa-a4d2-22c964b7f129-istio-envoy\") pod \"e2e-noten-70a2c9-openshift-default-68d7cc875b-m84w5\" (UID: \"8a9ed50b-ee46-40aa-a4d2-22c964b7f129\") " pod="openshift-ingress/e2e-noten-70a2c9-openshift-default-68d7cc875b-m84w5"
Sep 10 19:26:36.084461 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:36.084296    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a9ed50b-ee46-40aa-a4d2-22c964b7f129-istiod-ca-cert\") pod \"e2e-noten-70a2c9-openshift-default-68d7cc875b-m84w5\" (UID: \"8a9ed50b-ee46-40aa-a4d2-22c964b7f129\") " pod="openshift-ingress/e2e-noten-70a2c9-openshift-default-68d7cc875b-m84w5"
Sep 10 19:26:36.084461 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:36.084445    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a9ed50b-ee46-40aa-a4d2-22c964b7f129-istio-data\") pod \"e2e-noten-70a2c9-openshift-default-68d7cc875b-m84w5\" (UID: \"8a9ed50b-ee46-40aa-a4d2-22c964b7f129\") " pod="openshift-ingress/e2e-noten-70a2c9-openshift-default-68d7cc875b-m84w5"
Sep 10 19:26:36.084560 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:36.084477    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a9ed50b-ee46-40aa-a4d2-22c964b7f129-istio-podinfo\") pod \"e2e-noten-70a2c9-openshift-default-68d7cc875b-m84w5\" (UID: \"8a9ed50b-ee46-40aa-a4d2-22c964b7f129\") " pod="openshift-ingress/e2e-noten-70a2c9-openshift-default-68d7cc875b-m84w5"
Sep 10 19:26:36.084560 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:36.084518    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a9ed50b-ee46-40aa-a4d2-22c964b7f129-credential-socket\") pod \"e2e-noten-70a2c9-openshift-default-68d7cc875b-m84w5\" (UID: \"8a9ed50b-ee46-40aa-a4d2-22c964b7f129\") " pod="openshift-ingress/e2e-noten-70a2c9-openshift-default-68d7cc875b-m84w5"
Sep 10 19:26:36.084560 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:36.084549    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a9ed50b-ee46-40aa-a4d2-22c964b7f129-workload-certs\") pod \"e2e-noten-70a2c9-openshift-default-68d7cc875b-m84w5\" (UID: \"8a9ed50b-ee46-40aa-a4d2-22c964b7f129\") " pod="openshift-ingress/e2e-noten-70a2c9-openshift-default-68d7cc875b-m84w5"
Sep 10 19:26:36.084655 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:36.084614    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7vmj\" (UniqueName: \"kubernetes.io/projected/8a9ed50b-ee46-40aa-a4d2-22c964b7f129-kube-api-access-j7vmj\") pod \"e2e-noten-70a2c9-openshift-default-68d7cc875b-m84w5\" (UID: \"8a9ed50b-ee46-40aa-a4d2-22c964b7f129\") " pod="openshift-ingress/e2e-noten-70a2c9-openshift-default-68d7cc875b-m84w5"
Sep 10 19:26:36.186292 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:36.186197    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a9ed50b-ee46-40aa-a4d2-22c964b7f129-istio-token\") pod \"e2e-noten-70a2c9-openshift-default-68d7cc875b-m84w5\" (UID: \"8a9ed50b-ee46-40aa-a4d2-22c964b7f129\") " pod="openshift-ingress/e2e-noten-70a2c9-openshift-default-68d7cc875b-m84w5"
Sep 10 19:26:36.186504 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:36.186368    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a9ed50b-ee46-40aa-a4d2-22c964b7f129-istio-envoy\") pod \"e2e-noten-70a2c9-openshift-default-68d7cc875b-m84w5\" (UID: \"8a9ed50b-ee46-40aa-a4d2-22c964b7f129\") " pod="openshift-ingress/e2e-noten-70a2c9-openshift-default-68d7cc875b-m84w5"
Sep 10 19:26:36.186504 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:36.186398    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a9ed50b-ee46-40aa-a4d2-22c964b7f129-istiod-ca-cert\") pod \"e2e-noten-70a2c9-openshift-default-68d7cc875b-m84w5\" (UID: \"8a9ed50b-ee46-40aa-a4d2-22c964b7f129\") " pod="openshift-ingress/e2e-noten-70a2c9-openshift-default-68d7cc875b-m84w5"
Sep 10 19:26:36.186504 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:36.186482    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a9ed50b-ee46-40aa-a4d2-22c964b7f129-istio-data\") pod \"e2e-noten-70a2c9-openshift-default-68d7cc875b-m84w5\" (UID: \"8a9ed50b-ee46-40aa-a4d2-22c964b7f129\") " pod="openshift-ingress/e2e-noten-70a2c9-openshift-default-68d7cc875b-m84w5"
Sep 10 19:26:36.186687 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:36.186521    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a9ed50b-ee46-40aa-a4d2-22c964b7f129-istio-podinfo\") pod \"e2e-noten-70a2c9-openshift-default-68d7cc875b-m84w5\" (UID: \"8a9ed50b-ee46-40aa-a4d2-22c964b7f129\") " pod="openshift-ingress/e2e-noten-70a2c9-openshift-default-68d7cc875b-m84w5"
Sep 10 19:26:36.186687 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:36.186590    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a9ed50b-ee46-40aa-a4d2-22c964b7f129-credential-socket\") pod \"e2e-noten-70a2c9-openshift-default-68d7cc875b-m84w5\" (UID: \"8a9ed50b-ee46-40aa-a4d2-22c964b7f129\") " pod="openshift-ingress/e2e-noten-70a2c9-openshift-default-68d7cc875b-m84w5"
Sep 10 19:26:36.186687 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:36.186630    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a9ed50b-ee46-40aa-a4d2-22c964b7f129-workload-certs\") pod \"e2e-noten-70a2c9-openshift-default-68d7cc875b-m84w5\" (UID: \"8a9ed50b-ee46-40aa-a4d2-22c964b7f129\") " pod="openshift-ingress/e2e-noten-70a2c9-openshift-default-68d7cc875b-m84w5"
Sep 10 19:26:36.186687 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:36.186671    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7vmj\" (UniqueName: \"kubernetes.io/projected/8a9ed50b-ee46-40aa-a4d2-22c964b7f129-kube-api-access-j7vmj\") pod \"e2e-noten-70a2c9-openshift-default-68d7cc875b-m84w5\" (UID: \"8a9ed50b-ee46-40aa-a4d2-22c964b7f129\") " pod="openshift-ingress/e2e-noten-70a2c9-openshift-default-68d7cc875b-m84w5"
Sep 10 19:26:36.186887 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:36.186744    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a9ed50b-ee46-40aa-a4d2-22c964b7f129-workload-socket\") pod \"e2e-noten-70a2c9-openshift-default-68d7cc875b-m84w5\" (UID: \"8a9ed50b-ee46-40aa-a4d2-22c964b7f129\") " pod="openshift-ingress/e2e-noten-70a2c9-openshift-default-68d7cc875b-m84w5"
Sep 10 19:26:36.186973 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:36.186936    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a9ed50b-ee46-40aa-a4d2-22c964b7f129-istio-data\") pod \"e2e-noten-70a2c9-openshift-default-68d7cc875b-m84w5\" (UID: \"8a9ed50b-ee46-40aa-a4d2-22c964b7f129\") " pod="openshift-ingress/e2e-noten-70a2c9-openshift-default-68d7cc875b-m84w5"
Sep 10 19:26:36.187074 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:36.187050    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a9ed50b-ee46-40aa-a4d2-22c964b7f129-workload-socket\") pod \"e2e-noten-70a2c9-openshift-default-68d7cc875b-m84w5\" (UID: \"8a9ed50b-ee46-40aa-a4d2-22c964b7f129\") " pod="openshift-ingress/e2e-noten-70a2c9-openshift-default-68d7cc875b-m84w5"
Sep 10 19:26:36.187322 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:36.187298    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a9ed50b-ee46-40aa-a4d2-22c964b7f129-istiod-ca-cert\") pod \"e2e-noten-70a2c9-openshift-default-68d7cc875b-m84w5\" (UID: \"8a9ed50b-ee46-40aa-a4d2-22c964b7f129\") " pod="openshift-ingress/e2e-noten-70a2c9-openshift-default-68d7cc875b-m84w5"
Sep 10 19:26:36.187462 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:36.187399    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a9ed50b-ee46-40aa-a4d2-22c964b7f129-credential-socket\") pod \"e2e-noten-70a2c9-openshift-default-68d7cc875b-m84w5\" (UID: \"8a9ed50b-ee46-40aa-a4d2-22c964b7f129\") " pod="openshift-ingress/e2e-noten-70a2c9-openshift-default-68d7cc875b-m84w5"
Sep 10 19:26:36.187462 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:36.187303    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a9ed50b-ee46-40aa-a4d2-22c964b7f129-workload-certs\") pod \"e2e-noten-70a2c9-openshift-default-68d7cc875b-m84w5\" (UID: \"8a9ed50b-ee46-40aa-a4d2-22c964b7f129\") " pod="openshift-ingress/e2e-noten-70a2c9-openshift-default-68d7cc875b-m84w5"
Sep 10 19:26:36.188797 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:36.188777    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a9ed50b-ee46-40aa-a4d2-22c964b7f129-istio-envoy\") pod \"e2e-noten-70a2c9-openshift-default-68d7cc875b-m84w5\" (UID: \"8a9ed50b-ee46-40aa-a4d2-22c964b7f129\") " pod="openshift-ingress/e2e-noten-70a2c9-openshift-default-68d7cc875b-m84w5"
Sep 10 19:26:36.189210 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:36.189188    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a9ed50b-ee46-40aa-a4d2-22c964b7f129-istio-podinfo\") pod \"e2e-noten-70a2c9-openshift-default-68d7cc875b-m84w5\" (UID: \"8a9ed50b-ee46-40aa-a4d2-22c964b7f129\") " pod="openshift-ingress/e2e-noten-70a2c9-openshift-default-68d7cc875b-m84w5"
Sep 10 19:26:36.193864 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:36.193841    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a9ed50b-ee46-40aa-a4d2-22c964b7f129-istio-token\") pod \"e2e-noten-70a2c9-openshift-default-68d7cc875b-m84w5\" (UID: \"8a9ed50b-ee46-40aa-a4d2-22c964b7f129\") " pod="openshift-ingress/e2e-noten-70a2c9-openshift-default-68d7cc875b-m84w5"
Sep 10 19:26:36.194255 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:36.194236    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7vmj\" (UniqueName: \"kubernetes.io/projected/8a9ed50b-ee46-40aa-a4d2-22c964b7f129-kube-api-access-j7vmj\") pod \"e2e-noten-70a2c9-openshift-default-68d7cc875b-m84w5\" (UID: \"8a9ed50b-ee46-40aa-a4d2-22c964b7f129\") " pod="openshift-ingress/e2e-noten-70a2c9-openshift-default-68d7cc875b-m84w5"
Sep 10 19:26:36.300691 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:36.300604    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-70a2c9-openshift-default-68d7cc875b-m84w5"
Sep 10 19:26:36.578455 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:36.578417    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-70a2c9-openshift-default-68d7cc875b-m84w5"]
Sep 10 19:26:36.579713 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:36.579682    2546 patch_prober.go:28] interesting pod/e2e-mt-c4487c09-openshift-default-7c76fd77dd-8j85g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Sep 10 19:26:36.579846 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:36.579734    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-c4487c09-openshift-default-7c76fd77dd-8j85g" podUID="2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Sep 10 19:26:36.581133 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:26:36.581105    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a9ed50b_ee46_40aa_a4d2_22c964b7f129.slice/crio-3cedb5c437e9b9e030b8f132a8e0b1feaae666655548608d902e47fe31986910 WatchSource:0}: Error finding container 3cedb5c437e9b9e030b8f132a8e0b1feaae666655548608d902e47fe31986910: Status 404 returned error can't find the container with id 3cedb5c437e9b9e030b8f132a8e0b1feaae666655548608d902e47fe31986910
Sep 10 19:26:36.583317 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:36.583280    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 19:26:36.583416 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:36.583369    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 19:26:36.583416 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:36.583400    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 19:26:36.665508 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:36.665399    2546 generic.go:356] "Generic (PLEG): container finished" podID="2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc" containerID="3a0b56aac008ccaec717cb3177f20a7a7fbc8222706bad336b23ac9efe9f65ed" exitCode=0
Sep 10 19:26:36.666025 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:36.665437    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c4487c09-openshift-default-7c76fd77dd-8j85g" event={"ID":"2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc","Type":"ContainerDied","Data":"3a0b56aac008ccaec717cb3177f20a7a7fbc8222706bad336b23ac9efe9f65ed"}
Sep 10 19:26:36.666419 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:36.666189    2546 scope.go:117] "RemoveContainer" containerID="3a0b56aac008ccaec717cb3177f20a7a7fbc8222706bad336b23ac9efe9f65ed"
Sep 10 19:26:36.667448 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:36.667426    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-70a2c9-openshift-default-68d7cc875b-m84w5" event={"ID":"8a9ed50b-ee46-40aa-a4d2-22c964b7f129","Type":"ContainerStarted","Data":"319c986bdb9b7f003dcace7abd856c82c8232bc15d22809e681a64d0e522d5a1"}
Sep 10 19:26:36.667554 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:36.667473    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-70a2c9-openshift-default-68d7cc875b-m84w5" event={"ID":"8a9ed50b-ee46-40aa-a4d2-22c964b7f129","Type":"ContainerStarted","Data":"3cedb5c437e9b9e030b8f132a8e0b1feaae666655548608d902e47fe31986910"}
Sep 10 19:26:36.667554 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:36.667471    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 19:26:36.667554 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:36.667546    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 19:26:36.667720 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:36.667586    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 19:26:36.716001 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:36.710905    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-70a2c9-openshift-default-68d7cc875b-m84w5" podStartSLOduration=1.710882706 podStartE2EDuration="1.710882706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:26:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:26:36.706607752 +0000 UTC m=+959.935657779" watchObservedRunningTime="2026-09-10 19:26:36.710882706 +0000 UTC m=+959.939932732"
Sep 10 19:26:37.301030 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:37.301001    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-70a2c9-openshift-default-68d7cc875b-m84w5"
Sep 10 19:26:37.305971 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:37.305944    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-70a2c9-openshift-default-68d7cc875b-m84w5"
Sep 10 19:26:37.681751 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:37.681707    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c4487c09-openshift-default-7c76fd77dd-8j85g" event={"ID":"2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc","Type":"ContainerStarted","Data":"40c65d37725bfc98647f738e3391eddd0b06483d50fbbcdd09aa3e6624098c64"}
Sep 10 19:26:37.681969 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:37.681778    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-70a2c9-openshift-default-68d7cc875b-m84w5"
Sep 10 19:26:37.683740 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:37.683711    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-70a2c9-openshift-default-68d7cc875b-m84w5"
Sep 10 19:26:37.901506 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:37.901466    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c4487c09-openshift-default-7c76fd77dd-8j85g"
Sep 10 19:26:37.901730 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:37.901706    2546 patch_prober.go:28] interesting pod/e2e-mt-c4487c09-openshift-default-7c76fd77dd-8j85g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Sep 10 19:26:37.901783 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:37.901759    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c4487c09-openshift-default-7c76fd77dd-8j85g" podUID="2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Sep 10 19:26:38.901698 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:38.901658    2546 patch_prober.go:28] interesting pod/e2e-mt-c4487c09-openshift-default-7c76fd77dd-8j85g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Sep 10 19:26:38.902101 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:38.901739    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c4487c09-openshift-default-7c76fd77dd-8j85g" podUID="2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Sep 10 19:26:39.902002 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:39.901960    2546 patch_prober.go:28] interesting pod/e2e-mt-c4487c09-openshift-default-7c76fd77dd-8j85g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Sep 10 19:26:39.902554 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:39.902029    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c4487c09-openshift-default-7c76fd77dd-8j85g" podUID="2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Sep 10 19:26:40.901842 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:40.901805    2546 patch_prober.go:28] interesting pod/e2e-mt-c4487c09-openshift-default-7c76fd77dd-8j85g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Sep 10 19:26:40.902025 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:40.901878    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c4487c09-openshift-default-7c76fd77dd-8j85g" podUID="2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Sep 10 19:26:41.901140 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:41.901101    2546 patch_prober.go:28] interesting pod/e2e-mt-c4487c09-openshift-default-7c76fd77dd-8j85g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Sep 10 19:26:41.901323 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:41.901163    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c4487c09-openshift-default-7c76fd77dd-8j85g" podUID="2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Sep 10 19:26:42.905652 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:42.905618    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c4487c09-openshift-default-7c76fd77dd-8j85g"
Sep 10 19:26:42.906122 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:42.906006    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c4487c09-openshift-default-7c76fd77dd-8j85g"
Sep 10 19:26:42.906684 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:42.906664    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c4487c09-openshift-default-7c76fd77dd-8j85g"
Sep 10 19:26:50.529256 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:50.529218    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-88n8f"]
Sep 10 19:26:56.943017 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:56.942979    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-70a2c9-openshift-default-68d7cc875b-m84w5"]
Sep 10 19:26:56.943573 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:26:56.943305    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-70a2c9-openshift-default-68d7cc875b-m84w5" podUID="8a9ed50b-ee46-40aa-a4d2-22c964b7f129" containerName="istio-proxy" containerID="cri-o://319c986bdb9b7f003dcace7abd856c82c8232bc15d22809e681a64d0e522d5a1" gracePeriod=30
Sep 10 19:27:02.199305 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:02.199280    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-70a2c9-openshift-default-68d7cc875b-m84w5"
Sep 10 19:27:02.251005 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:02.250932    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j7vmj\" (UniqueName: \"kubernetes.io/projected/8a9ed50b-ee46-40aa-a4d2-22c964b7f129-kube-api-access-j7vmj\") pod \"8a9ed50b-ee46-40aa-a4d2-22c964b7f129\" (UID: \"8a9ed50b-ee46-40aa-a4d2-22c964b7f129\") "
Sep 10 19:27:02.251005 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:02.250980    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a9ed50b-ee46-40aa-a4d2-22c964b7f129-workload-certs\") pod \"8a9ed50b-ee46-40aa-a4d2-22c964b7f129\" (UID: \"8a9ed50b-ee46-40aa-a4d2-22c964b7f129\") "
Sep 10 19:27:02.251005 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:02.250998    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a9ed50b-ee46-40aa-a4d2-22c964b7f129-istio-podinfo\") pod \"8a9ed50b-ee46-40aa-a4d2-22c964b7f129\" (UID: \"8a9ed50b-ee46-40aa-a4d2-22c964b7f129\") "
Sep 10 19:27:02.251276 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:02.251025    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a9ed50b-ee46-40aa-a4d2-22c964b7f129-credential-socket\") pod \"8a9ed50b-ee46-40aa-a4d2-22c964b7f129\" (UID: \"8a9ed50b-ee46-40aa-a4d2-22c964b7f129\") "
Sep 10 19:27:02.251276 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:02.251062    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a9ed50b-ee46-40aa-a4d2-22c964b7f129-istiod-ca-cert\") pod \"8a9ed50b-ee46-40aa-a4d2-22c964b7f129\" (UID: \"8a9ed50b-ee46-40aa-a4d2-22c964b7f129\") "
Sep 10 19:27:02.251276 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:02.251119    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a9ed50b-ee46-40aa-a4d2-22c964b7f129-workload-socket\") pod \"8a9ed50b-ee46-40aa-a4d2-22c964b7f129\" (UID: \"8a9ed50b-ee46-40aa-a4d2-22c964b7f129\") "
Sep 10 19:27:02.251276 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:02.251156    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a9ed50b-ee46-40aa-a4d2-22c964b7f129-istio-data\") pod \"8a9ed50b-ee46-40aa-a4d2-22c964b7f129\" (UID: \"8a9ed50b-ee46-40aa-a4d2-22c964b7f129\") "
Sep 10 19:27:02.251276 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:02.251197    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a9ed50b-ee46-40aa-a4d2-22c964b7f129-istio-envoy\") pod \"8a9ed50b-ee46-40aa-a4d2-22c964b7f129\" (UID: \"8a9ed50b-ee46-40aa-a4d2-22c964b7f129\") "
Sep 10 19:27:02.251276 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:02.251220    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a9ed50b-ee46-40aa-a4d2-22c964b7f129-istio-token\") pod \"8a9ed50b-ee46-40aa-a4d2-22c964b7f129\" (UID: \"8a9ed50b-ee46-40aa-a4d2-22c964b7f129\") "
Sep 10 19:27:02.251609 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:02.251277    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a9ed50b-ee46-40aa-a4d2-22c964b7f129-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8a9ed50b-ee46-40aa-a4d2-22c964b7f129" (UID: "8a9ed50b-ee46-40aa-a4d2-22c964b7f129"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:27:02.251609 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:02.251460    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a9ed50b-ee46-40aa-a4d2-22c964b7f129-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8a9ed50b-ee46-40aa-a4d2-22c964b7f129" (UID: "8a9ed50b-ee46-40aa-a4d2-22c964b7f129"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:27:02.251609 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:02.251523    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a9ed50b-ee46-40aa-a4d2-22c964b7f129-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8a9ed50b-ee46-40aa-a4d2-22c964b7f129" (UID: "8a9ed50b-ee46-40aa-a4d2-22c964b7f129"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:27:02.251779 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:02.251626    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a9ed50b-ee46-40aa-a4d2-22c964b7f129-istio-data" (OuterVolumeSpecName: "istio-data") pod "8a9ed50b-ee46-40aa-a4d2-22c964b7f129" (UID: "8a9ed50b-ee46-40aa-a4d2-22c964b7f129"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:27:02.251846 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:02.251815    2546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a9ed50b-ee46-40aa-a4d2-22c964b7f129-workload-socket\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:27:02.251899 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:02.251870    2546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a9ed50b-ee46-40aa-a4d2-22c964b7f129-istio-data\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:27:02.251899 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:02.251886    2546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a9ed50b-ee46-40aa-a4d2-22c964b7f129-workload-certs\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:27:02.251997 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:02.251898    2546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a9ed50b-ee46-40aa-a4d2-22c964b7f129-credential-socket\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:27:02.251997 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:02.251984    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a9ed50b-ee46-40aa-a4d2-22c964b7f129-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8a9ed50b-ee46-40aa-a4d2-22c964b7f129" (UID: "8a9ed50b-ee46-40aa-a4d2-22c964b7f129"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 19:27:02.253427 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:02.253393    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a9ed50b-ee46-40aa-a4d2-22c964b7f129-kube-api-access-j7vmj" (OuterVolumeSpecName: "kube-api-access-j7vmj") pod "8a9ed50b-ee46-40aa-a4d2-22c964b7f129" (UID: "8a9ed50b-ee46-40aa-a4d2-22c964b7f129"). InnerVolumeSpecName "kube-api-access-j7vmj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 19:27:02.253564 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:02.253516    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8a9ed50b-ee46-40aa-a4d2-22c964b7f129-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8a9ed50b-ee46-40aa-a4d2-22c964b7f129" (UID: "8a9ed50b-ee46-40aa-a4d2-22c964b7f129"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 19:27:02.253770 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:02.253756    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a9ed50b-ee46-40aa-a4d2-22c964b7f129-istio-token" (OuterVolumeSpecName: "istio-token") pod "8a9ed50b-ee46-40aa-a4d2-22c964b7f129" (UID: "8a9ed50b-ee46-40aa-a4d2-22c964b7f129"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 19:27:02.253866 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:02.253854    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a9ed50b-ee46-40aa-a4d2-22c964b7f129-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8a9ed50b-ee46-40aa-a4d2-22c964b7f129" (UID: "8a9ed50b-ee46-40aa-a4d2-22c964b7f129"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:27:02.352765 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:02.352728    2546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a9ed50b-ee46-40aa-a4d2-22c964b7f129-istio-envoy\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:27:02.352765 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:02.352759    2546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a9ed50b-ee46-40aa-a4d2-22c964b7f129-istio-token\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:27:02.352765 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:02.352769    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7vmj\" (UniqueName: \"kubernetes.io/projected/8a9ed50b-ee46-40aa-a4d2-22c964b7f129-kube-api-access-j7vmj\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:27:02.353006 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:02.352778    2546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a9ed50b-ee46-40aa-a4d2-22c964b7f129-istio-podinfo\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:27:02.353006 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:02.352786    2546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a9ed50b-ee46-40aa-a4d2-22c964b7f129-istiod-ca-cert\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:27:02.811317 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:02.811283    2546 generic.go:356] "Generic (PLEG): container finished" podID="8a9ed50b-ee46-40aa-a4d2-22c964b7f129" containerID="319c986bdb9b7f003dcace7abd856c82c8232bc15d22809e681a64d0e522d5a1" exitCode=0
Sep 10 19:27:02.811511 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:02.811370    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-70a2c9-openshift-default-68d7cc875b-m84w5"
Sep 10 19:27:02.811511 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:02.811369    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-70a2c9-openshift-default-68d7cc875b-m84w5" event={"ID":"8a9ed50b-ee46-40aa-a4d2-22c964b7f129","Type":"ContainerDied","Data":"319c986bdb9b7f003dcace7abd856c82c8232bc15d22809e681a64d0e522d5a1"}
Sep 10 19:27:02.811588 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:02.811515    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-70a2c9-openshift-default-68d7cc875b-m84w5" event={"ID":"8a9ed50b-ee46-40aa-a4d2-22c964b7f129","Type":"ContainerDied","Data":"3cedb5c437e9b9e030b8f132a8e0b1feaae666655548608d902e47fe31986910"}
Sep 10 19:27:02.811588 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:02.811543    2546 scope.go:117] "RemoveContainer" containerID="319c986bdb9b7f003dcace7abd856c82c8232bc15d22809e681a64d0e522d5a1"
Sep 10 19:27:02.820920 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:02.820902    2546 scope.go:117] "RemoveContainer" containerID="319c986bdb9b7f003dcace7abd856c82c8232bc15d22809e681a64d0e522d5a1"
Sep 10 19:27:02.821186 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:27:02.821166    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"319c986bdb9b7f003dcace7abd856c82c8232bc15d22809e681a64d0e522d5a1\": container with ID starting with 319c986bdb9b7f003dcace7abd856c82c8232bc15d22809e681a64d0e522d5a1 not found: ID does not exist" containerID="319c986bdb9b7f003dcace7abd856c82c8232bc15d22809e681a64d0e522d5a1"
Sep 10 19:27:02.821232 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:02.821194    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"319c986bdb9b7f003dcace7abd856c82c8232bc15d22809e681a64d0e522d5a1"} err="failed to get container status \"319c986bdb9b7f003dcace7abd856c82c8232bc15d22809e681a64d0e522d5a1\": rpc error: code = NotFound desc = could not find container \"319c986bdb9b7f003dcace7abd856c82c8232bc15d22809e681a64d0e522d5a1\": container with ID starting with 319c986bdb9b7f003dcace7abd856c82c8232bc15d22809e681a64d0e522d5a1 not found: ID does not exist"
Sep 10 19:27:02.836559 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:02.836528    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-70a2c9-openshift-default-68d7cc875b-m84w5"]
Sep 10 19:27:02.847400 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:02.847375    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-70a2c9-openshift-default-68d7cc875b-m84w5"]
Sep 10 19:27:03.281530 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:03.281503    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a9ed50b-ee46-40aa-a4d2-22c964b7f129" path="/var/lib/kubelet/pods/8a9ed50b-ee46-40aa-a4d2-22c964b7f129/volumes"
Sep 10 19:27:08.152380 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:08.152329    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-88n8f"]
Sep 10 19:27:09.848016 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:09.847987    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-6b562c/test-model-6b562c-kserve-56cd4564d9-b5vr6"]
Sep 10 19:27:12.543860 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:12.543825    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c4487c09-openshift-default-7c76fd77dd-8j85g"]
Sep 10 19:27:12.544283 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:12.544030    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-c4487c09-openshift-default-7c76fd77dd-8j85g" podUID="2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc" containerName="istio-proxy" containerID="cri-o://40c65d37725bfc98647f738e3391eddd0b06483d50fbbcdd09aa3e6624098c64" gracePeriod=30
Sep 10 19:27:12.908447 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:12.908409    2546 patch_prober.go:28] interesting pod/e2e-mt-c4487c09-openshift-default-7c76fd77dd-8j85g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused
Sep 10 19:27:12.908635 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:12.908476    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-c4487c09-openshift-default-7c76fd77dd-8j85g" podUID="2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 10 19:27:17.795676 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:17.795650    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c4487c09-openshift-default-7c76fd77dd-8j85g"
Sep 10 19:27:17.886177 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:17.886140    2546 generic.go:356] "Generic (PLEG): container finished" podID="2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc" containerID="40c65d37725bfc98647f738e3391eddd0b06483d50fbbcdd09aa3e6624098c64" exitCode=0
Sep 10 19:27:17.886364 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:17.886193    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c4487c09-openshift-default-7c76fd77dd-8j85g"
Sep 10 19:27:17.886364 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:17.886225    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c4487c09-openshift-default-7c76fd77dd-8j85g" event={"ID":"2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc","Type":"ContainerDied","Data":"40c65d37725bfc98647f738e3391eddd0b06483d50fbbcdd09aa3e6624098c64"}
Sep 10 19:27:17.886364 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:17.886279    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c4487c09-openshift-default-7c76fd77dd-8j85g" event={"ID":"2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc","Type":"ContainerDied","Data":"6a8f91d89b5aca53f6479447ca9f7befe6e4fe3b07c544f7ea4aa82974812be5"}
Sep 10 19:27:17.886364 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:17.886301    2546 scope.go:117] "RemoveContainer" containerID="40c65d37725bfc98647f738e3391eddd0b06483d50fbbcdd09aa3e6624098c64"
Sep 10 19:27:17.894781 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:17.894764    2546 scope.go:117] "RemoveContainer" containerID="3a0b56aac008ccaec717cb3177f20a7a7fbc8222706bad336b23ac9efe9f65ed"
Sep 10 19:27:17.901012 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:17.900991    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc-credential-socket\") pod \"2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc\" (UID: \"2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc\") "
Sep 10 19:27:17.901131 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:17.901022    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc-istio-podinfo\") pod \"2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc\" (UID: \"2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc\") "
Sep 10 19:27:17.901131 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:17.901055    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc-istio-token\") pod \"2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc\" (UID: \"2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc\") "
Sep 10 19:27:17.901131 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:17.901077    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc-workload-certs\") pod \"2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc\" (UID: \"2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc\") "
Sep 10 19:27:17.901296 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:17.901197    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc-istiod-ca-cert\") pod \"2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc\" (UID: \"2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc\") "
Sep 10 19:27:17.901296 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:17.901284    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc-istio-data\") pod \"2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc\" (UID: \"2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc\") "
Sep 10 19:27:17.901423 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:17.901303    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc" (UID: "2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:27:17.901423 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:17.901320    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc" (UID: "2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:27:17.901423 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:17.901312    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc-istio-envoy\") pod \"2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc\" (UID: \"2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc\") "
Sep 10 19:27:17.901423 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:17.901395    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc-workload-socket\") pod \"2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc\" (UID: \"2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc\") "
Sep 10 19:27:17.901629 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:17.901498    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l7fc7\" (UniqueName: \"kubernetes.io/projected/2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc-kube-api-access-l7fc7\") pod \"2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc\" (UID: \"2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc\") "
Sep 10 19:27:17.901629 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:17.901569    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc-istio-data" (OuterVolumeSpecName: "istio-data") pod "2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc" (UID: "2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:27:17.901629 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:17.901575    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc" (UID: "2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 19:27:17.901839 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:17.901799    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc" (UID: "2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:27:17.901937 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:17.901924    2546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc-workload-certs\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:27:17.901999 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:17.901942    2546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc-istiod-ca-cert\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:27:17.901999 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:17.901977    2546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc-istio-data\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:27:17.901999 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:17.901990    2546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc-workload-socket\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:27:17.902152 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:17.902002    2546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc-credential-socket\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:27:17.903592 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:17.903569    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc" (UID: "2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:27:17.903674 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:17.903608    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc-kube-api-access-l7fc7" (OuterVolumeSpecName: "kube-api-access-l7fc7") pod "2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc" (UID: "2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc"). InnerVolumeSpecName "kube-api-access-l7fc7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 19:27:17.903674 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:17.903652    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc" (UID: "2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 19:27:17.903777 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:17.903658    2546 scope.go:117] "RemoveContainer" containerID="40c65d37725bfc98647f738e3391eddd0b06483d50fbbcdd09aa3e6624098c64"
Sep 10 19:27:17.903777 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:17.903698    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc-istio-token" (OuterVolumeSpecName: "istio-token") pod "2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc" (UID: "2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 19:27:17.904002 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:27:17.903985    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40c65d37725bfc98647f738e3391eddd0b06483d50fbbcdd09aa3e6624098c64\": container with ID starting with 40c65d37725bfc98647f738e3391eddd0b06483d50fbbcdd09aa3e6624098c64 not found: ID does not exist" containerID="40c65d37725bfc98647f738e3391eddd0b06483d50fbbcdd09aa3e6624098c64"
Sep 10 19:27:17.904042 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:17.904010    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40c65d37725bfc98647f738e3391eddd0b06483d50fbbcdd09aa3e6624098c64"} err="failed to get container status \"40c65d37725bfc98647f738e3391eddd0b06483d50fbbcdd09aa3e6624098c64\": rpc error: code = NotFound desc = could not find container \"40c65d37725bfc98647f738e3391eddd0b06483d50fbbcdd09aa3e6624098c64\": container with ID starting with 40c65d37725bfc98647f738e3391eddd0b06483d50fbbcdd09aa3e6624098c64 not found: ID does not exist"
Sep 10 19:27:17.904042 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:17.904030    2546 scope.go:117] "RemoveContainer" containerID="3a0b56aac008ccaec717cb3177f20a7a7fbc8222706bad336b23ac9efe9f65ed"
Sep 10 19:27:17.904320 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:27:17.904300    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a0b56aac008ccaec717cb3177f20a7a7fbc8222706bad336b23ac9efe9f65ed\": container with ID starting with 3a0b56aac008ccaec717cb3177f20a7a7fbc8222706bad336b23ac9efe9f65ed not found: ID does not exist" containerID="3a0b56aac008ccaec717cb3177f20a7a7fbc8222706bad336b23ac9efe9f65ed"
Sep 10 19:27:17.904401 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:17.904330    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a0b56aac008ccaec717cb3177f20a7a7fbc8222706bad336b23ac9efe9f65ed"} err="failed to get container status \"3a0b56aac008ccaec717cb3177f20a7a7fbc8222706bad336b23ac9efe9f65ed\": rpc error: code = NotFound desc = could not find container \"3a0b56aac008ccaec717cb3177f20a7a7fbc8222706bad336b23ac9efe9f65ed\": container with ID starting with 3a0b56aac008ccaec717cb3177f20a7a7fbc8222706bad336b23ac9efe9f65ed not found: ID does not exist"
Sep 10 19:27:18.002645 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:18.002604    2546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc-istio-podinfo\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:27:18.002645 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:18.002637    2546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc-istio-token\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:27:18.002645 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:18.002646    2546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc-istio-envoy\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:27:18.002645 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:18.002655    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7fc7\" (UniqueName: \"kubernetes.io/projected/2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc-kube-api-access-l7fc7\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:27:18.216235 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:18.211942    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c4487c09-openshift-default-7c76fd77dd-8j85g"]
Sep 10 19:27:18.218533 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:18.218507    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-c4487c09-openshift-default-7c76fd77dd-8j85g"]
Sep 10 19:27:18.756374 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:18.756322    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-88n8f"]
Sep 10 19:27:19.282251 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:19.282220    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc" path="/var/lib/kubelet/pods/2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc/volumes"
Sep 10 19:27:21.814316 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:21.814280    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6b562c-774f4867fc-bmk4k"]
Sep 10 19:27:21.814818 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:21.814531    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6b562c-774f4867fc-bmk4k" podUID="f7e6dc57-f6fe-408b-bf57-0bf15512ade4" containerName="maas-api" containerID="cri-o://46aef71e4b6cefc22010f9f9100d4aaf0e68245514c884fa83f8ce6e70082417" gracePeriod=30
Sep 10 19:27:22.081115 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:22.081087    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6b562c-774f4867fc-bmk4k"
Sep 10 19:27:22.143056 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:22.143003    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n77xr\" (UniqueName: \"kubernetes.io/projected/f7e6dc57-f6fe-408b-bf57-0bf15512ade4-kube-api-access-n77xr\") pod \"f7e6dc57-f6fe-408b-bf57-0bf15512ade4\" (UID: \"f7e6dc57-f6fe-408b-bf57-0bf15512ade4\") "
Sep 10 19:27:22.143233 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:22.143214    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7e6dc57-f6fe-408b-bf57-0bf15512ade4-maas-api-tls\") pod \"f7e6dc57-f6fe-408b-bf57-0bf15512ade4\" (UID: \"f7e6dc57-f6fe-408b-bf57-0bf15512ade4\") "
Sep 10 19:27:22.143272 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:22.143253    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7e6dc57-f6fe-408b-bf57-0bf15512ade4-metrics-certs\") pod \"f7e6dc57-f6fe-408b-bf57-0bf15512ade4\" (UID: \"f7e6dc57-f6fe-408b-bf57-0bf15512ade4\") "
Sep 10 19:27:22.143310 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:22.143296    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7e6dc57-f6fe-408b-bf57-0bf15512ade4-openshift-service-ca\") pod \"f7e6dc57-f6fe-408b-bf57-0bf15512ade4\" (UID: \"f7e6dc57-f6fe-408b-bf57-0bf15512ade4\") "
Sep 10 19:27:22.143723 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:22.143689    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7e6dc57-f6fe-408b-bf57-0bf15512ade4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f7e6dc57-f6fe-408b-bf57-0bf15512ade4" (UID: "f7e6dc57-f6fe-408b-bf57-0bf15512ade4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 19:27:22.145295 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:22.145270    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7e6dc57-f6fe-408b-bf57-0bf15512ade4-kube-api-access-n77xr" (OuterVolumeSpecName: "kube-api-access-n77xr") pod "f7e6dc57-f6fe-408b-bf57-0bf15512ade4" (UID: "f7e6dc57-f6fe-408b-bf57-0bf15512ade4"). InnerVolumeSpecName "kube-api-access-n77xr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 19:27:22.145295 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:22.145283    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7e6dc57-f6fe-408b-bf57-0bf15512ade4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f7e6dc57-f6fe-408b-bf57-0bf15512ade4" (UID: "f7e6dc57-f6fe-408b-bf57-0bf15512ade4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 19:27:22.145295 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:22.145268    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7e6dc57-f6fe-408b-bf57-0bf15512ade4-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "f7e6dc57-f6fe-408b-bf57-0bf15512ade4" (UID: "f7e6dc57-f6fe-408b-bf57-0bf15512ade4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 19:27:22.244377 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:22.244325    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n77xr\" (UniqueName: \"kubernetes.io/projected/f7e6dc57-f6fe-408b-bf57-0bf15512ade4-kube-api-access-n77xr\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:27:22.244377 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:22.244378    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7e6dc57-f6fe-408b-bf57-0bf15512ade4-maas-api-tls\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:27:22.244631 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:22.244393    2546 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7e6dc57-f6fe-408b-bf57-0bf15512ade4-metrics-certs\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:27:22.244631 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:22.244406    2546 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7e6dc57-f6fe-408b-bf57-0bf15512ade4-openshift-service-ca\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:27:22.915334 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:22.915295    2546 generic.go:356] "Generic (PLEG): container finished" podID="f7e6dc57-f6fe-408b-bf57-0bf15512ade4" containerID="46aef71e4b6cefc22010f9f9100d4aaf0e68245514c884fa83f8ce6e70082417" exitCode=0
Sep 10 19:27:22.915334 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:22.915334    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6b562c-774f4867fc-bmk4k" event={"ID":"f7e6dc57-f6fe-408b-bf57-0bf15512ade4","Type":"ContainerDied","Data":"46aef71e4b6cefc22010f9f9100d4aaf0e68245514c884fa83f8ce6e70082417"}
Sep 10 19:27:22.915897 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:22.915367    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6b562c-774f4867fc-bmk4k"
Sep 10 19:27:22.915897 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:22.915405    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6b562c-774f4867fc-bmk4k" event={"ID":"f7e6dc57-f6fe-408b-bf57-0bf15512ade4","Type":"ContainerDied","Data":"9ae313e6a32a05a915c11633534a8cc8d6d1e8a0a7fb7243a5ab5ec22360be92"}
Sep 10 19:27:22.915897 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:22.915428    2546 scope.go:117] "RemoveContainer" containerID="46aef71e4b6cefc22010f9f9100d4aaf0e68245514c884fa83f8ce6e70082417"
Sep 10 19:27:22.925033 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:22.925010    2546 scope.go:117] "RemoveContainer" containerID="46aef71e4b6cefc22010f9f9100d4aaf0e68245514c884fa83f8ce6e70082417"
Sep 10 19:27:22.925378 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:27:22.925356    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46aef71e4b6cefc22010f9f9100d4aaf0e68245514c884fa83f8ce6e70082417\": container with ID starting with 46aef71e4b6cefc22010f9f9100d4aaf0e68245514c884fa83f8ce6e70082417 not found: ID does not exist" containerID="46aef71e4b6cefc22010f9f9100d4aaf0e68245514c884fa83f8ce6e70082417"
Sep 10 19:27:22.925467 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:22.925385    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46aef71e4b6cefc22010f9f9100d4aaf0e68245514c884fa83f8ce6e70082417"} err="failed to get container status \"46aef71e4b6cefc22010f9f9100d4aaf0e68245514c884fa83f8ce6e70082417\": rpc error: code = NotFound desc = could not find container \"46aef71e4b6cefc22010f9f9100d4aaf0e68245514c884fa83f8ce6e70082417\": container with ID starting with 46aef71e4b6cefc22010f9f9100d4aaf0e68245514c884fa83f8ce6e70082417 not found: ID does not exist"
Sep 10 19:27:22.956577 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:22.956546    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6b562c-774f4867fc-bmk4k"]
Sep 10 19:27:22.965999 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:22.965969    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6b562c-774f4867fc-bmk4k"]
Sep 10 19:27:23.281543 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:23.281456    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7e6dc57-f6fe-408b-bf57-0bf15512ade4" path="/var/lib/kubelet/pods/f7e6dc57-f6fe-408b-bf57-0bf15512ade4/volumes"
Sep 10 19:27:24.871036 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:24.870973    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-6b562c/test-model-6b562c-kserve-56cd4564d9-b5vr6" podUID="24e6afb8-845c-4c89-8cea-5236276c5725" containerName="main" containerID="cri-o://174c86bdb0c6085e487d1fd4d130f1a19054be8e43b2c93ae34cd3d5cd8656a1" gracePeriod=45
Sep 10 19:27:25.129113 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:25.129046    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-6b562c/test-model-6b562c-kserve-56cd4564d9-b5vr6"
Sep 10 19:27:25.273831 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:25.273787    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24e6afb8-845c-4c89-8cea-5236276c5725-tmp-dir\") pod \"24e6afb8-845c-4c89-8cea-5236276c5725\" (UID: \"24e6afb8-845c-4c89-8cea-5236276c5725\") "
Sep 10 19:27:25.273831 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:25.273841    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-45mm8\" (UniqueName: \"kubernetes.io/projected/24e6afb8-845c-4c89-8cea-5236276c5725-kube-api-access-45mm8\") pod \"24e6afb8-845c-4c89-8cea-5236276c5725\" (UID: \"24e6afb8-845c-4c89-8cea-5236276c5725\") "
Sep 10 19:27:25.274065 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:25.273883    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24e6afb8-845c-4c89-8cea-5236276c5725-model-cache\") pod \"24e6afb8-845c-4c89-8cea-5236276c5725\" (UID: \"24e6afb8-845c-4c89-8cea-5236276c5725\") "
Sep 10 19:27:25.274065 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:25.273923    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24e6afb8-845c-4c89-8cea-5236276c5725-tls-certs\") pod \"24e6afb8-845c-4c89-8cea-5236276c5725\" (UID: \"24e6afb8-845c-4c89-8cea-5236276c5725\") "
Sep 10 19:27:25.274065 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:25.273955    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24e6afb8-845c-4c89-8cea-5236276c5725-home\") pod \"24e6afb8-845c-4c89-8cea-5236276c5725\" (UID: \"24e6afb8-845c-4c89-8cea-5236276c5725\") "
Sep 10 19:27:25.274065 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:25.273999    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24e6afb8-845c-4c89-8cea-5236276c5725-dshm\") pod \"24e6afb8-845c-4c89-8cea-5236276c5725\" (UID: \"24e6afb8-845c-4c89-8cea-5236276c5725\") "
Sep 10 19:27:25.274280 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:25.274147    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24e6afb8-845c-4c89-8cea-5236276c5725-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "24e6afb8-845c-4c89-8cea-5236276c5725" (UID: "24e6afb8-845c-4c89-8cea-5236276c5725"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:27:25.274280 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:25.274241    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24e6afb8-845c-4c89-8cea-5236276c5725-model-cache" (OuterVolumeSpecName: "model-cache") pod "24e6afb8-845c-4c89-8cea-5236276c5725" (UID: "24e6afb8-845c-4c89-8cea-5236276c5725"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:27:25.274422 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:25.274314    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24e6afb8-845c-4c89-8cea-5236276c5725-home" (OuterVolumeSpecName: "home") pod "24e6afb8-845c-4c89-8cea-5236276c5725" (UID: "24e6afb8-845c-4c89-8cea-5236276c5725"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:27:25.274422 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:25.274325    2546 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24e6afb8-845c-4c89-8cea-5236276c5725-tmp-dir\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:27:25.276394 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:25.276366    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24e6afb8-845c-4c89-8cea-5236276c5725-dshm" (OuterVolumeSpecName: "dshm") pod "24e6afb8-845c-4c89-8cea-5236276c5725" (UID: "24e6afb8-845c-4c89-8cea-5236276c5725"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:27:25.276744 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:25.276723    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24e6afb8-845c-4c89-8cea-5236276c5725-kube-api-access-45mm8" (OuterVolumeSpecName: "kube-api-access-45mm8") pod "24e6afb8-845c-4c89-8cea-5236276c5725" (UID: "24e6afb8-845c-4c89-8cea-5236276c5725"). InnerVolumeSpecName "kube-api-access-45mm8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 19:27:25.276834 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:25.276726    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24e6afb8-845c-4c89-8cea-5236276c5725-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "24e6afb8-845c-4c89-8cea-5236276c5725" (UID: "24e6afb8-845c-4c89-8cea-5236276c5725"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 19:27:25.374940 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:25.374900    2546 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24e6afb8-845c-4c89-8cea-5236276c5725-home\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:27:25.374940 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:25.374936    2546 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24e6afb8-845c-4c89-8cea-5236276c5725-dshm\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:27:25.375206 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:25.374950    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45mm8\" (UniqueName: \"kubernetes.io/projected/24e6afb8-845c-4c89-8cea-5236276c5725-kube-api-access-45mm8\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:27:25.375206 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:25.374965    2546 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24e6afb8-845c-4c89-8cea-5236276c5725-model-cache\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:27:25.375206 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:25.374979    2546 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24e6afb8-845c-4c89-8cea-5236276c5725-tls-certs\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:27:25.937733 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:25.937698    2546 generic.go:356] "Generic (PLEG): container finished" podID="24e6afb8-845c-4c89-8cea-5236276c5725" containerID="174c86bdb0c6085e487d1fd4d130f1a19054be8e43b2c93ae34cd3d5cd8656a1" exitCode=0
Sep 10 19:27:25.938208 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:25.937748    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-6b562c/test-model-6b562c-kserve-56cd4564d9-b5vr6"
Sep 10 19:27:25.938208 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:25.937789    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-6b562c/test-model-6b562c-kserve-56cd4564d9-b5vr6" event={"ID":"24e6afb8-845c-4c89-8cea-5236276c5725","Type":"ContainerDied","Data":"174c86bdb0c6085e487d1fd4d130f1a19054be8e43b2c93ae34cd3d5cd8656a1"}
Sep 10 19:27:25.938208 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:25.937834    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-6b562c/test-model-6b562c-kserve-56cd4564d9-b5vr6" event={"ID":"24e6afb8-845c-4c89-8cea-5236276c5725","Type":"ContainerDied","Data":"f283e0eee5ada272fb2f18c2ed9b8727fa7fd671d86c8b4d6321622e114a620b"}
Sep 10 19:27:25.938208 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:25.937860    2546 scope.go:117] "RemoveContainer" containerID="174c86bdb0c6085e487d1fd4d130f1a19054be8e43b2c93ae34cd3d5cd8656a1"
Sep 10 19:27:25.948471 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:25.948447    2546 scope.go:117] "RemoveContainer" containerID="174c86bdb0c6085e487d1fd4d130f1a19054be8e43b2c93ae34cd3d5cd8656a1"
Sep 10 19:27:25.948853 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:27:25.948831    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"174c86bdb0c6085e487d1fd4d130f1a19054be8e43b2c93ae34cd3d5cd8656a1\": container with ID starting with 174c86bdb0c6085e487d1fd4d130f1a19054be8e43b2c93ae34cd3d5cd8656a1 not found: ID does not exist" containerID="174c86bdb0c6085e487d1fd4d130f1a19054be8e43b2c93ae34cd3d5cd8656a1"
Sep 10 19:27:25.948935 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:25.948861    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"174c86bdb0c6085e487d1fd4d130f1a19054be8e43b2c93ae34cd3d5cd8656a1"} err="failed to get container status \"174c86bdb0c6085e487d1fd4d130f1a19054be8e43b2c93ae34cd3d5cd8656a1\": rpc error: code = NotFound desc = could not find container \"174c86bdb0c6085e487d1fd4d130f1a19054be8e43b2c93ae34cd3d5cd8656a1\": container with ID starting with 174c86bdb0c6085e487d1fd4d130f1a19054be8e43b2c93ae34cd3d5cd8656a1 not found: ID does not exist"
Sep 10 19:27:25.955758 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:25.955731    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-6b562c/test-model-6b562c-kserve-56cd4564d9-b5vr6"]
Sep 10 19:27:25.963331 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:25.963301    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-6b562c/test-model-6b562c-kserve-56cd4564d9-b5vr6"]
Sep 10 19:27:26.120150 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:26.120113    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-88n8f"]
Sep 10 19:27:27.282245 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:27.282207    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24e6afb8-845c-4c89-8cea-5236276c5725" path="/var/lib/kubelet/pods/24e6afb8-845c-4c89-8cea-5236276c5725/volumes"
Sep 10 19:27:27.398314 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:27.398283    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-cfa578-openshift-default-5d84f8dfd6-b8f6s"]
Sep 10 19:27:27.398786 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:27.398749    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc" containerName="istio-proxy"
Sep 10 19:27:27.398786 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:27.398772    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc" containerName="istio-proxy"
Sep 10 19:27:27.398969 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:27.398796    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a9ed50b-ee46-40aa-a4d2-22c964b7f129" containerName="istio-proxy"
Sep 10 19:27:27.398969 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:27.398806    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a9ed50b-ee46-40aa-a4d2-22c964b7f129" containerName="istio-proxy"
Sep 10 19:27:27.398969 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:27.398816    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24e6afb8-845c-4c89-8cea-5236276c5725" containerName="main"
Sep 10 19:27:27.398969 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:27.398822    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="24e6afb8-845c-4c89-8cea-5236276c5725" containerName="main"
Sep 10 19:27:27.398969 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:27.398829    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc" containerName="istio-proxy"
Sep 10 19:27:27.398969 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:27.398835    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc" containerName="istio-proxy"
Sep 10 19:27:27.398969 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:27.398848    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7e6dc57-f6fe-408b-bf57-0bf15512ade4" containerName="maas-api"
Sep 10 19:27:27.398969 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:27.398856    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7e6dc57-f6fe-408b-bf57-0bf15512ade4" containerName="maas-api"
Sep 10 19:27:27.398969 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:27.398924    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a9ed50b-ee46-40aa-a4d2-22c964b7f129" containerName="istio-proxy"
Sep 10 19:27:27.398969 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:27.398935    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="24e6afb8-845c-4c89-8cea-5236276c5725" containerName="main"
Sep 10 19:27:27.398969 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:27.398945    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc" containerName="istio-proxy"
Sep 10 19:27:27.398969 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:27.398953    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e741ef0-d7d4-47f1-b41a-b3795a5b9fdc" containerName="istio-proxy"
Sep 10 19:27:27.398969 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:27.398964    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7e6dc57-f6fe-408b-bf57-0bf15512ade4" containerName="maas-api"
Sep 10 19:27:27.403565 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:27.403540    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-cfa578-openshift-default-5d84f8dfd6-b8f6s"
Sep 10 19:27:27.406639 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:27.406611    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-cfa578-openshift-default-dockercfg-zpdcs\""
Sep 10 19:27:27.415311 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:27.415282    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-cfa578-openshift-default-5d84f8dfd6-b8f6s"]
Sep 10 19:27:27.496832 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:27.496794    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f9d35e3-2d14-4a94-9e02-30760eeb631d-workload-certs\") pod \"e2e-shared-a-w2-cfa578-openshift-default-5d84f8dfd6-b8f6s\" (UID: \"2f9d35e3-2d14-4a94-9e02-30760eeb631d\") " pod="openshift-ingress/e2e-shared-a-w2-cfa578-openshift-default-5d84f8dfd6-b8f6s"
Sep 10 19:27:27.496832 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:27.496832    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f9d35e3-2d14-4a94-9e02-30760eeb631d-istio-token\") pod \"e2e-shared-a-w2-cfa578-openshift-default-5d84f8dfd6-b8f6s\" (UID: \"2f9d35e3-2d14-4a94-9e02-30760eeb631d\") " pod="openshift-ingress/e2e-shared-a-w2-cfa578-openshift-default-5d84f8dfd6-b8f6s"
Sep 10 19:27:27.497094 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:27.496902    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f9d35e3-2d14-4a94-9e02-30760eeb631d-istio-podinfo\") pod \"e2e-shared-a-w2-cfa578-openshift-default-5d84f8dfd6-b8f6s\" (UID: \"2f9d35e3-2d14-4a94-9e02-30760eeb631d\") " pod="openshift-ingress/e2e-shared-a-w2-cfa578-openshift-default-5d84f8dfd6-b8f6s"
Sep 10 19:27:27.497094 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:27.496926    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f9d35e3-2d14-4a94-9e02-30760eeb631d-workload-socket\") pod \"e2e-shared-a-w2-cfa578-openshift-default-5d84f8dfd6-b8f6s\" (UID: \"2f9d35e3-2d14-4a94-9e02-30760eeb631d\") " pod="openshift-ingress/e2e-shared-a-w2-cfa578-openshift-default-5d84f8dfd6-b8f6s"
Sep 10 19:27:27.497094 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:27.497014    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f9d35e3-2d14-4a94-9e02-30760eeb631d-credential-socket\") pod \"e2e-shared-a-w2-cfa578-openshift-default-5d84f8dfd6-b8f6s\" (UID: \"2f9d35e3-2d14-4a94-9e02-30760eeb631d\") " pod="openshift-ingress/e2e-shared-a-w2-cfa578-openshift-default-5d84f8dfd6-b8f6s"
Sep 10 19:27:27.497094 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:27.497047    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f9d35e3-2d14-4a94-9e02-30760eeb631d-istio-envoy\") pod \"e2e-shared-a-w2-cfa578-openshift-default-5d84f8dfd6-b8f6s\" (UID: \"2f9d35e3-2d14-4a94-9e02-30760eeb631d\") " pod="openshift-ingress/e2e-shared-a-w2-cfa578-openshift-default-5d84f8dfd6-b8f6s"
Sep 10 19:27:27.497268 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:27.497095    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qrkr\" (UniqueName: \"kubernetes.io/projected/2f9d35e3-2d14-4a94-9e02-30760eeb631d-kube-api-access-7qrkr\") pod \"e2e-shared-a-w2-cfa578-openshift-default-5d84f8dfd6-b8f6s\" (UID: \"2f9d35e3-2d14-4a94-9e02-30760eeb631d\") " pod="openshift-ingress/e2e-shared-a-w2-cfa578-openshift-default-5d84f8dfd6-b8f6s"
Sep 10 19:27:27.497268 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:27.497124    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f9d35e3-2d14-4a94-9e02-30760eeb631d-istiod-ca-cert\") pod \"e2e-shared-a-w2-cfa578-openshift-default-5d84f8dfd6-b8f6s\" (UID: \"2f9d35e3-2d14-4a94-9e02-30760eeb631d\") " pod="openshift-ingress/e2e-shared-a-w2-cfa578-openshift-default-5d84f8dfd6-b8f6s"
Sep 10 19:27:27.497268 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:27.497161    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f9d35e3-2d14-4a94-9e02-30760eeb631d-istio-data\") pod \"e2e-shared-a-w2-cfa578-openshift-default-5d84f8dfd6-b8f6s\" (UID: \"2f9d35e3-2d14-4a94-9e02-30760eeb631d\") " pod="openshift-ingress/e2e-shared-a-w2-cfa578-openshift-default-5d84f8dfd6-b8f6s"
Sep 10 19:27:27.598191 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:27.598089    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f9d35e3-2d14-4a94-9e02-30760eeb631d-istio-data\") pod \"e2e-shared-a-w2-cfa578-openshift-default-5d84f8dfd6-b8f6s\" (UID: \"2f9d35e3-2d14-4a94-9e02-30760eeb631d\") " pod="openshift-ingress/e2e-shared-a-w2-cfa578-openshift-default-5d84f8dfd6-b8f6s"
Sep 10 19:27:27.598191 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:27.598166    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f9d35e3-2d14-4a94-9e02-30760eeb631d-workload-certs\") pod \"e2e-shared-a-w2-cfa578-openshift-default-5d84f8dfd6-b8f6s\" (UID: \"2f9d35e3-2d14-4a94-9e02-30760eeb631d\") " pod="openshift-ingress/e2e-shared-a-w2-cfa578-openshift-default-5d84f8dfd6-b8f6s"
Sep 10 19:27:27.598191 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:27.598185    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f9d35e3-2d14-4a94-9e02-30760eeb631d-istio-token\") pod \"e2e-shared-a-w2-cfa578-openshift-default-5d84f8dfd6-b8f6s\" (UID: \"2f9d35e3-2d14-4a94-9e02-30760eeb631d\") " pod="openshift-ingress/e2e-shared-a-w2-cfa578-openshift-default-5d84f8dfd6-b8f6s"
Sep 10 19:27:27.598551 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:27.598221    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f9d35e3-2d14-4a94-9e02-30760eeb631d-istio-podinfo\") pod \"e2e-shared-a-w2-cfa578-openshift-default-5d84f8dfd6-b8f6s\" (UID: \"2f9d35e3-2d14-4a94-9e02-30760eeb631d\") " pod="openshift-ingress/e2e-shared-a-w2-cfa578-openshift-default-5d84f8dfd6-b8f6s"
Sep 10 19:27:27.598551 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:27.598241    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f9d35e3-2d14-4a94-9e02-30760eeb631d-workload-socket\") pod \"e2e-shared-a-w2-cfa578-openshift-default-5d84f8dfd6-b8f6s\" (UID: \"2f9d35e3-2d14-4a94-9e02-30760eeb631d\") " pod="openshift-ingress/e2e-shared-a-w2-cfa578-openshift-default-5d84f8dfd6-b8f6s"
Sep 10 19:27:27.598551 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:27.598266    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f9d35e3-2d14-4a94-9e02-30760eeb631d-credential-socket\") pod \"e2e-shared-a-w2-cfa578-openshift-default-5d84f8dfd6-b8f6s\" (UID: \"2f9d35e3-2d14-4a94-9e02-30760eeb631d\") " pod="openshift-ingress/e2e-shared-a-w2-cfa578-openshift-default-5d84f8dfd6-b8f6s"
Sep 10 19:27:27.598551 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:27.598292    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f9d35e3-2d14-4a94-9e02-30760eeb631d-istio-envoy\") pod \"e2e-shared-a-w2-cfa578-openshift-default-5d84f8dfd6-b8f6s\" (UID: \"2f9d35e3-2d14-4a94-9e02-30760eeb631d\") " pod="openshift-ingress/e2e-shared-a-w2-cfa578-openshift-default-5d84f8dfd6-b8f6s"
Sep 10 19:27:27.598551 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:27.598525    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qrkr\" (UniqueName: \"kubernetes.io/projected/2f9d35e3-2d14-4a94-9e02-30760eeb631d-kube-api-access-7qrkr\") pod \"e2e-shared-a-w2-cfa578-openshift-default-5d84f8dfd6-b8f6s\" (UID: \"2f9d35e3-2d14-4a94-9e02-30760eeb631d\") " pod="openshift-ingress/e2e-shared-a-w2-cfa578-openshift-default-5d84f8dfd6-b8f6s"
Sep 10 19:27:27.598843 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:27.598570    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f9d35e3-2d14-4a94-9e02-30760eeb631d-istiod-ca-cert\") pod \"e2e-shared-a-w2-cfa578-openshift-default-5d84f8dfd6-b8f6s\" (UID: \"2f9d35e3-2d14-4a94-9e02-30760eeb631d\") " pod="openshift-ingress/e2e-shared-a-w2-cfa578-openshift-default-5d84f8dfd6-b8f6s"
Sep 10 19:27:27.598843 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:27.598579    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f9d35e3-2d14-4a94-9e02-30760eeb631d-istio-data\") pod \"e2e-shared-a-w2-cfa578-openshift-default-5d84f8dfd6-b8f6s\" (UID: \"2f9d35e3-2d14-4a94-9e02-30760eeb631d\") " pod="openshift-ingress/e2e-shared-a-w2-cfa578-openshift-default-5d84f8dfd6-b8f6s"
Sep 10 19:27:27.598843 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:27.598687    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f9d35e3-2d14-4a94-9e02-30760eeb631d-workload-certs\") pod \"e2e-shared-a-w2-cfa578-openshift-default-5d84f8dfd6-b8f6s\" (UID: \"2f9d35e3-2d14-4a94-9e02-30760eeb631d\") " pod="openshift-ingress/e2e-shared-a-w2-cfa578-openshift-default-5d84f8dfd6-b8f6s"
Sep 10 19:27:27.598843 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:27.598724    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f9d35e3-2d14-4a94-9e02-30760eeb631d-credential-socket\") pod \"e2e-shared-a-w2-cfa578-openshift-default-5d84f8dfd6-b8f6s\" (UID: \"2f9d35e3-2d14-4a94-9e02-30760eeb631d\") " pod="openshift-ingress/e2e-shared-a-w2-cfa578-openshift-default-5d84f8dfd6-b8f6s"
Sep 10 19:27:27.598843 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:27.598787    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f9d35e3-2d14-4a94-9e02-30760eeb631d-workload-socket\") pod \"e2e-shared-a-w2-cfa578-openshift-default-5d84f8dfd6-b8f6s\" (UID: \"2f9d35e3-2d14-4a94-9e02-30760eeb631d\") " pod="openshift-ingress/e2e-shared-a-w2-cfa578-openshift-default-5d84f8dfd6-b8f6s"
Sep 10 19:27:27.599206 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:27.599189    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f9d35e3-2d14-4a94-9e02-30760eeb631d-istiod-ca-cert\") pod \"e2e-shared-a-w2-cfa578-openshift-default-5d84f8dfd6-b8f6s\" (UID: \"2f9d35e3-2d14-4a94-9e02-30760eeb631d\") " pod="openshift-ingress/e2e-shared-a-w2-cfa578-openshift-default-5d84f8dfd6-b8f6s"
Sep 10 19:27:27.600672 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:27.600648    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f9d35e3-2d14-4a94-9e02-30760eeb631d-istio-envoy\") pod \"e2e-shared-a-w2-cfa578-openshift-default-5d84f8dfd6-b8f6s\" (UID: \"2f9d35e3-2d14-4a94-9e02-30760eeb631d\") " pod="openshift-ingress/e2e-shared-a-w2-cfa578-openshift-default-5d84f8dfd6-b8f6s"
Sep 10 19:27:27.600956 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:27.600939    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f9d35e3-2d14-4a94-9e02-30760eeb631d-istio-podinfo\") pod \"e2e-shared-a-w2-cfa578-openshift-default-5d84f8dfd6-b8f6s\" (UID: \"2f9d35e3-2d14-4a94-9e02-30760eeb631d\") " pod="openshift-ingress/e2e-shared-a-w2-cfa578-openshift-default-5d84f8dfd6-b8f6s"
Sep 10 19:27:27.606915 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:27.606886    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f9d35e3-2d14-4a94-9e02-30760eeb631d-istio-token\") pod \"e2e-shared-a-w2-cfa578-openshift-default-5d84f8dfd6-b8f6s\" (UID: \"2f9d35e3-2d14-4a94-9e02-30760eeb631d\") " pod="openshift-ingress/e2e-shared-a-w2-cfa578-openshift-default-5d84f8dfd6-b8f6s"
Sep 10 19:27:27.607020 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:27.606967    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qrkr\" (UniqueName: \"kubernetes.io/projected/2f9d35e3-2d14-4a94-9e02-30760eeb631d-kube-api-access-7qrkr\") pod \"e2e-shared-a-w2-cfa578-openshift-default-5d84f8dfd6-b8f6s\" (UID: \"2f9d35e3-2d14-4a94-9e02-30760eeb631d\") " pod="openshift-ingress/e2e-shared-a-w2-cfa578-openshift-default-5d84f8dfd6-b8f6s"
Sep 10 19:27:27.715305 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:27.715267    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-cfa578-openshift-default-5d84f8dfd6-b8f6s"
Sep 10 19:27:28.007266 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:28.007239    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-cfa578-openshift-default-5d84f8dfd6-b8f6s"]
Sep 10 19:27:28.009616 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:27:28.009588    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f9d35e3_2d14_4a94_9e02_30760eeb631d.slice/crio-27292c9728688b76b7fdbb2aa6f6c2ddd358b38581a00a1e58565f8466d33a9b WatchSource:0}: Error finding container 27292c9728688b76b7fdbb2aa6f6c2ddd358b38581a00a1e58565f8466d33a9b: Status 404 returned error can't find the container with id 27292c9728688b76b7fdbb2aa6f6c2ddd358b38581a00a1e58565f8466d33a9b
Sep 10 19:27:28.011845 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:28.011809    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 19:27:28.011941 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:28.011891    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 19:27:28.012011 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:28.011937    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 19:27:28.954581 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:28.954546    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-cfa578-openshift-default-5d84f8dfd6-b8f6s" event={"ID":"2f9d35e3-2d14-4a94-9e02-30760eeb631d","Type":"ContainerStarted","Data":"34b7713956ca261c4297f998ad74a15f30dae7324ee4cd2365cefaa5bd149d7c"}
Sep 10 19:27:28.955092 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:28.954612    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-cfa578-openshift-default-5d84f8dfd6-b8f6s" event={"ID":"2f9d35e3-2d14-4a94-9e02-30760eeb631d","Type":"ContainerStarted","Data":"27292c9728688b76b7fdbb2aa6f6c2ddd358b38581a00a1e58565f8466d33a9b"}
Sep 10 19:27:28.993198 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:28.993138    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-cfa578-openshift-default-5d84f8dfd6-b8f6s" podStartSLOduration=1.993121324 podStartE2EDuration="1.993121324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:27:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:27:28.990875165 +0000 UTC m=+1012.219925193" watchObservedRunningTime="2026-09-10 19:27:28.993121324 +0000 UTC m=+1012.222171387"
Sep 10 19:27:29.715839 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:29.715792    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-cfa578-openshift-default-5d84f8dfd6-b8f6s"
Sep 10 19:27:29.721215 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:29.721186    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-cfa578-openshift-default-5d84f8dfd6-b8f6s"
Sep 10 19:27:29.961107 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:29.961072    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-cfa578-openshift-default-5d84f8dfd6-b8f6s"
Sep 10 19:27:29.962297 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:29.962271    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-cfa578-openshift-default-5d84f8dfd6-b8f6s"
Sep 10 19:27:30.094421 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:30.094281    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-6b562c/test-model-6b562c-kserve-56cd4564d9-b5vr6" podUID="24e6afb8-845c-4c89-8cea-5236276c5725" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Sep 10 19:27:30.153243 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:27:30.153148    2546 event.go:359] "Server rejected event (will not retry!)" err="namespaces \"ai-tenant-e2e-inf-a-6b562c\" not found" event="&Event{ObjectMeta:{test-model-6b562c-kserve-56cd4564d9-b5vr6.18d40c46c6bb29d6  ai-tenant-e2e-inf-a-6b562c    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:ai-tenant-e2e-inf-a-6b562c,Name:test-model-6b562c-kserve-56cd4564d9-b5vr6,UID:24e6afb8-845c-4c89-8cea-5236276c5725,APIVersion:v1,ResourceVersion:29947,FieldPath:spec.containers{main},},Reason:Unhealthy,Message:Readiness probe failed: Get \"https://10.133.0.43:8000/ready\": context deadline exceeded,Source:EventSource{Component:kubelet,Host:ip-10-0-132-222.ec2.internal,},FirstTimestamp:2026-09-10 19:27:30.094328278 +0000 UTC m=+1013.323378297,LastTimestamp:2026-09-10 19:27:30.094328278 +0000 UTC m=+1013.323378297,Count:1,Type:Warning,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-222.ec2.internal,}"
Sep 10 19:27:30.275713 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:30.275679    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6b562c-openshift-default-854c68595d-lrkhb"]
Sep 10 19:27:30.275882 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:30.275865    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-6b562c-openshift-default-854c68595d-lrkhb" podUID="919bfe97-f0e4-4fcf-8d71-58d7b93357f4" containerName="istio-proxy" containerID="cri-o://d44cbc2f89a78d27f24bb4548dd4b273a11279c2b0cf6bd4abaf49510355094c" gracePeriod=30
Sep 10 19:27:35.542378 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:35.542284    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6b562c-openshift-default-854c68595d-lrkhb"
Sep 10 19:27:35.678676 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:35.678643    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/919bfe97-f0e4-4fcf-8d71-58d7b93357f4-istio-token\") pod \"919bfe97-f0e4-4fcf-8d71-58d7b93357f4\" (UID: \"919bfe97-f0e4-4fcf-8d71-58d7b93357f4\") "
Sep 10 19:27:35.678858 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:35.678707    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/919bfe97-f0e4-4fcf-8d71-58d7b93357f4-workload-certs\") pod \"919bfe97-f0e4-4fcf-8d71-58d7b93357f4\" (UID: \"919bfe97-f0e4-4fcf-8d71-58d7b93357f4\") "
Sep 10 19:27:35.678858 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:35.678808    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/919bfe97-f0e4-4fcf-8d71-58d7b93357f4-istiod-ca-cert\") pod \"919bfe97-f0e4-4fcf-8d71-58d7b93357f4\" (UID: \"919bfe97-f0e4-4fcf-8d71-58d7b93357f4\") "
Sep 10 19:27:35.678955 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:35.678873    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-97tfj\" (UniqueName: \"kubernetes.io/projected/919bfe97-f0e4-4fcf-8d71-58d7b93357f4-kube-api-access-97tfj\") pod \"919bfe97-f0e4-4fcf-8d71-58d7b93357f4\" (UID: \"919bfe97-f0e4-4fcf-8d71-58d7b93357f4\") "
Sep 10 19:27:35.678955 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:35.678925    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/919bfe97-f0e4-4fcf-8d71-58d7b93357f4-istio-envoy\") pod \"919bfe97-f0e4-4fcf-8d71-58d7b93357f4\" (UID: \"919bfe97-f0e4-4fcf-8d71-58d7b93357f4\") "
Sep 10 19:27:35.679089 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:35.678965    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/919bfe97-f0e4-4fcf-8d71-58d7b93357f4-workload-socket\") pod \"919bfe97-f0e4-4fcf-8d71-58d7b93357f4\" (UID: \"919bfe97-f0e4-4fcf-8d71-58d7b93357f4\") "
Sep 10 19:27:35.679089 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:35.678992    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/919bfe97-f0e4-4fcf-8d71-58d7b93357f4-credential-socket\") pod \"919bfe97-f0e4-4fcf-8d71-58d7b93357f4\" (UID: \"919bfe97-f0e4-4fcf-8d71-58d7b93357f4\") "
Sep 10 19:27:35.679089 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:35.679020    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/919bfe97-f0e4-4fcf-8d71-58d7b93357f4-istio-data\") pod \"919bfe97-f0e4-4fcf-8d71-58d7b93357f4\" (UID: \"919bfe97-f0e4-4fcf-8d71-58d7b93357f4\") "
Sep 10 19:27:35.679089 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:35.678961    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/919bfe97-f0e4-4fcf-8d71-58d7b93357f4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "919bfe97-f0e4-4fcf-8d71-58d7b93357f4" (UID: "919bfe97-f0e4-4fcf-8d71-58d7b93357f4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:27:35.679089 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:35.679064    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/919bfe97-f0e4-4fcf-8d71-58d7b93357f4-istio-podinfo\") pod \"919bfe97-f0e4-4fcf-8d71-58d7b93357f4\" (UID: \"919bfe97-f0e4-4fcf-8d71-58d7b93357f4\") "
Sep 10 19:27:35.679383 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:35.679193    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/919bfe97-f0e4-4fcf-8d71-58d7b93357f4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "919bfe97-f0e4-4fcf-8d71-58d7b93357f4" (UID: "919bfe97-f0e4-4fcf-8d71-58d7b93357f4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 19:27:35.679383 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:35.679273    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/919bfe97-f0e4-4fcf-8d71-58d7b93357f4-istio-data" (OuterVolumeSpecName: "istio-data") pod "919bfe97-f0e4-4fcf-8d71-58d7b93357f4" (UID: "919bfe97-f0e4-4fcf-8d71-58d7b93357f4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:27:35.679383 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:35.679295    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/919bfe97-f0e4-4fcf-8d71-58d7b93357f4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "919bfe97-f0e4-4fcf-8d71-58d7b93357f4" (UID: "919bfe97-f0e4-4fcf-8d71-58d7b93357f4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:27:35.679559 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:35.679505    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/919bfe97-f0e4-4fcf-8d71-58d7b93357f4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "919bfe97-f0e4-4fcf-8d71-58d7b93357f4" (UID: "919bfe97-f0e4-4fcf-8d71-58d7b93357f4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:27:35.679559 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:35.679508    2546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/919bfe97-f0e4-4fcf-8d71-58d7b93357f4-workload-socket\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:27:35.679559 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:35.679550    2546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/919bfe97-f0e4-4fcf-8d71-58d7b93357f4-istio-data\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:27:35.679692 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:35.679568    2546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/919bfe97-f0e4-4fcf-8d71-58d7b93357f4-workload-certs\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:27:35.679692 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:35.679582    2546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/919bfe97-f0e4-4fcf-8d71-58d7b93357f4-istiod-ca-cert\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:27:35.681150 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:35.681122    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/919bfe97-f0e4-4fcf-8d71-58d7b93357f4-istio-token" (OuterVolumeSpecName: "istio-token") pod "919bfe97-f0e4-4fcf-8d71-58d7b93357f4" (UID: "919bfe97-f0e4-4fcf-8d71-58d7b93357f4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 19:27:35.681262 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:35.681198    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/919bfe97-f0e4-4fcf-8d71-58d7b93357f4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "919bfe97-f0e4-4fcf-8d71-58d7b93357f4" (UID: "919bfe97-f0e4-4fcf-8d71-58d7b93357f4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 19:27:35.681589 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:35.681567    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/919bfe97-f0e4-4fcf-8d71-58d7b93357f4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "919bfe97-f0e4-4fcf-8d71-58d7b93357f4" (UID: "919bfe97-f0e4-4fcf-8d71-58d7b93357f4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:27:35.681735 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:35.681714    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/919bfe97-f0e4-4fcf-8d71-58d7b93357f4-kube-api-access-97tfj" (OuterVolumeSpecName: "kube-api-access-97tfj") pod "919bfe97-f0e4-4fcf-8d71-58d7b93357f4" (UID: "919bfe97-f0e4-4fcf-8d71-58d7b93357f4"). InnerVolumeSpecName "kube-api-access-97tfj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 19:27:35.780970 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:35.780929    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97tfj\" (UniqueName: \"kubernetes.io/projected/919bfe97-f0e4-4fcf-8d71-58d7b93357f4-kube-api-access-97tfj\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:27:35.780970 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:35.780960    2546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/919bfe97-f0e4-4fcf-8d71-58d7b93357f4-istio-envoy\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:27:35.780970 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:35.780970    2546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/919bfe97-f0e4-4fcf-8d71-58d7b93357f4-credential-socket\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:27:35.780970 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:35.780980    2546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/919bfe97-f0e4-4fcf-8d71-58d7b93357f4-istio-podinfo\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:27:35.781250 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:35.780988    2546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/919bfe97-f0e4-4fcf-8d71-58d7b93357f4-istio-token\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:27:35.993098 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:35.993060    2546 generic.go:356] "Generic (PLEG): container finished" podID="919bfe97-f0e4-4fcf-8d71-58d7b93357f4" containerID="d44cbc2f89a78d27f24bb4548dd4b273a11279c2b0cf6bd4abaf49510355094c" exitCode=0
Sep 10 19:27:35.993271 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:35.993116    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6b562c-openshift-default-854c68595d-lrkhb"
Sep 10 19:27:35.993271 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:35.993121    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6b562c-openshift-default-854c68595d-lrkhb" event={"ID":"919bfe97-f0e4-4fcf-8d71-58d7b93357f4","Type":"ContainerDied","Data":"d44cbc2f89a78d27f24bb4548dd4b273a11279c2b0cf6bd4abaf49510355094c"}
Sep 10 19:27:35.993271 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:35.993173    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6b562c-openshift-default-854c68595d-lrkhb" event={"ID":"919bfe97-f0e4-4fcf-8d71-58d7b93357f4","Type":"ContainerDied","Data":"f9722c31e986f03dce279a587d2cf99bed325d56868998be02e8537d86051926"}
Sep 10 19:27:35.993271 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:35.993202    2546 scope.go:117] "RemoveContainer" containerID="d44cbc2f89a78d27f24bb4548dd4b273a11279c2b0cf6bd4abaf49510355094c"
Sep 10 19:27:36.002262 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:36.002241    2546 scope.go:117] "RemoveContainer" containerID="891a86758496e92e8a45c37e81dbcdd6a961bf206c09bb9d3da18da26ba9225d"
Sep 10 19:27:36.009713 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:36.009695    2546 scope.go:117] "RemoveContainer" containerID="d44cbc2f89a78d27f24bb4548dd4b273a11279c2b0cf6bd4abaf49510355094c"
Sep 10 19:27:36.009976 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:27:36.009959    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d44cbc2f89a78d27f24bb4548dd4b273a11279c2b0cf6bd4abaf49510355094c\": container with ID starting with d44cbc2f89a78d27f24bb4548dd4b273a11279c2b0cf6bd4abaf49510355094c not found: ID does not exist" containerID="d44cbc2f89a78d27f24bb4548dd4b273a11279c2b0cf6bd4abaf49510355094c"
Sep 10 19:27:36.010020 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:36.009984    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d44cbc2f89a78d27f24bb4548dd4b273a11279c2b0cf6bd4abaf49510355094c"} err="failed to get container status \"d44cbc2f89a78d27f24bb4548dd4b273a11279c2b0cf6bd4abaf49510355094c\": rpc error: code = NotFound desc = could not find container \"d44cbc2f89a78d27f24bb4548dd4b273a11279c2b0cf6bd4abaf49510355094c\": container with ID starting with d44cbc2f89a78d27f24bb4548dd4b273a11279c2b0cf6bd4abaf49510355094c not found: ID does not exist"
Sep 10 19:27:36.010020 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:36.010000    2546 scope.go:117] "RemoveContainer" containerID="891a86758496e92e8a45c37e81dbcdd6a961bf206c09bb9d3da18da26ba9225d"
Sep 10 19:27:36.010252 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:27:36.010210    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"891a86758496e92e8a45c37e81dbcdd6a961bf206c09bb9d3da18da26ba9225d\": container with ID starting with 891a86758496e92e8a45c37e81dbcdd6a961bf206c09bb9d3da18da26ba9225d not found: ID does not exist" containerID="891a86758496e92e8a45c37e81dbcdd6a961bf206c09bb9d3da18da26ba9225d"
Sep 10 19:27:36.010294 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:36.010259    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"891a86758496e92e8a45c37e81dbcdd6a961bf206c09bb9d3da18da26ba9225d"} err="failed to get container status \"891a86758496e92e8a45c37e81dbcdd6a961bf206c09bb9d3da18da26ba9225d\": rpc error: code = NotFound desc = could not find container \"891a86758496e92e8a45c37e81dbcdd6a961bf206c09bb9d3da18da26ba9225d\": container with ID starting with 891a86758496e92e8a45c37e81dbcdd6a961bf206c09bb9d3da18da26ba9225d not found: ID does not exist"
Sep 10 19:27:36.014821 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:36.014800    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6b562c-openshift-default-854c68595d-lrkhb"]
Sep 10 19:27:36.019356 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:36.019323    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-6b562c-openshift-default-854c68595d-lrkhb"]
Sep 10 19:27:36.827000 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:36.826966    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-88n8f"]
Sep 10 19:27:37.281522 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:37.281488    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="919bfe97-f0e4-4fcf-8d71-58d7b93357f4" path="/var/lib/kubelet/pods/919bfe97-f0e4-4fcf-8d71-58d7b93357f4/volumes"
Sep 10 19:27:41.783365 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:41.782520    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b960a2-7f955bb764-ftlgf"]
Sep 10 19:27:41.783365 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:41.782795    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b960a2-7f955bb764-ftlgf" podUID="1bd47a6e-dd0f-47ae-b0a3-d25207d9b209" containerName="maas-api" containerID="cri-o://bb0f607bb948d989abfb6cf3226a9f521b7e92f29eae34c8f74b5815fc8d213c" gracePeriod=30
Sep 10 19:27:42.024578 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:42.024548    2546 generic.go:356] "Generic (PLEG): container finished" podID="1bd47a6e-dd0f-47ae-b0a3-d25207d9b209" containerID="bb0f607bb948d989abfb6cf3226a9f521b7e92f29eae34c8f74b5815fc8d213c" exitCode=0
Sep 10 19:27:42.024720 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:42.024617    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b960a2-7f955bb764-ftlgf" event={"ID":"1bd47a6e-dd0f-47ae-b0a3-d25207d9b209","Type":"ContainerDied","Data":"bb0f607bb948d989abfb6cf3226a9f521b7e92f29eae34c8f74b5815fc8d213c"}
Sep 10 19:27:42.108548 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:42.108523    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b960a2-7f955bb764-ftlgf"
Sep 10 19:27:42.130187 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:42.130154    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1bd47a6e-dd0f-47ae-b0a3-d25207d9b209-metrics-certs\") pod \"1bd47a6e-dd0f-47ae-b0a3-d25207d9b209\" (UID: \"1bd47a6e-dd0f-47ae-b0a3-d25207d9b209\") "
Sep 10 19:27:42.130388 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:42.130208    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1bd47a6e-dd0f-47ae-b0a3-d25207d9b209-maas-api-tls\") pod \"1bd47a6e-dd0f-47ae-b0a3-d25207d9b209\" (UID: \"1bd47a6e-dd0f-47ae-b0a3-d25207d9b209\") "
Sep 10 19:27:42.130388 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:42.130236    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4k49t\" (UniqueName: \"kubernetes.io/projected/1bd47a6e-dd0f-47ae-b0a3-d25207d9b209-kube-api-access-4k49t\") pod \"1bd47a6e-dd0f-47ae-b0a3-d25207d9b209\" (UID: \"1bd47a6e-dd0f-47ae-b0a3-d25207d9b209\") "
Sep 10 19:27:42.132437 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:42.132404    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bd47a6e-dd0f-47ae-b0a3-d25207d9b209-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "1bd47a6e-dd0f-47ae-b0a3-d25207d9b209" (UID: "1bd47a6e-dd0f-47ae-b0a3-d25207d9b209"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 19:27:42.132437 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:42.132415    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bd47a6e-dd0f-47ae-b0a3-d25207d9b209-kube-api-access-4k49t" (OuterVolumeSpecName: "kube-api-access-4k49t") pod "1bd47a6e-dd0f-47ae-b0a3-d25207d9b209" (UID: "1bd47a6e-dd0f-47ae-b0a3-d25207d9b209"). InnerVolumeSpecName "kube-api-access-4k49t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 19:27:42.132437 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:42.132417    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bd47a6e-dd0f-47ae-b0a3-d25207d9b209-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1bd47a6e-dd0f-47ae-b0a3-d25207d9b209" (UID: "1bd47a6e-dd0f-47ae-b0a3-d25207d9b209"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 19:27:42.230879 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:42.230843    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bd47a6e-dd0f-47ae-b0a3-d25207d9b209-openshift-service-ca\") pod \"1bd47a6e-dd0f-47ae-b0a3-d25207d9b209\" (UID: \"1bd47a6e-dd0f-47ae-b0a3-d25207d9b209\") "
Sep 10 19:27:42.231115 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:42.231099    2546 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1bd47a6e-dd0f-47ae-b0a3-d25207d9b209-metrics-certs\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:27:42.231198 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:42.231121    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1bd47a6e-dd0f-47ae-b0a3-d25207d9b209-maas-api-tls\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:27:42.231198 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:42.231134    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4k49t\" (UniqueName: \"kubernetes.io/projected/1bd47a6e-dd0f-47ae-b0a3-d25207d9b209-kube-api-access-4k49t\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:27:42.231198 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:42.231143    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bd47a6e-dd0f-47ae-b0a3-d25207d9b209-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1bd47a6e-dd0f-47ae-b0a3-d25207d9b209" (UID: "1bd47a6e-dd0f-47ae-b0a3-d25207d9b209"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 19:27:42.331577 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:42.331534    2546 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bd47a6e-dd0f-47ae-b0a3-d25207d9b209-openshift-service-ca\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:27:43.031019 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:43.030989    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b960a2-7f955bb764-ftlgf"
Sep 10 19:27:43.031539 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:43.030988    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b960a2-7f955bb764-ftlgf" event={"ID":"1bd47a6e-dd0f-47ae-b0a3-d25207d9b209","Type":"ContainerDied","Data":"88019d4cf3dbce72822273808aa202da180dadbe5bd81ec639eb168ba8020bf7"}
Sep 10 19:27:43.031539 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:43.031131    2546 scope.go:117] "RemoveContainer" containerID="bb0f607bb948d989abfb6cf3226a9f521b7e92f29eae34c8f74b5815fc8d213c"
Sep 10 19:27:43.061598 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:43.061537    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b960a2-7f955bb764-ftlgf"]
Sep 10 19:27:43.070154 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:43.070122    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b960a2-7f955bb764-ftlgf"]
Sep 10 19:27:43.282724 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:43.282648    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1bd47a6e-dd0f-47ae-b0a3-d25207d9b209" path="/var/lib/kubelet/pods/1bd47a6e-dd0f-47ae-b0a3-d25207d9b209/volumes"
Sep 10 19:27:46.002547 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:46.002504    2546 patch_prober.go:28] interesting pod/e2e-shared-a-w2-cfa578-openshift-default-5d84f8dfd6-b8f6s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 10 19:27:46.002948 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:46.002596    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-cfa578-openshift-default-5d84f8dfd6-b8f6s" podUID="2f9d35e3-2d14-4a94-9e02-30760eeb631d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 10 19:27:46.620798 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:46.620765    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-88n8f"]
Sep 10 19:27:47.057595 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:47.057554    2546 generic.go:356] "Generic (PLEG): container finished" podID="2f9d35e3-2d14-4a94-9e02-30760eeb631d" containerID="34b7713956ca261c4297f998ad74a15f30dae7324ee4cd2365cefaa5bd149d7c" exitCode=0
Sep 10 19:27:47.057595 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:47.057583    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-cfa578-openshift-default-5d84f8dfd6-b8f6s" event={"ID":"2f9d35e3-2d14-4a94-9e02-30760eeb631d","Type":"ContainerDied","Data":"34b7713956ca261c4297f998ad74a15f30dae7324ee4cd2365cefaa5bd149d7c"}
Sep 10 19:27:47.058133 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:47.058031    2546 scope.go:117] "RemoveContainer" containerID="34b7713956ca261c4297f998ad74a15f30dae7324ee4cd2365cefaa5bd149d7c"
Sep 10 19:27:47.059167 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:47.059132    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 19:27:47.059255 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:47.059224    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 19:27:47.059309 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:47.059259    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 19:27:48.067234 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:48.067199    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-cfa578-openshift-default-5d84f8dfd6-b8f6s" event={"ID":"2f9d35e3-2d14-4a94-9e02-30760eeb631d","Type":"ContainerStarted","Data":"007c8aa351f17dbc8897c33310d9f839a537cc6588b00479b885243f9dc822b9"}
Sep 10 19:27:48.715498 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:48.715457    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-cfa578-openshift-default-5d84f8dfd6-b8f6s"
Sep 10 19:27:48.715751 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:48.715725    2546 patch_prober.go:28] interesting pod/e2e-shared-a-w2-cfa578-openshift-default-5d84f8dfd6-b8f6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Sep 10 19:27:48.715839 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:48.715784    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-cfa578-openshift-default-5d84f8dfd6-b8f6s" podUID="2f9d35e3-2d14-4a94-9e02-30760eeb631d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Sep 10 19:27:49.715810 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:49.715768    2546 patch_prober.go:28] interesting pod/e2e-shared-a-w2-cfa578-openshift-default-5d84f8dfd6-b8f6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Sep 10 19:27:49.716186 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:49.715840    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-cfa578-openshift-default-5d84f8dfd6-b8f6s" podUID="2f9d35e3-2d14-4a94-9e02-30760eeb631d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Sep 10 19:27:50.715768 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:50.715730    2546 patch_prober.go:28] interesting pod/e2e-shared-a-w2-cfa578-openshift-default-5d84f8dfd6-b8f6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Sep 10 19:27:50.715956 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:50.715793    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-cfa578-openshift-default-5d84f8dfd6-b8f6s" podUID="2f9d35e3-2d14-4a94-9e02-30760eeb631d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Sep 10 19:27:51.715705 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:51.715665    2546 patch_prober.go:28] interesting pod/e2e-shared-a-w2-cfa578-openshift-default-5d84f8dfd6-b8f6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Sep 10 19:27:51.715882 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:51.715731    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-cfa578-openshift-default-5d84f8dfd6-b8f6s" podUID="2f9d35e3-2d14-4a94-9e02-30760eeb631d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Sep 10 19:27:52.716179 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:52.716147    2546 patch_prober.go:28] interesting pod/e2e-shared-a-w2-cfa578-openshift-default-5d84f8dfd6-b8f6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Sep 10 19:27:52.716638 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:52.716214    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-cfa578-openshift-default-5d84f8dfd6-b8f6s" podUID="2f9d35e3-2d14-4a94-9e02-30760eeb631d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Sep 10 19:27:53.719769 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:53.719735    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-cfa578-openshift-default-5d84f8dfd6-b8f6s"
Sep 10 19:27:53.720204 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:53.720076    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-cfa578-openshift-default-5d84f8dfd6-b8f6s"
Sep 10 19:27:53.720827 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:53.720811    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-cfa578-openshift-default-5d84f8dfd6-b8f6s"
Sep 10 19:27:54.800986 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:54.800954    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b64b1c-549df87f58-fz2jg"]
Sep 10 19:27:54.801480 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:54.801464    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="919bfe97-f0e4-4fcf-8d71-58d7b93357f4" containerName="istio-proxy"
Sep 10 19:27:54.801538 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:54.801483    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="919bfe97-f0e4-4fcf-8d71-58d7b93357f4" containerName="istio-proxy"
Sep 10 19:27:54.801538 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:54.801521    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1bd47a6e-dd0f-47ae-b0a3-d25207d9b209" containerName="maas-api"
Sep 10 19:27:54.801538 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:54.801530    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="1bd47a6e-dd0f-47ae-b0a3-d25207d9b209" containerName="maas-api"
Sep 10 19:27:54.801648 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:54.801609    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="919bfe97-f0e4-4fcf-8d71-58d7b93357f4" containerName="istio-proxy"
Sep 10 19:27:54.801648 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:54.801622    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="1bd47a6e-dd0f-47ae-b0a3-d25207d9b209" containerName="maas-api"
Sep 10 19:27:54.805282 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:54.805258    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b64b1c-549df87f58-fz2jg"
Sep 10 19:27:54.808108 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:54.808085    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-b64b1c-serving-cert\""
Sep 10 19:27:54.816683 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:54.816656    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b64b1c-549df87f58-fz2jg"]
Sep 10 19:27:54.851362 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:54.851310    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9f3bf25-92a3-411e-8b4c-10681ba8611b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-b64b1c-549df87f58-fz2jg\" (UID: \"e9f3bf25-92a3-411e-8b4c-10681ba8611b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b64b1c-549df87f58-fz2jg"
Sep 10 19:27:54.851362 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:54.851360    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9f3bf25-92a3-411e-8b4c-10681ba8611b-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-b64b1c-549df87f58-fz2jg\" (UID: \"e9f3bf25-92a3-411e-8b4c-10681ba8611b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b64b1c-549df87f58-fz2jg"
Sep 10 19:27:54.851745 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:54.851393    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9f3bf25-92a3-411e-8b4c-10681ba8611b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-b64b1c-549df87f58-fz2jg\" (UID: \"e9f3bf25-92a3-411e-8b4c-10681ba8611b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b64b1c-549df87f58-fz2jg"
Sep 10 19:27:54.851745 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:54.851488    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bw8qx\" (UniqueName: \"kubernetes.io/projected/e9f3bf25-92a3-411e-8b4c-10681ba8611b-kube-api-access-bw8qx\") pod \"maas-api-e2e-shared-a-w3-b64b1c-549df87f58-fz2jg\" (UID: \"e9f3bf25-92a3-411e-8b4c-10681ba8611b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b64b1c-549df87f58-fz2jg"
Sep 10 19:27:54.952163 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:54.952127    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9f3bf25-92a3-411e-8b4c-10681ba8611b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-b64b1c-549df87f58-fz2jg\" (UID: \"e9f3bf25-92a3-411e-8b4c-10681ba8611b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b64b1c-549df87f58-fz2jg"
Sep 10 19:27:54.952163 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:54.952164    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9f3bf25-92a3-411e-8b4c-10681ba8611b-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-b64b1c-549df87f58-fz2jg\" (UID: \"e9f3bf25-92a3-411e-8b4c-10681ba8611b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b64b1c-549df87f58-fz2jg"
Sep 10 19:27:54.952456 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:54.952189    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9f3bf25-92a3-411e-8b4c-10681ba8611b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-b64b1c-549df87f58-fz2jg\" (UID: \"e9f3bf25-92a3-411e-8b4c-10681ba8611b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b64b1c-549df87f58-fz2jg"
Sep 10 19:27:54.952456 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:54.952265    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bw8qx\" (UniqueName: \"kubernetes.io/projected/e9f3bf25-92a3-411e-8b4c-10681ba8611b-kube-api-access-bw8qx\") pod \"maas-api-e2e-shared-a-w3-b64b1c-549df87f58-fz2jg\" (UID: \"e9f3bf25-92a3-411e-8b4c-10681ba8611b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b64b1c-549df87f58-fz2jg"
Sep 10 19:27:54.952456 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:27:54.952289    2546 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b64b1c-serving-cert: secret "maas-api-e2e-shared-a-w3-b64b1c-serving-cert" not found
Sep 10 19:27:54.952456 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:27:54.952377    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9f3bf25-92a3-411e-8b4c-10681ba8611b-maas-api-tls podName:e9f3bf25-92a3-411e-8b4c-10681ba8611b nodeName:}" failed. No retries permitted until 2026-09-10 19:27:55.452352379 +0000 UTC m=+1038.681402399 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e9f3bf25-92a3-411e-8b4c-10681ba8611b-maas-api-tls") pod "maas-api-e2e-shared-a-w3-b64b1c-549df87f58-fz2jg" (UID: "e9f3bf25-92a3-411e-8b4c-10681ba8611b") : secret "maas-api-e2e-shared-a-w3-b64b1c-serving-cert" not found
Sep 10 19:27:54.953033 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:54.953014    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9f3bf25-92a3-411e-8b4c-10681ba8611b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-b64b1c-549df87f58-fz2jg\" (UID: \"e9f3bf25-92a3-411e-8b4c-10681ba8611b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b64b1c-549df87f58-fz2jg"
Sep 10 19:27:54.954655 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:54.954628    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9f3bf25-92a3-411e-8b4c-10681ba8611b-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-b64b1c-549df87f58-fz2jg\" (UID: \"e9f3bf25-92a3-411e-8b4c-10681ba8611b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b64b1c-549df87f58-fz2jg"
Sep 10 19:27:54.966970 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:54.966941    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bw8qx\" (UniqueName: \"kubernetes.io/projected/e9f3bf25-92a3-411e-8b4c-10681ba8611b-kube-api-access-bw8qx\") pod \"maas-api-e2e-shared-a-w3-b64b1c-549df87f58-fz2jg\" (UID: \"e9f3bf25-92a3-411e-8b4c-10681ba8611b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b64b1c-549df87f58-fz2jg"
Sep 10 19:27:55.458589 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:55.458540    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9f3bf25-92a3-411e-8b4c-10681ba8611b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-b64b1c-549df87f58-fz2jg\" (UID: \"e9f3bf25-92a3-411e-8b4c-10681ba8611b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b64b1c-549df87f58-fz2jg"
Sep 10 19:27:55.461026 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:55.460999    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9f3bf25-92a3-411e-8b4c-10681ba8611b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-b64b1c-549df87f58-fz2jg\" (UID: \"e9f3bf25-92a3-411e-8b4c-10681ba8611b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b64b1c-549df87f58-fz2jg"
Sep 10 19:27:55.718923 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:55.718830    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b64b1c-549df87f58-fz2jg"
Sep 10 19:27:56.001257 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:56.001153    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b64b1c-549df87f58-fz2jg"]
Sep 10 19:27:56.004046 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:27:56.004020    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9f3bf25_92a3_411e_8b4c_10681ba8611b.slice/crio-8ad2f4a99c7b15d44a12b550ca9b5d05e301359c1a7632cf4178a75a428b4848 WatchSource:0}: Error finding container 8ad2f4a99c7b15d44a12b550ca9b5d05e301359c1a7632cf4178a75a428b4848: Status 404 returned error can't find the container with id 8ad2f4a99c7b15d44a12b550ca9b5d05e301359c1a7632cf4178a75a428b4848
Sep 10 19:27:56.005473 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:56.005456    2546 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 19:27:56.113486 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:56.113456    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b64b1c-549df87f58-fz2jg" event={"ID":"e9f3bf25-92a3-411e-8b4c-10681ba8611b","Type":"ContainerStarted","Data":"8ad2f4a99c7b15d44a12b550ca9b5d05e301359c1a7632cf4178a75a428b4848"}
Sep 10 19:27:56.523579 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:56.523541    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-94ca92-openshift-default-84d989b484-lgrhw"]
Sep 10 19:27:56.524134 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:56.524118    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="919bfe97-f0e4-4fcf-8d71-58d7b93357f4" containerName="istio-proxy"
Sep 10 19:27:56.524181 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:56.524139    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="919bfe97-f0e4-4fcf-8d71-58d7b93357f4" containerName="istio-proxy"
Sep 10 19:27:56.524232 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:56.524221    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="919bfe97-f0e4-4fcf-8d71-58d7b93357f4" containerName="istio-proxy"
Sep 10 19:27:56.527654 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:56.527633    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-94ca92-openshift-default-84d989b484-lgrhw"
Sep 10 19:27:56.530397 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:56.530375    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-94ca92-openshift-default-dockercfg-xmqql\""
Sep 10 19:27:56.541063 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:56.541038    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-94ca92-openshift-default-84d989b484-lgrhw"]
Sep 10 19:27:56.567738 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:56.567635    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klhc9\" (UniqueName: \"kubernetes.io/projected/d7a16dbc-c7e3-4368-bbca-4c80a7604b31-kube-api-access-klhc9\") pod \"e2e-shared-b-w2-94ca92-openshift-default-84d989b484-lgrhw\" (UID: \"d7a16dbc-c7e3-4368-bbca-4c80a7604b31\") " pod="openshift-ingress/e2e-shared-b-w2-94ca92-openshift-default-84d989b484-lgrhw"
Sep 10 19:27:56.567738 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:56.567685    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7a16dbc-c7e3-4368-bbca-4c80a7604b31-workload-certs\") pod \"e2e-shared-b-w2-94ca92-openshift-default-84d989b484-lgrhw\" (UID: \"d7a16dbc-c7e3-4368-bbca-4c80a7604b31\") " pod="openshift-ingress/e2e-shared-b-w2-94ca92-openshift-default-84d989b484-lgrhw"
Sep 10 19:27:56.568149 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:56.568120    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7a16dbc-c7e3-4368-bbca-4c80a7604b31-istio-envoy\") pod \"e2e-shared-b-w2-94ca92-openshift-default-84d989b484-lgrhw\" (UID: \"d7a16dbc-c7e3-4368-bbca-4c80a7604b31\") " pod="openshift-ingress/e2e-shared-b-w2-94ca92-openshift-default-84d989b484-lgrhw"
Sep 10 19:27:56.568268 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:56.568175    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7a16dbc-c7e3-4368-bbca-4c80a7604b31-istiod-ca-cert\") pod \"e2e-shared-b-w2-94ca92-openshift-default-84d989b484-lgrhw\" (UID: \"d7a16dbc-c7e3-4368-bbca-4c80a7604b31\") " pod="openshift-ingress/e2e-shared-b-w2-94ca92-openshift-default-84d989b484-lgrhw"
Sep 10 19:27:56.568268 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:56.568214    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7a16dbc-c7e3-4368-bbca-4c80a7604b31-istio-token\") pod \"e2e-shared-b-w2-94ca92-openshift-default-84d989b484-lgrhw\" (UID: \"d7a16dbc-c7e3-4368-bbca-4c80a7604b31\") " pod="openshift-ingress/e2e-shared-b-w2-94ca92-openshift-default-84d989b484-lgrhw"
Sep 10 19:27:56.568268 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:56.568249    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7a16dbc-c7e3-4368-bbca-4c80a7604b31-workload-socket\") pod \"e2e-shared-b-w2-94ca92-openshift-default-84d989b484-lgrhw\" (UID: \"d7a16dbc-c7e3-4368-bbca-4c80a7604b31\") " pod="openshift-ingress/e2e-shared-b-w2-94ca92-openshift-default-84d989b484-lgrhw"
Sep 10 19:27:56.568497 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:56.568272    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7a16dbc-c7e3-4368-bbca-4c80a7604b31-istio-data\") pod \"e2e-shared-b-w2-94ca92-openshift-default-84d989b484-lgrhw\" (UID: \"d7a16dbc-c7e3-4368-bbca-4c80a7604b31\") " pod="openshift-ingress/e2e-shared-b-w2-94ca92-openshift-default-84d989b484-lgrhw"
Sep 10 19:27:56.568497 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:56.568368    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7a16dbc-c7e3-4368-bbca-4c80a7604b31-credential-socket\") pod \"e2e-shared-b-w2-94ca92-openshift-default-84d989b484-lgrhw\" (UID: \"d7a16dbc-c7e3-4368-bbca-4c80a7604b31\") " pod="openshift-ingress/e2e-shared-b-w2-94ca92-openshift-default-84d989b484-lgrhw"
Sep 10 19:27:56.568497 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:56.568434    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7a16dbc-c7e3-4368-bbca-4c80a7604b31-istio-podinfo\") pod \"e2e-shared-b-w2-94ca92-openshift-default-84d989b484-lgrhw\" (UID: \"d7a16dbc-c7e3-4368-bbca-4c80a7604b31\") " pod="openshift-ingress/e2e-shared-b-w2-94ca92-openshift-default-84d989b484-lgrhw"
Sep 10 19:27:56.669559 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:56.669468    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7a16dbc-c7e3-4368-bbca-4c80a7604b31-istio-envoy\") pod \"e2e-shared-b-w2-94ca92-openshift-default-84d989b484-lgrhw\" (UID: \"d7a16dbc-c7e3-4368-bbca-4c80a7604b31\") " pod="openshift-ingress/e2e-shared-b-w2-94ca92-openshift-default-84d989b484-lgrhw"
Sep 10 19:27:56.669559 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:56.669521    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7a16dbc-c7e3-4368-bbca-4c80a7604b31-istiod-ca-cert\") pod \"e2e-shared-b-w2-94ca92-openshift-default-84d989b484-lgrhw\" (UID: \"d7a16dbc-c7e3-4368-bbca-4c80a7604b31\") " pod="openshift-ingress/e2e-shared-b-w2-94ca92-openshift-default-84d989b484-lgrhw"
Sep 10 19:27:56.669559 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:56.669556    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7a16dbc-c7e3-4368-bbca-4c80a7604b31-istio-token\") pod \"e2e-shared-b-w2-94ca92-openshift-default-84d989b484-lgrhw\" (UID: \"d7a16dbc-c7e3-4368-bbca-4c80a7604b31\") " pod="openshift-ingress/e2e-shared-b-w2-94ca92-openshift-default-84d989b484-lgrhw"
Sep 10 19:27:56.669842 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:56.669587    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7a16dbc-c7e3-4368-bbca-4c80a7604b31-workload-socket\") pod \"e2e-shared-b-w2-94ca92-openshift-default-84d989b484-lgrhw\" (UID: \"d7a16dbc-c7e3-4368-bbca-4c80a7604b31\") " pod="openshift-ingress/e2e-shared-b-w2-94ca92-openshift-default-84d989b484-lgrhw"
Sep 10 19:27:56.669842 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:56.669614    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7a16dbc-c7e3-4368-bbca-4c80a7604b31-istio-data\") pod \"e2e-shared-b-w2-94ca92-openshift-default-84d989b484-lgrhw\" (UID: \"d7a16dbc-c7e3-4368-bbca-4c80a7604b31\") " pod="openshift-ingress/e2e-shared-b-w2-94ca92-openshift-default-84d989b484-lgrhw"
Sep 10 19:27:56.669842 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:56.669648    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7a16dbc-c7e3-4368-bbca-4c80a7604b31-credential-socket\") pod \"e2e-shared-b-w2-94ca92-openshift-default-84d989b484-lgrhw\" (UID: \"d7a16dbc-c7e3-4368-bbca-4c80a7604b31\") " pod="openshift-ingress/e2e-shared-b-w2-94ca92-openshift-default-84d989b484-lgrhw"
Sep 10 19:27:56.669842 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:56.669679    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7a16dbc-c7e3-4368-bbca-4c80a7604b31-istio-podinfo\") pod \"e2e-shared-b-w2-94ca92-openshift-default-84d989b484-lgrhw\" (UID: \"d7a16dbc-c7e3-4368-bbca-4c80a7604b31\") " pod="openshift-ingress/e2e-shared-b-w2-94ca92-openshift-default-84d989b484-lgrhw"
Sep 10 19:27:56.669842 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:56.669761    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klhc9\" (UniqueName: \"kubernetes.io/projected/d7a16dbc-c7e3-4368-bbca-4c80a7604b31-kube-api-access-klhc9\") pod \"e2e-shared-b-w2-94ca92-openshift-default-84d989b484-lgrhw\" (UID: \"d7a16dbc-c7e3-4368-bbca-4c80a7604b31\") " pod="openshift-ingress/e2e-shared-b-w2-94ca92-openshift-default-84d989b484-lgrhw"
Sep 10 19:27:56.669842 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:56.669789    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7a16dbc-c7e3-4368-bbca-4c80a7604b31-workload-certs\") pod \"e2e-shared-b-w2-94ca92-openshift-default-84d989b484-lgrhw\" (UID: \"d7a16dbc-c7e3-4368-bbca-4c80a7604b31\") " pod="openshift-ingress/e2e-shared-b-w2-94ca92-openshift-default-84d989b484-lgrhw"
Sep 10 19:27:56.670144 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:56.670024    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7a16dbc-c7e3-4368-bbca-4c80a7604b31-istio-data\") pod \"e2e-shared-b-w2-94ca92-openshift-default-84d989b484-lgrhw\" (UID: \"d7a16dbc-c7e3-4368-bbca-4c80a7604b31\") " pod="openshift-ingress/e2e-shared-b-w2-94ca92-openshift-default-84d989b484-lgrhw"
Sep 10 19:27:56.670144 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:56.670087    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7a16dbc-c7e3-4368-bbca-4c80a7604b31-workload-socket\") pod \"e2e-shared-b-w2-94ca92-openshift-default-84d989b484-lgrhw\" (UID: \"d7a16dbc-c7e3-4368-bbca-4c80a7604b31\") " pod="openshift-ingress/e2e-shared-b-w2-94ca92-openshift-default-84d989b484-lgrhw"
Sep 10 19:27:56.670258 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:56.670140    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7a16dbc-c7e3-4368-bbca-4c80a7604b31-workload-certs\") pod \"e2e-shared-b-w2-94ca92-openshift-default-84d989b484-lgrhw\" (UID: \"d7a16dbc-c7e3-4368-bbca-4c80a7604b31\") " pod="openshift-ingress/e2e-shared-b-w2-94ca92-openshift-default-84d989b484-lgrhw"
Sep 10 19:27:56.670407 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:56.670375    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7a16dbc-c7e3-4368-bbca-4c80a7604b31-credential-socket\") pod \"e2e-shared-b-w2-94ca92-openshift-default-84d989b484-lgrhw\" (UID: \"d7a16dbc-c7e3-4368-bbca-4c80a7604b31\") " pod="openshift-ingress/e2e-shared-b-w2-94ca92-openshift-default-84d989b484-lgrhw"
Sep 10 19:27:56.670513 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:56.670378    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7a16dbc-c7e3-4368-bbca-4c80a7604b31-istiod-ca-cert\") pod \"e2e-shared-b-w2-94ca92-openshift-default-84d989b484-lgrhw\" (UID: \"d7a16dbc-c7e3-4368-bbca-4c80a7604b31\") " pod="openshift-ingress/e2e-shared-b-w2-94ca92-openshift-default-84d989b484-lgrhw"
Sep 10 19:27:56.672259 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:56.672237    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7a16dbc-c7e3-4368-bbca-4c80a7604b31-istio-envoy\") pod \"e2e-shared-b-w2-94ca92-openshift-default-84d989b484-lgrhw\" (UID: \"d7a16dbc-c7e3-4368-bbca-4c80a7604b31\") " pod="openshift-ingress/e2e-shared-b-w2-94ca92-openshift-default-84d989b484-lgrhw"
Sep 10 19:27:56.672800 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:56.672779    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7a16dbc-c7e3-4368-bbca-4c80a7604b31-istio-podinfo\") pod \"e2e-shared-b-w2-94ca92-openshift-default-84d989b484-lgrhw\" (UID: \"d7a16dbc-c7e3-4368-bbca-4c80a7604b31\") " pod="openshift-ingress/e2e-shared-b-w2-94ca92-openshift-default-84d989b484-lgrhw"
Sep 10 19:27:56.681902 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:56.681881    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7a16dbc-c7e3-4368-bbca-4c80a7604b31-istio-token\") pod \"e2e-shared-b-w2-94ca92-openshift-default-84d989b484-lgrhw\" (UID: \"d7a16dbc-c7e3-4368-bbca-4c80a7604b31\") " pod="openshift-ingress/e2e-shared-b-w2-94ca92-openshift-default-84d989b484-lgrhw"
Sep 10 19:27:56.682150 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:56.682131    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-klhc9\" (UniqueName: \"kubernetes.io/projected/d7a16dbc-c7e3-4368-bbca-4c80a7604b31-kube-api-access-klhc9\") pod \"e2e-shared-b-w2-94ca92-openshift-default-84d989b484-lgrhw\" (UID: \"d7a16dbc-c7e3-4368-bbca-4c80a7604b31\") " pod="openshift-ingress/e2e-shared-b-w2-94ca92-openshift-default-84d989b484-lgrhw"
Sep 10 19:27:56.837943 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:56.837909    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-94ca92-openshift-default-84d989b484-lgrhw"
Sep 10 19:27:57.119571 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:57.119540    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b64b1c-549df87f58-fz2jg" event={"ID":"e9f3bf25-92a3-411e-8b4c-10681ba8611b","Type":"ContainerStarted","Data":"4a92492bd8809bc5884f2fa27f68c7507e19156e5cd5bb9e835eb39ce9a26231"}
Sep 10 19:27:57.119940 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:57.119671    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b64b1c-549df87f58-fz2jg"
Sep 10 19:27:57.131716 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:57.131688    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-94ca92-openshift-default-84d989b484-lgrhw"]
Sep 10 19:27:57.134582 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:27:57.134559    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7a16dbc_c7e3_4368_bbca_4c80a7604b31.slice/crio-d6398d63170dc768c740fcbcdbc19730bfab37852b1b62b685db46e0a951fbae WatchSource:0}: Error finding container d6398d63170dc768c740fcbcdbc19730bfab37852b1b62b685db46e0a951fbae: Status 404 returned error can't find the container with id d6398d63170dc768c740fcbcdbc19730bfab37852b1b62b685db46e0a951fbae
Sep 10 19:27:57.136713 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:57.136683    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 19:27:57.136813 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:57.136756    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 19:27:57.136872 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:57.136832    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 19:27:57.153569 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:57.153526    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b64b1c-549df87f58-fz2jg" podStartSLOduration=2.792797547 podStartE2EDuration="3.153510106s" totalImagesPullingTime="360.712559ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:27:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:27:57.152464836 +0000 UTC m=+1040.381514861" watchObservedRunningTime="2026-09-10 19:27:57.153510106 +0000 UTC m=+1040.382560130"
Sep 10 19:27:58.129189 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:58.129153    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-94ca92-openshift-default-84d989b484-lgrhw" event={"ID":"d7a16dbc-c7e3-4368-bbca-4c80a7604b31","Type":"ContainerStarted","Data":"1a1e8d1706687f8aa059de8d61a1b1fc0a6343f5b4addef05c15c44b06b48bb9"}
Sep 10 19:27:58.129714 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:58.129209    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-94ca92-openshift-default-84d989b484-lgrhw" event={"ID":"d7a16dbc-c7e3-4368-bbca-4c80a7604b31","Type":"ContainerStarted","Data":"d6398d63170dc768c740fcbcdbc19730bfab37852b1b62b685db46e0a951fbae"}
Sep 10 19:27:58.164489 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:58.164430    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-94ca92-openshift-default-84d989b484-lgrhw" podStartSLOduration=2.164408292 podStartE2EDuration="2.164408292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:27:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:27:58.162785931 +0000 UTC m=+1041.391835956" watchObservedRunningTime="2026-09-10 19:27:58.164408292 +0000 UTC m=+1041.393458317"
Sep 10 19:27:58.838936 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:58.838895    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-94ca92-openshift-default-84d989b484-lgrhw"
Sep 10 19:27:58.844007 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:58.843975    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-94ca92-openshift-default-84d989b484-lgrhw"
Sep 10 19:27:59.134162 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:59.134124    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-94ca92-openshift-default-84d989b484-lgrhw"
Sep 10 19:27:59.135131 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:59.135110    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-94ca92-openshift-default-84d989b484-lgrhw"
Sep 10 19:27:59.256061 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:27:59.256023    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-88n8f"]
Sep 10 19:28:03.134883 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:03.134846    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b64b1c-549df87f58-fz2jg"
Sep 10 19:28:15.121919 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:15.121888    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-88n8f"]
Sep 10 19:28:15.175508 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:15.175464    2546 patch_prober.go:28] interesting pod/e2e-shared-b-w2-94ca92-openshift-default-84d989b484-lgrhw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 10 19:28:15.175680 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:15.175525    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-94ca92-openshift-default-84d989b484-lgrhw" podUID="d7a16dbc-c7e3-4368-bbca-4c80a7604b31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 10 19:28:16.221399 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:16.221294    2546 generic.go:356] "Generic (PLEG): container finished" podID="d7a16dbc-c7e3-4368-bbca-4c80a7604b31" containerID="1a1e8d1706687f8aa059de8d61a1b1fc0a6343f5b4addef05c15c44b06b48bb9" exitCode=0
Sep 10 19:28:16.221399 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:16.221377    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-94ca92-openshift-default-84d989b484-lgrhw" event={"ID":"d7a16dbc-c7e3-4368-bbca-4c80a7604b31","Type":"ContainerDied","Data":"1a1e8d1706687f8aa059de8d61a1b1fc0a6343f5b4addef05c15c44b06b48bb9"}
Sep 10 19:28:16.221821 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:16.221774    2546 scope.go:117] "RemoveContainer" containerID="1a1e8d1706687f8aa059de8d61a1b1fc0a6343f5b4addef05c15c44b06b48bb9"
Sep 10 19:28:16.222860 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:16.222828    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 19:28:16.222973 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:16.222915    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 19:28:16.222973 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:16.222967    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 19:28:17.228781 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:17.228745    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-94ca92-openshift-default-84d989b484-lgrhw" event={"ID":"d7a16dbc-c7e3-4368-bbca-4c80a7604b31","Type":"ContainerStarted","Data":"7a1cfba999ffbf7e46f993bc5ed080f4ac0989b0f544d215c21cc32ec27a20b8"}
Sep 10 19:28:17.838511 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:17.838462    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-94ca92-openshift-default-84d989b484-lgrhw"
Sep 10 19:28:17.838754 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:17.838728    2546 patch_prober.go:28] interesting pod/e2e-shared-b-w2-94ca92-openshift-default-84d989b484-lgrhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Sep 10 19:28:17.838845 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:17.838788    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-94ca92-openshift-default-84d989b484-lgrhw" podUID="d7a16dbc-c7e3-4368-bbca-4c80a7604b31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Sep 10 19:28:18.838831 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:18.838792    2546 patch_prober.go:28] interesting pod/e2e-shared-b-w2-94ca92-openshift-default-84d989b484-lgrhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Sep 10 19:28:18.839224 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:18.838857    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-94ca92-openshift-default-84d989b484-lgrhw" podUID="d7a16dbc-c7e3-4368-bbca-4c80a7604b31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Sep 10 19:28:18.918629 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:18.918594    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-f15c09-openshift-default-7b7497d674-g7xs6"]
Sep 10 19:28:18.922017 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:18.921993    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-f15c09-openshift-default-7b7497d674-g7xs6"
Sep 10 19:28:18.926710 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:18.926683    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-f15c09-openshift-default-dockercfg-kj2g6\""
Sep 10 19:28:18.985444 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:18.985384    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-f15c09-openshift-default-7b7497d674-g7xs6"]
Sep 10 19:28:18.999073 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:18.995671    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2f23b51-72c3-45c6-82cf-da36af76ec82-workload-socket\") pod \"e2e-shared-b-w3-f15c09-openshift-default-7b7497d674-g7xs6\" (UID: \"e2f23b51-72c3-45c6-82cf-da36af76ec82\") " pod="openshift-ingress/e2e-shared-b-w3-f15c09-openshift-default-7b7497d674-g7xs6"
Sep 10 19:28:18.999073 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:18.995750    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2f23b51-72c3-45c6-82cf-da36af76ec82-workload-certs\") pod \"e2e-shared-b-w3-f15c09-openshift-default-7b7497d674-g7xs6\" (UID: \"e2f23b51-72c3-45c6-82cf-da36af76ec82\") " pod="openshift-ingress/e2e-shared-b-w3-f15c09-openshift-default-7b7497d674-g7xs6"
Sep 10 19:28:18.999073 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:18.995789    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2f23b51-72c3-45c6-82cf-da36af76ec82-istio-data\") pod \"e2e-shared-b-w3-f15c09-openshift-default-7b7497d674-g7xs6\" (UID: \"e2f23b51-72c3-45c6-82cf-da36af76ec82\") " pod="openshift-ingress/e2e-shared-b-w3-f15c09-openshift-default-7b7497d674-g7xs6"
Sep 10 19:28:18.999073 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:18.995844    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2f23b51-72c3-45c6-82cf-da36af76ec82-istio-envoy\") pod \"e2e-shared-b-w3-f15c09-openshift-default-7b7497d674-g7xs6\" (UID: \"e2f23b51-72c3-45c6-82cf-da36af76ec82\") " pod="openshift-ingress/e2e-shared-b-w3-f15c09-openshift-default-7b7497d674-g7xs6"
Sep 10 19:28:18.999073 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:18.995882    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2f23b51-72c3-45c6-82cf-da36af76ec82-istio-podinfo\") pod \"e2e-shared-b-w3-f15c09-openshift-default-7b7497d674-g7xs6\" (UID: \"e2f23b51-72c3-45c6-82cf-da36af76ec82\") " pod="openshift-ingress/e2e-shared-b-w3-f15c09-openshift-default-7b7497d674-g7xs6"
Sep 10 19:28:18.999073 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:18.995941    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2f23b51-72c3-45c6-82cf-da36af76ec82-istio-token\") pod \"e2e-shared-b-w3-f15c09-openshift-default-7b7497d674-g7xs6\" (UID: \"e2f23b51-72c3-45c6-82cf-da36af76ec82\") " pod="openshift-ingress/e2e-shared-b-w3-f15c09-openshift-default-7b7497d674-g7xs6"
Sep 10 19:28:18.999073 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:18.995976    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2f23b51-72c3-45c6-82cf-da36af76ec82-istiod-ca-cert\") pod \"e2e-shared-b-w3-f15c09-openshift-default-7b7497d674-g7xs6\" (UID: \"e2f23b51-72c3-45c6-82cf-da36af76ec82\") " pod="openshift-ingress/e2e-shared-b-w3-f15c09-openshift-default-7b7497d674-g7xs6"
Sep 10 19:28:18.999073 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:18.996032    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2f23b51-72c3-45c6-82cf-da36af76ec82-credential-socket\") pod \"e2e-shared-b-w3-f15c09-openshift-default-7b7497d674-g7xs6\" (UID: \"e2f23b51-72c3-45c6-82cf-da36af76ec82\") " pod="openshift-ingress/e2e-shared-b-w3-f15c09-openshift-default-7b7497d674-g7xs6"
Sep 10 19:28:18.999073 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:18.996063    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nctdh\" (UniqueName: \"kubernetes.io/projected/e2f23b51-72c3-45c6-82cf-da36af76ec82-kube-api-access-nctdh\") pod \"e2e-shared-b-w3-f15c09-openshift-default-7b7497d674-g7xs6\" (UID: \"e2f23b51-72c3-45c6-82cf-da36af76ec82\") " pod="openshift-ingress/e2e-shared-b-w3-f15c09-openshift-default-7b7497d674-g7xs6"
Sep 10 19:28:19.096998 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:19.096904    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2f23b51-72c3-45c6-82cf-da36af76ec82-istiod-ca-cert\") pod \"e2e-shared-b-w3-f15c09-openshift-default-7b7497d674-g7xs6\" (UID: \"e2f23b51-72c3-45c6-82cf-da36af76ec82\") " pod="openshift-ingress/e2e-shared-b-w3-f15c09-openshift-default-7b7497d674-g7xs6"
Sep 10 19:28:19.096998 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:19.096966    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2f23b51-72c3-45c6-82cf-da36af76ec82-credential-socket\") pod \"e2e-shared-b-w3-f15c09-openshift-default-7b7497d674-g7xs6\" (UID: \"e2f23b51-72c3-45c6-82cf-da36af76ec82\") " pod="openshift-ingress/e2e-shared-b-w3-f15c09-openshift-default-7b7497d674-g7xs6"
Sep 10 19:28:19.096998 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:19.096992    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nctdh\" (UniqueName: \"kubernetes.io/projected/e2f23b51-72c3-45c6-82cf-da36af76ec82-kube-api-access-nctdh\") pod \"e2e-shared-b-w3-f15c09-openshift-default-7b7497d674-g7xs6\" (UID: \"e2f23b51-72c3-45c6-82cf-da36af76ec82\") " pod="openshift-ingress/e2e-shared-b-w3-f15c09-openshift-default-7b7497d674-g7xs6"
Sep 10 19:28:19.097239 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:19.097109    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2f23b51-72c3-45c6-82cf-da36af76ec82-workload-socket\") pod \"e2e-shared-b-w3-f15c09-openshift-default-7b7497d674-g7xs6\" (UID: \"e2f23b51-72c3-45c6-82cf-da36af76ec82\") " pod="openshift-ingress/e2e-shared-b-w3-f15c09-openshift-default-7b7497d674-g7xs6"
Sep 10 19:28:19.097239 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:19.097202    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2f23b51-72c3-45c6-82cf-da36af76ec82-workload-certs\") pod \"e2e-shared-b-w3-f15c09-openshift-default-7b7497d674-g7xs6\" (UID: \"e2f23b51-72c3-45c6-82cf-da36af76ec82\") " pod="openshift-ingress/e2e-shared-b-w3-f15c09-openshift-default-7b7497d674-g7xs6"
Sep 10 19:28:19.097303 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:19.097240    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2f23b51-72c3-45c6-82cf-da36af76ec82-istio-data\") pod \"e2e-shared-b-w3-f15c09-openshift-default-7b7497d674-g7xs6\" (UID: \"e2f23b51-72c3-45c6-82cf-da36af76ec82\") " pod="openshift-ingress/e2e-shared-b-w3-f15c09-openshift-default-7b7497d674-g7xs6"
Sep 10 19:28:19.097369 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:19.097298    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2f23b51-72c3-45c6-82cf-da36af76ec82-istio-envoy\") pod \"e2e-shared-b-w3-f15c09-openshift-default-7b7497d674-g7xs6\" (UID: \"e2f23b51-72c3-45c6-82cf-da36af76ec82\") " pod="openshift-ingress/e2e-shared-b-w3-f15c09-openshift-default-7b7497d674-g7xs6"
Sep 10 19:28:19.097369 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:19.097334    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2f23b51-72c3-45c6-82cf-da36af76ec82-istio-podinfo\") pod \"e2e-shared-b-w3-f15c09-openshift-default-7b7497d674-g7xs6\" (UID: \"e2f23b51-72c3-45c6-82cf-da36af76ec82\") " pod="openshift-ingress/e2e-shared-b-w3-f15c09-openshift-default-7b7497d674-g7xs6"
Sep 10 19:28:19.097447 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:19.097408    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2f23b51-72c3-45c6-82cf-da36af76ec82-istio-token\") pod \"e2e-shared-b-w3-f15c09-openshift-default-7b7497d674-g7xs6\" (UID: \"e2f23b51-72c3-45c6-82cf-da36af76ec82\") " pod="openshift-ingress/e2e-shared-b-w3-f15c09-openshift-default-7b7497d674-g7xs6"
Sep 10 19:28:19.097730 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:19.097709    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2f23b51-72c3-45c6-82cf-da36af76ec82-credential-socket\") pod \"e2e-shared-b-w3-f15c09-openshift-default-7b7497d674-g7xs6\" (UID: \"e2f23b51-72c3-45c6-82cf-da36af76ec82\") " pod="openshift-ingress/e2e-shared-b-w3-f15c09-openshift-default-7b7497d674-g7xs6"
Sep 10 19:28:19.097876 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:19.097779    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2f23b51-72c3-45c6-82cf-da36af76ec82-istio-data\") pod \"e2e-shared-b-w3-f15c09-openshift-default-7b7497d674-g7xs6\" (UID: \"e2f23b51-72c3-45c6-82cf-da36af76ec82\") " pod="openshift-ingress/e2e-shared-b-w3-f15c09-openshift-default-7b7497d674-g7xs6"
Sep 10 19:28:19.097947 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:19.097789    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2f23b51-72c3-45c6-82cf-da36af76ec82-istiod-ca-cert\") pod \"e2e-shared-b-w3-f15c09-openshift-default-7b7497d674-g7xs6\" (UID: \"e2f23b51-72c3-45c6-82cf-da36af76ec82\") " pod="openshift-ingress/e2e-shared-b-w3-f15c09-openshift-default-7b7497d674-g7xs6"
Sep 10 19:28:19.098031 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:19.098013    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2f23b51-72c3-45c6-82cf-da36af76ec82-workload-socket\") pod \"e2e-shared-b-w3-f15c09-openshift-default-7b7497d674-g7xs6\" (UID: \"e2f23b51-72c3-45c6-82cf-da36af76ec82\") " pod="openshift-ingress/e2e-shared-b-w3-f15c09-openshift-default-7b7497d674-g7xs6"
Sep 10 19:28:19.098081 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:19.098028    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2f23b51-72c3-45c6-82cf-da36af76ec82-workload-certs\") pod \"e2e-shared-b-w3-f15c09-openshift-default-7b7497d674-g7xs6\" (UID: \"e2f23b51-72c3-45c6-82cf-da36af76ec82\") " pod="openshift-ingress/e2e-shared-b-w3-f15c09-openshift-default-7b7497d674-g7xs6"
Sep 10 19:28:19.100167 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:19.100123    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2f23b51-72c3-45c6-82cf-da36af76ec82-istio-envoy\") pod \"e2e-shared-b-w3-f15c09-openshift-default-7b7497d674-g7xs6\" (UID: \"e2f23b51-72c3-45c6-82cf-da36af76ec82\") " pod="openshift-ingress/e2e-shared-b-w3-f15c09-openshift-default-7b7497d674-g7xs6"
Sep 10 19:28:19.100327 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:19.100307    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2f23b51-72c3-45c6-82cf-da36af76ec82-istio-podinfo\") pod \"e2e-shared-b-w3-f15c09-openshift-default-7b7497d674-g7xs6\" (UID: \"e2f23b51-72c3-45c6-82cf-da36af76ec82\") " pod="openshift-ingress/e2e-shared-b-w3-f15c09-openshift-default-7b7497d674-g7xs6"
Sep 10 19:28:19.109072 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:19.109042    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nctdh\" (UniqueName: \"kubernetes.io/projected/e2f23b51-72c3-45c6-82cf-da36af76ec82-kube-api-access-nctdh\") pod \"e2e-shared-b-w3-f15c09-openshift-default-7b7497d674-g7xs6\" (UID: \"e2f23b51-72c3-45c6-82cf-da36af76ec82\") " pod="openshift-ingress/e2e-shared-b-w3-f15c09-openshift-default-7b7497d674-g7xs6"
Sep 10 19:28:19.109989 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:19.109966    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2f23b51-72c3-45c6-82cf-da36af76ec82-istio-token\") pod \"e2e-shared-b-w3-f15c09-openshift-default-7b7497d674-g7xs6\" (UID: \"e2f23b51-72c3-45c6-82cf-da36af76ec82\") " pod="openshift-ingress/e2e-shared-b-w3-f15c09-openshift-default-7b7497d674-g7xs6"
Sep 10 19:28:19.233075 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:19.233037    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-f15c09-openshift-default-7b7497d674-g7xs6"
Sep 10 19:28:19.535819 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:19.535791    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-f15c09-openshift-default-7b7497d674-g7xs6"]
Sep 10 19:28:19.537957 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:28:19.537927    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2f23b51_72c3_45c6_82cf_da36af76ec82.slice/crio-3095a190d4f9ca1d9ee4ba4bac3199fce5b8e39d8f68cba88a8d1c4749ac424f WatchSource:0}: Error finding container 3095a190d4f9ca1d9ee4ba4bac3199fce5b8e39d8f68cba88a8d1c4749ac424f: Status 404 returned error can't find the container with id 3095a190d4f9ca1d9ee4ba4bac3199fce5b8e39d8f68cba88a8d1c4749ac424f
Sep 10 19:28:19.540287 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:19.540251    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 19:28:19.540371 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:19.540320    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 19:28:19.540420 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:19.540366    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 19:28:19.839232 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:19.839145    2546 patch_prober.go:28] interesting pod/e2e-shared-b-w2-94ca92-openshift-default-84d989b484-lgrhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Sep 10 19:28:19.839232 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:19.839198    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-94ca92-openshift-default-84d989b484-lgrhw" podUID="d7a16dbc-c7e3-4368-bbca-4c80a7604b31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Sep 10 19:28:20.244355 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:20.244309    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-f15c09-openshift-default-7b7497d674-g7xs6" event={"ID":"e2f23b51-72c3-45c6-82cf-da36af76ec82","Type":"ContainerStarted","Data":"da3562f518237999c0b345e73729ebb1f0b8302e1e166f90b4d4383e1a29e5b3"}
Sep 10 19:28:20.244568 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:20.244372    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-f15c09-openshift-default-7b7497d674-g7xs6" event={"ID":"e2f23b51-72c3-45c6-82cf-da36af76ec82","Type":"ContainerStarted","Data":"3095a190d4f9ca1d9ee4ba4bac3199fce5b8e39d8f68cba88a8d1c4749ac424f"}
Sep 10 19:28:20.268141 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:20.268064    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-f15c09-openshift-default-7b7497d674-g7xs6" podStartSLOduration=2.268042208 podStartE2EDuration="2.268042208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:28:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:28:20.266803837 +0000 UTC m=+1063.495853861" watchObservedRunningTime="2026-09-10 19:28:20.268042208 +0000 UTC m=+1063.497092227"
Sep 10 19:28:20.839305 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:20.839274    2546 patch_prober.go:28] interesting pod/e2e-shared-b-w2-94ca92-openshift-default-84d989b484-lgrhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Sep 10 19:28:20.839707 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:20.839332    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-94ca92-openshift-default-84d989b484-lgrhw" podUID="d7a16dbc-c7e3-4368-bbca-4c80a7604b31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Sep 10 19:28:21.233711 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:21.233670    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-f15c09-openshift-default-7b7497d674-g7xs6"
Sep 10 19:28:21.238956 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:21.238931    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-f15c09-openshift-default-7b7497d674-g7xs6"
Sep 10 19:28:21.249170 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:21.249136    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-f15c09-openshift-default-7b7497d674-g7xs6"
Sep 10 19:28:21.250121 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:21.250104    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-f15c09-openshift-default-7b7497d674-g7xs6"
Sep 10 19:28:21.838496 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:21.838455    2546 patch_prober.go:28] interesting pod/e2e-shared-b-w2-94ca92-openshift-default-84d989b484-lgrhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Sep 10 19:28:21.838686 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:21.838512    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-94ca92-openshift-default-84d989b484-lgrhw" podUID="d7a16dbc-c7e3-4368-bbca-4c80a7604b31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Sep 10 19:28:22.842731 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:22.842696    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-94ca92-openshift-default-84d989b484-lgrhw"
Sep 10 19:28:22.843189 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:22.843001    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-94ca92-openshift-default-84d989b484-lgrhw"
Sep 10 19:28:22.843909 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:22.843890    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-94ca92-openshift-default-84d989b484-lgrhw"
Sep 10 19:28:29.153252 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:29.153163    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-88n8f"]
Sep 10 19:28:31.009885 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:31.009844    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7525-58f8c5"]
Sep 10 19:28:31.014538 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:31.014520    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7525-58f8c5"
Sep 10 19:28:31.017124 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:31.017100    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-sbhkd\""
Sep 10 19:28:31.020591 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:31.020563    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7525-58f8c5"]
Sep 10 19:28:31.111082 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:31.111041    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lh9z2\" (UniqueName: \"kubernetes.io/projected/e59911f2-2c42-4bb9-ba32-4fdaf38a4ee2-kube-api-access-lh9z2\") pod \"test-curl-7525-58f8c5\" (UID: \"e59911f2-2c42-4bb9-ba32-4fdaf38a4ee2\") " pod="opendatahub/test-curl-7525-58f8c5"
Sep 10 19:28:31.211882 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:31.211826    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lh9z2\" (UniqueName: \"kubernetes.io/projected/e59911f2-2c42-4bb9-ba32-4fdaf38a4ee2-kube-api-access-lh9z2\") pod \"test-curl-7525-58f8c5\" (UID: \"e59911f2-2c42-4bb9-ba32-4fdaf38a4ee2\") " pod="opendatahub/test-curl-7525-58f8c5"
Sep 10 19:28:31.221504 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:31.221472    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lh9z2\" (UniqueName: \"kubernetes.io/projected/e59911f2-2c42-4bb9-ba32-4fdaf38a4ee2-kube-api-access-lh9z2\") pod \"test-curl-7525-58f8c5\" (UID: \"e59911f2-2c42-4bb9-ba32-4fdaf38a4ee2\") " pod="opendatahub/test-curl-7525-58f8c5"
Sep 10 19:28:31.325519 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:31.325434    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7525-58f8c5"
Sep 10 19:28:31.622147 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:31.622097    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7525-58f8c5"]
Sep 10 19:28:31.624558 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:28:31.624531    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode59911f2_2c42_4bb9_ba32_4fdaf38a4ee2.slice/crio-eee5e5db769399699d50835bd1e5366c779897ad5eab92e2794c1cb765dff3d6 WatchSource:0}: Error finding container eee5e5db769399699d50835bd1e5366c779897ad5eab92e2794c1cb765dff3d6: Status 404 returned error can't find the container with id eee5e5db769399699d50835bd1e5366c779897ad5eab92e2794c1cb765dff3d6
Sep 10 19:28:32.307405 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:32.307364    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7525-58f8c5" event={"ID":"e59911f2-2c42-4bb9-ba32-4fdaf38a4ee2","Type":"ContainerStarted","Data":"eee5e5db769399699d50835bd1e5366c779897ad5eab92e2794c1cb765dff3d6"}
Sep 10 19:28:34.319469 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:34.319369    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7525-58f8c5" event={"ID":"e59911f2-2c42-4bb9-ba32-4fdaf38a4ee2","Type":"ContainerStarted","Data":"20fb1bad44591c81984b1bfdc98390b7843028dc9765ffa7ea043a4817cd2047"}
Sep 10 19:28:34.337023 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:34.336970    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7525-58f8c5" podStartSLOduration=2.063227133 podStartE2EDuration="4.33695052s" totalImagesPullingTime="2.273723387s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:28:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:28:34.336300898 +0000 UTC m=+1077.565350924" watchObservedRunningTime="2026-09-10 19:28:34.33695052 +0000 UTC m=+1077.566000545"
Sep 10 19:28:35.026580 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:35.026533    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7525-58f8c5"]
Sep 10 19:28:35.029926 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:35.029899    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7525-58f8c5"]
Sep 10 19:28:35.152875 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:35.152840    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7525-fec0c2"]
Sep 10 19:28:35.153245 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:35.153232    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e59911f2-2c42-4bb9-ba32-4fdaf38a4ee2" containerName="test-curl-7525-58f8c5"
Sep 10 19:28:35.153291 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:35.153247    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="e59911f2-2c42-4bb9-ba32-4fdaf38a4ee2" containerName="test-curl-7525-58f8c5"
Sep 10 19:28:35.153361 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:35.153350    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="e59911f2-2c42-4bb9-ba32-4fdaf38a4ee2" containerName="test-curl-7525-58f8c5"
Sep 10 19:28:35.156105 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:35.156082    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7525-fec0c2"
Sep 10 19:28:35.175623 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:35.175579    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7525-fec0c2"]
Sep 10 19:28:35.250649 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:35.250605    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7srf2\" (UniqueName: \"kubernetes.io/projected/55efd3c9-c811-4717-9eac-ddda613d8060-kube-api-access-7srf2\") pod \"test-curl-7525-fec0c2\" (UID: \"55efd3c9-c811-4717-9eac-ddda613d8060\") " pod="opendatahub/test-curl-7525-fec0c2"
Sep 10 19:28:35.325068 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:28:35.325036    2546 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7525-58f8c5" container="test-curl-7525-58f8c5"
Sep 10 19:28:35.352049 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:35.352019    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7srf2\" (UniqueName: \"kubernetes.io/projected/55efd3c9-c811-4717-9eac-ddda613d8060-kube-api-access-7srf2\") pod \"test-curl-7525-fec0c2\" (UID: \"55efd3c9-c811-4717-9eac-ddda613d8060\") " pod="opendatahub/test-curl-7525-fec0c2"
Sep 10 19:28:35.373221 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:35.361640    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7srf2\" (UniqueName: \"kubernetes.io/projected/55efd3c9-c811-4717-9eac-ddda613d8060-kube-api-access-7srf2\") pod \"test-curl-7525-fec0c2\" (UID: \"55efd3c9-c811-4717-9eac-ddda613d8060\") " pod="opendatahub/test-curl-7525-fec0c2"
Sep 10 19:28:35.492846 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:35.492764    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7525-fec0c2"
Sep 10 19:28:35.779600 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:35.779492    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7525-fec0c2"]
Sep 10 19:28:35.788928 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:28:35.788900    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod55efd3c9_c811_4717_9eac_ddda613d8060.slice/crio-70f8d75ec5716356eb9a4a8b9ad72aac747b8f1ccccfd41199ed056b16ee4975 WatchSource:0}: Error finding container 70f8d75ec5716356eb9a4a8b9ad72aac747b8f1ccccfd41199ed056b16ee4975: Status 404 returned error can't find the container with id 70f8d75ec5716356eb9a4a8b9ad72aac747b8f1ccccfd41199ed056b16ee4975
Sep 10 19:28:36.331075 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:36.331031    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7525-fec0c2" event={"ID":"55efd3c9-c811-4717-9eac-ddda613d8060","Type":"ContainerStarted","Data":"eaf89efea50421fa20628b8e099bca551edeb8c3d9699806871975e4ecaecd3a"}
Sep 10 19:28:36.331541 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:36.331085    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7525-fec0c2" event={"ID":"55efd3c9-c811-4717-9eac-ddda613d8060","Type":"ContainerStarted","Data":"70f8d75ec5716356eb9a4a8b9ad72aac747b8f1ccccfd41199ed056b16ee4975"}
Sep 10 19:28:36.331541 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:36.331201    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7525-58f8c5" podUID="e59911f2-2c42-4bb9-ba32-4fdaf38a4ee2" containerName="test-curl-7525-58f8c5" containerID="cri-o://20fb1bad44591c81984b1bfdc98390b7843028dc9765ffa7ea043a4817cd2047" gracePeriod=2
Sep 10 19:28:36.346442 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:36.346289    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7525-fec0c2" podStartSLOduration=1.346268741 podStartE2EDuration="1.346268741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:28:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:28:36.346128623 +0000 UTC m=+1079.575178649" watchObservedRunningTime="2026-09-10 19:28:36.346268741 +0000 UTC m=+1079.575318766"
Sep 10 19:28:36.967904 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:36.967863    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7525-fec0c2"]
Sep 10 19:28:36.972250 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:36.972220    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7525-fec0c2"]
Sep 10 19:28:37.083399 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:37.083297    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7525-b3b7d3"]
Sep 10 19:28:37.083723 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:37.083708    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55efd3c9-c811-4717-9eac-ddda613d8060" containerName="test-curl-7525-fec0c2"
Sep 10 19:28:37.083769 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:37.083725    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="55efd3c9-c811-4717-9eac-ddda613d8060" containerName="test-curl-7525-fec0c2"
Sep 10 19:28:37.083804 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:37.083798    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="55efd3c9-c811-4717-9eac-ddda613d8060" containerName="test-curl-7525-fec0c2"
Sep 10 19:28:37.086864 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:37.086840    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7525-b3b7d3"
Sep 10 19:28:37.091510 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:37.091481    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7525-b3b7d3"]
Sep 10 19:28:37.180665 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:37.180630    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4tzs\" (UniqueName: \"kubernetes.io/projected/61357968-ce02-4902-aa63-25573ca4cf0f-kube-api-access-f4tzs\") pod \"test-curl-7525-b3b7d3\" (UID: \"61357968-ce02-4902-aa63-25573ca4cf0f\") " pod="opendatahub/test-curl-7525-b3b7d3"
Sep 10 19:28:37.281687 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:37.281654    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4tzs\" (UniqueName: \"kubernetes.io/projected/61357968-ce02-4902-aa63-25573ca4cf0f-kube-api-access-f4tzs\") pod \"test-curl-7525-b3b7d3\" (UID: \"61357968-ce02-4902-aa63-25573ca4cf0f\") " pod="opendatahub/test-curl-7525-b3b7d3"
Sep 10 19:28:37.300960 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:37.300924    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4tzs\" (UniqueName: \"kubernetes.io/projected/61357968-ce02-4902-aa63-25573ca4cf0f-kube-api-access-f4tzs\") pod \"test-curl-7525-b3b7d3\" (UID: \"61357968-ce02-4902-aa63-25573ca4cf0f\") " pod="opendatahub/test-curl-7525-b3b7d3"
Sep 10 19:28:37.336230 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:28:37.336155    2546 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7525-fec0c2" container="test-curl-7525-fec0c2"
Sep 10 19:28:37.398550 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:37.398515    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7525-b3b7d3"
Sep 10 19:28:37.684928 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:37.684888    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7525-b3b7d3"]
Sep 10 19:28:37.686241 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:28:37.686211    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod61357968_ce02_4902_aa63_25573ca4cf0f.slice/crio-9fed259c29856df683d40e196d56b0893de9d615b0b68ea11314497c400f6302 WatchSource:0}: Error finding container 9fed259c29856df683d40e196d56b0893de9d615b0b68ea11314497c400f6302: Status 404 returned error can't find the container with id 9fed259c29856df683d40e196d56b0893de9d615b0b68ea11314497c400f6302
Sep 10 19:28:38.344015 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:38.343977    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7525-b3b7d3" event={"ID":"61357968-ce02-4902-aa63-25573ca4cf0f","Type":"ContainerStarted","Data":"23ba869a0571c63e633ea9e252ce4460f24f1f5cd33c1e2eaf89980dadc19d81"}
Sep 10 19:28:38.344468 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:38.344032    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7525-b3b7d3" event={"ID":"61357968-ce02-4902-aa63-25573ca4cf0f","Type":"ContainerStarted","Data":"9fed259c29856df683d40e196d56b0893de9d615b0b68ea11314497c400f6302"}
Sep 10 19:28:38.344468 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:38.344100    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7525-fec0c2" podUID="55efd3c9-c811-4717-9eac-ddda613d8060" containerName="test-curl-7525-fec0c2" containerID="cri-o://eaf89efea50421fa20628b8e099bca551edeb8c3d9699806871975e4ecaecd3a" gracePeriod=2
Sep 10 19:28:38.378962 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:38.378914    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7525-b3b7d3" podStartSLOduration=1.378898023 podStartE2EDuration="1.378898023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:28:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:28:38.378424406 +0000 UTC m=+1081.607474428" watchObservedRunningTime="2026-09-10 19:28:38.378898023 +0000 UTC m=+1081.607948048"
Sep 10 19:28:38.530136 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:38.530113    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7525-58f8c5"
Sep 10 19:28:38.594154 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:38.594073    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lh9z2\" (UniqueName: \"kubernetes.io/projected/e59911f2-2c42-4bb9-ba32-4fdaf38a4ee2-kube-api-access-lh9z2\") pod \"e59911f2-2c42-4bb9-ba32-4fdaf38a4ee2\" (UID: \"e59911f2-2c42-4bb9-ba32-4fdaf38a4ee2\") "
Sep 10 19:28:38.596248 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:38.596213    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e59911f2-2c42-4bb9-ba32-4fdaf38a4ee2-kube-api-access-lh9z2" (OuterVolumeSpecName: "kube-api-access-lh9z2") pod "e59911f2-2c42-4bb9-ba32-4fdaf38a4ee2" (UID: "e59911f2-2c42-4bb9-ba32-4fdaf38a4ee2"). InnerVolumeSpecName "kube-api-access-lh9z2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 19:28:38.694926 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:38.694890    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lh9z2\" (UniqueName: \"kubernetes.io/projected/e59911f2-2c42-4bb9-ba32-4fdaf38a4ee2-kube-api-access-lh9z2\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:28:38.974822 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:38.974781    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7525-b3b7d3"]
Sep 10 19:28:38.978953 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:38.978923    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7525-b3b7d3"]
Sep 10 19:28:39.102860 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:39.102823    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7525-a7ed95"]
Sep 10 19:28:39.103231 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:39.103217    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61357968-ce02-4902-aa63-25573ca4cf0f" containerName="test-curl-7525-b3b7d3"
Sep 10 19:28:39.103277 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:39.103233    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="61357968-ce02-4902-aa63-25573ca4cf0f" containerName="test-curl-7525-b3b7d3"
Sep 10 19:28:39.103312 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:39.103293    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="61357968-ce02-4902-aa63-25573ca4cf0f" containerName="test-curl-7525-b3b7d3"
Sep 10 19:28:39.106280 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:39.106264    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7525-a7ed95"
Sep 10 19:28:39.116587 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:39.116557    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7525-a7ed95"]
Sep 10 19:28:39.199512 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:39.199468    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndlv2\" (UniqueName: \"kubernetes.io/projected/872d7dfb-d478-4c64-8e9e-527f5eef54aa-kube-api-access-ndlv2\") pod \"test-curl-7525-a7ed95\" (UID: \"872d7dfb-d478-4c64-8e9e-527f5eef54aa\") " pod="opendatahub/test-curl-7525-a7ed95"
Sep 10 19:28:39.281109 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:39.281011    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e59911f2-2c42-4bb9-ba32-4fdaf38a4ee2" path="/var/lib/kubelet/pods/e59911f2-2c42-4bb9-ba32-4fdaf38a4ee2/volumes"
Sep 10 19:28:39.300664 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:39.300634    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndlv2\" (UniqueName: \"kubernetes.io/projected/872d7dfb-d478-4c64-8e9e-527f5eef54aa-kube-api-access-ndlv2\") pod \"test-curl-7525-a7ed95\" (UID: \"872d7dfb-d478-4c64-8e9e-527f5eef54aa\") " pod="opendatahub/test-curl-7525-a7ed95"
Sep 10 19:28:39.310466 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:39.310440    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndlv2\" (UniqueName: \"kubernetes.io/projected/872d7dfb-d478-4c64-8e9e-527f5eef54aa-kube-api-access-ndlv2\") pod \"test-curl-7525-a7ed95\" (UID: \"872d7dfb-d478-4c64-8e9e-527f5eef54aa\") " pod="opendatahub/test-curl-7525-a7ed95"
Sep 10 19:28:39.349509 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:39.349475    2546 generic.go:356] "Generic (PLEG): container finished" podID="e59911f2-2c42-4bb9-ba32-4fdaf38a4ee2" containerID="20fb1bad44591c81984b1bfdc98390b7843028dc9765ffa7ea043a4817cd2047" exitCode=137
Sep 10 19:28:39.349946 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:39.349515    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7525-58f8c5"
Sep 10 19:28:39.349946 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:39.349568    2546 scope.go:117] "RemoveContainer" containerID="20fb1bad44591c81984b1bfdc98390b7843028dc9765ffa7ea043a4817cd2047"
Sep 10 19:28:39.349946 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:28:39.349793    2546 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7525-b3b7d3" container="test-curl-7525-b3b7d3"
Sep 10 19:28:39.358897 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:39.358879    2546 scope.go:117] "RemoveContainer" containerID="20fb1bad44591c81984b1bfdc98390b7843028dc9765ffa7ea043a4817cd2047"
Sep 10 19:28:39.359199 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:28:39.359163    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20fb1bad44591c81984b1bfdc98390b7843028dc9765ffa7ea043a4817cd2047\": container with ID starting with 20fb1bad44591c81984b1bfdc98390b7843028dc9765ffa7ea043a4817cd2047 not found: ID does not exist" containerID="20fb1bad44591c81984b1bfdc98390b7843028dc9765ffa7ea043a4817cd2047"
Sep 10 19:28:39.359312 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:39.359202    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20fb1bad44591c81984b1bfdc98390b7843028dc9765ffa7ea043a4817cd2047"} err="failed to get container status \"20fb1bad44591c81984b1bfdc98390b7843028dc9765ffa7ea043a4817cd2047\": rpc error: code = NotFound desc = could not find container \"20fb1bad44591c81984b1bfdc98390b7843028dc9765ffa7ea043a4817cd2047\": container with ID starting with 20fb1bad44591c81984b1bfdc98390b7843028dc9765ffa7ea043a4817cd2047 not found: ID does not exist"
Sep 10 19:28:39.416171 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:39.416127    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7525-a7ed95"
Sep 10 19:28:39.693534 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:39.693510    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7525-a7ed95"]
Sep 10 19:28:39.695906 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:28:39.695877    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod872d7dfb_d478_4c64_8e9e_527f5eef54aa.slice/crio-12bb2d373c952791a6a2ed7e44897c43e9f6b8939ff03c46426e9c8854b0eb86 WatchSource:0}: Error finding container 12bb2d373c952791a6a2ed7e44897c43e9f6b8939ff03c46426e9c8854b0eb86: Status 404 returned error can't find the container with id 12bb2d373c952791a6a2ed7e44897c43e9f6b8939ff03c46426e9c8854b0eb86
Sep 10 19:28:40.357247 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:40.357216    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7525-a7ed95" event={"ID":"872d7dfb-d478-4c64-8e9e-527f5eef54aa","Type":"ContainerStarted","Data":"c1c97e702cabe3618e5361222e9c066ace367ede135686aab3975ccd61848ab5"}
Sep 10 19:28:40.357653 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:40.357267    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7525-a7ed95" event={"ID":"872d7dfb-d478-4c64-8e9e-527f5eef54aa","Type":"ContainerStarted","Data":"12bb2d373c952791a6a2ed7e44897c43e9f6b8939ff03c46426e9c8854b0eb86"}
Sep 10 19:28:40.359128 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:40.359103    2546 generic.go:356] "Generic (PLEG): container finished" podID="e2f23b51-72c3-45c6-82cf-da36af76ec82" containerID="da3562f518237999c0b345e73729ebb1f0b8302e1e166f90b4d4383e1a29e5b3" exitCode=0
Sep 10 19:28:40.359243 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:40.359180    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-f15c09-openshift-default-7b7497d674-g7xs6" event={"ID":"e2f23b51-72c3-45c6-82cf-da36af76ec82","Type":"ContainerDied","Data":"da3562f518237999c0b345e73729ebb1f0b8302e1e166f90b4d4383e1a29e5b3"}
Sep 10 19:28:40.359635 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:40.359617    2546 scope.go:117] "RemoveContainer" containerID="da3562f518237999c0b345e73729ebb1f0b8302e1e166f90b4d4383e1a29e5b3"
Sep 10 19:28:40.360562 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:40.360532    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7525-b3b7d3" podUID="61357968-ce02-4902-aa63-25573ca4cf0f" containerName="test-curl-7525-b3b7d3" containerID="cri-o://23ba869a0571c63e633ea9e252ce4460f24f1f5cd33c1e2eaf89980dadc19d81" gracePeriod=2
Sep 10 19:28:40.360763 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:40.360728    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 19:28:40.360889 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:40.360819    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 19:28:40.360889 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:40.360871    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 19:28:40.374063 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:40.374017    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7525-a7ed95" podStartSLOduration=1.374001836 podStartE2EDuration="1.374001836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:28:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:28:40.372482185 +0000 UTC m=+1083.601532211" watchObservedRunningTime="2026-09-10 19:28:40.374001836 +0000 UTC m=+1083.603051860"
Sep 10 19:28:40.516589 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:40.516558    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7525-fec0c2"
Sep 10 19:28:40.614897 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:40.614862    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7srf2\" (UniqueName: \"kubernetes.io/projected/55efd3c9-c811-4717-9eac-ddda613d8060-kube-api-access-7srf2\") pod \"55efd3c9-c811-4717-9eac-ddda613d8060\" (UID: \"55efd3c9-c811-4717-9eac-ddda613d8060\") "
Sep 10 19:28:40.616986 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:40.616953    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55efd3c9-c811-4717-9eac-ddda613d8060-kube-api-access-7srf2" (OuterVolumeSpecName: "kube-api-access-7srf2") pod "55efd3c9-c811-4717-9eac-ddda613d8060" (UID: "55efd3c9-c811-4717-9eac-ddda613d8060"). InnerVolumeSpecName "kube-api-access-7srf2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 19:28:40.715980 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:40.715931    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7srf2\" (UniqueName: \"kubernetes.io/projected/55efd3c9-c811-4717-9eac-ddda613d8060-kube-api-access-7srf2\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:28:41.038485 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:41.038374    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7525-a7ed95"]
Sep 10 19:28:41.041043 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:41.041004    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7525-a7ed95"]
Sep 10 19:28:41.284620 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:41.284570    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55efd3c9-c811-4717-9eac-ddda613d8060" path="/var/lib/kubelet/pods/55efd3c9-c811-4717-9eac-ddda613d8060/volumes"
Sep 10 19:28:41.372191 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:41.372145    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-f15c09-openshift-default-7b7497d674-g7xs6" event={"ID":"e2f23b51-72c3-45c6-82cf-da36af76ec82","Type":"ContainerStarted","Data":"4fcec4413d8e8cac002177160fc5d6bf867fc242d6f5610c70c593220e2f311f"}
Sep 10 19:28:41.374741 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:41.374706    2546 generic.go:356] "Generic (PLEG): container finished" podID="55efd3c9-c811-4717-9eac-ddda613d8060" containerID="eaf89efea50421fa20628b8e099bca551edeb8c3d9699806871975e4ecaecd3a" exitCode=137
Sep 10 19:28:41.375080 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:28:41.375054    2546 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7525-a7ed95" container="test-curl-7525-a7ed95"
Sep 10 19:28:41.375287 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:41.375270    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7525-fec0c2"
Sep 10 19:28:41.375655 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:41.375638    2546 scope.go:117] "RemoveContainer" containerID="eaf89efea50421fa20628b8e099bca551edeb8c3d9699806871975e4ecaecd3a"
Sep 10 19:28:41.391776 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:41.391747    2546 scope.go:117] "RemoveContainer" containerID="eaf89efea50421fa20628b8e099bca551edeb8c3d9699806871975e4ecaecd3a"
Sep 10 19:28:41.392188 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:28:41.392164    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eaf89efea50421fa20628b8e099bca551edeb8c3d9699806871975e4ecaecd3a\": container with ID starting with eaf89efea50421fa20628b8e099bca551edeb8c3d9699806871975e4ecaecd3a not found: ID does not exist" containerID="eaf89efea50421fa20628b8e099bca551edeb8c3d9699806871975e4ecaecd3a"
Sep 10 19:28:41.392266 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:41.392199    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eaf89efea50421fa20628b8e099bca551edeb8c3d9699806871975e4ecaecd3a"} err="failed to get container status \"eaf89efea50421fa20628b8e099bca551edeb8c3d9699806871975e4ecaecd3a\": rpc error: code = NotFound desc = could not find container \"eaf89efea50421fa20628b8e099bca551edeb8c3d9699806871975e4ecaecd3a\": container with ID starting with eaf89efea50421fa20628b8e099bca551edeb8c3d9699806871975e4ecaecd3a not found: ID does not exist"
Sep 10 19:28:42.029548 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:42.029514    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-84cb5814-openshift-default-68f49b5764-ncwcg"]
Sep 10 19:28:42.029896 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:42.029884    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="872d7dfb-d478-4c64-8e9e-527f5eef54aa" containerName="test-curl-7525-a7ed95"
Sep 10 19:28:42.029940 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:42.029897    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="872d7dfb-d478-4c64-8e9e-527f5eef54aa" containerName="test-curl-7525-a7ed95"
Sep 10 19:28:42.029972 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:42.029960    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="872d7dfb-d478-4c64-8e9e-527f5eef54aa" containerName="test-curl-7525-a7ed95"
Sep 10 19:28:42.035685 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:42.035663    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-84cb5814-openshift-default-68f49b5764-ncwcg"
Sep 10 19:28:42.038260 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:42.038234    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-84cb5814-openshift-default-dockercfg-ll7m8\""
Sep 10 19:28:42.046816 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:42.046788    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-84cb5814-openshift-default-68f49b5764-ncwcg"]
Sep 10 19:28:42.127111 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:42.127079    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rm9m2\" (UniqueName: \"kubernetes.io/projected/caf8ba4f-361d-467b-b6a5-26dd763612a3-kube-api-access-rm9m2\") pod \"e2e-mt-84cb5814-openshift-default-68f49b5764-ncwcg\" (UID: \"caf8ba4f-361d-467b-b6a5-26dd763612a3\") " pod="openshift-ingress/e2e-mt-84cb5814-openshift-default-68f49b5764-ncwcg"
Sep 10 19:28:42.127303 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:42.127131    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caf8ba4f-361d-467b-b6a5-26dd763612a3-workload-socket\") pod \"e2e-mt-84cb5814-openshift-default-68f49b5764-ncwcg\" (UID: \"caf8ba4f-361d-467b-b6a5-26dd763612a3\") " pod="openshift-ingress/e2e-mt-84cb5814-openshift-default-68f49b5764-ncwcg"
Sep 10 19:28:42.127303 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:42.127196    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caf8ba4f-361d-467b-b6a5-26dd763612a3-workload-certs\") pod \"e2e-mt-84cb5814-openshift-default-68f49b5764-ncwcg\" (UID: \"caf8ba4f-361d-467b-b6a5-26dd763612a3\") " pod="openshift-ingress/e2e-mt-84cb5814-openshift-default-68f49b5764-ncwcg"
Sep 10 19:28:42.127303 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:42.127247    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/caf8ba4f-361d-467b-b6a5-26dd763612a3-istio-token\") pod \"e2e-mt-84cb5814-openshift-default-68f49b5764-ncwcg\" (UID: \"caf8ba4f-361d-467b-b6a5-26dd763612a3\") " pod="openshift-ingress/e2e-mt-84cb5814-openshift-default-68f49b5764-ncwcg"
Sep 10 19:28:42.127303 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:42.127281    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caf8ba4f-361d-467b-b6a5-26dd763612a3-istiod-ca-cert\") pod \"e2e-mt-84cb5814-openshift-default-68f49b5764-ncwcg\" (UID: \"caf8ba4f-361d-467b-b6a5-26dd763612a3\") " pod="openshift-ingress/e2e-mt-84cb5814-openshift-default-68f49b5764-ncwcg"
Sep 10 19:28:42.127553 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:42.127315    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caf8ba4f-361d-467b-b6a5-26dd763612a3-istio-podinfo\") pod \"e2e-mt-84cb5814-openshift-default-68f49b5764-ncwcg\" (UID: \"caf8ba4f-361d-467b-b6a5-26dd763612a3\") " pod="openshift-ingress/e2e-mt-84cb5814-openshift-default-68f49b5764-ncwcg"
Sep 10 19:28:42.127553 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:42.127430    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caf8ba4f-361d-467b-b6a5-26dd763612a3-credential-socket\") pod \"e2e-mt-84cb5814-openshift-default-68f49b5764-ncwcg\" (UID: \"caf8ba4f-361d-467b-b6a5-26dd763612a3\") " pod="openshift-ingress/e2e-mt-84cb5814-openshift-default-68f49b5764-ncwcg"
Sep 10 19:28:42.127553 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:42.127480    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caf8ba4f-361d-467b-b6a5-26dd763612a3-istio-data\") pod \"e2e-mt-84cb5814-openshift-default-68f49b5764-ncwcg\" (UID: \"caf8ba4f-361d-467b-b6a5-26dd763612a3\") " pod="openshift-ingress/e2e-mt-84cb5814-openshift-default-68f49b5764-ncwcg"
Sep 10 19:28:42.127553 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:42.127507    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caf8ba4f-361d-467b-b6a5-26dd763612a3-istio-envoy\") pod \"e2e-mt-84cb5814-openshift-default-68f49b5764-ncwcg\" (UID: \"caf8ba4f-361d-467b-b6a5-26dd763612a3\") " pod="openshift-ingress/e2e-mt-84cb5814-openshift-default-68f49b5764-ncwcg"
Sep 10 19:28:42.228312 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:42.228275    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caf8ba4f-361d-467b-b6a5-26dd763612a3-workload-socket\") pod \"e2e-mt-84cb5814-openshift-default-68f49b5764-ncwcg\" (UID: \"caf8ba4f-361d-467b-b6a5-26dd763612a3\") " pod="openshift-ingress/e2e-mt-84cb5814-openshift-default-68f49b5764-ncwcg"
Sep 10 19:28:42.228312 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:42.228319    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caf8ba4f-361d-467b-b6a5-26dd763612a3-workload-certs\") pod \"e2e-mt-84cb5814-openshift-default-68f49b5764-ncwcg\" (UID: \"caf8ba4f-361d-467b-b6a5-26dd763612a3\") " pod="openshift-ingress/e2e-mt-84cb5814-openshift-default-68f49b5764-ncwcg"
Sep 10 19:28:42.228594 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:42.228359    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/caf8ba4f-361d-467b-b6a5-26dd763612a3-istio-token\") pod \"e2e-mt-84cb5814-openshift-default-68f49b5764-ncwcg\" (UID: \"caf8ba4f-361d-467b-b6a5-26dd763612a3\") " pod="openshift-ingress/e2e-mt-84cb5814-openshift-default-68f49b5764-ncwcg"
Sep 10 19:28:42.228594 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:42.228390    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caf8ba4f-361d-467b-b6a5-26dd763612a3-istiod-ca-cert\") pod \"e2e-mt-84cb5814-openshift-default-68f49b5764-ncwcg\" (UID: \"caf8ba4f-361d-467b-b6a5-26dd763612a3\") " pod="openshift-ingress/e2e-mt-84cb5814-openshift-default-68f49b5764-ncwcg"
Sep 10 19:28:42.228594 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:42.228409    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caf8ba4f-361d-467b-b6a5-26dd763612a3-istio-podinfo\") pod \"e2e-mt-84cb5814-openshift-default-68f49b5764-ncwcg\" (UID: \"caf8ba4f-361d-467b-b6a5-26dd763612a3\") " pod="openshift-ingress/e2e-mt-84cb5814-openshift-default-68f49b5764-ncwcg"
Sep 10 19:28:42.228594 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:42.228448    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caf8ba4f-361d-467b-b6a5-26dd763612a3-credential-socket\") pod \"e2e-mt-84cb5814-openshift-default-68f49b5764-ncwcg\" (UID: \"caf8ba4f-361d-467b-b6a5-26dd763612a3\") " pod="openshift-ingress/e2e-mt-84cb5814-openshift-default-68f49b5764-ncwcg"
Sep 10 19:28:42.228594 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:42.228477    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caf8ba4f-361d-467b-b6a5-26dd763612a3-istio-data\") pod \"e2e-mt-84cb5814-openshift-default-68f49b5764-ncwcg\" (UID: \"caf8ba4f-361d-467b-b6a5-26dd763612a3\") " pod="openshift-ingress/e2e-mt-84cb5814-openshift-default-68f49b5764-ncwcg"
Sep 10 19:28:42.228594 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:42.228501    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caf8ba4f-361d-467b-b6a5-26dd763612a3-istio-envoy\") pod \"e2e-mt-84cb5814-openshift-default-68f49b5764-ncwcg\" (UID: \"caf8ba4f-361d-467b-b6a5-26dd763612a3\") " pod="openshift-ingress/e2e-mt-84cb5814-openshift-default-68f49b5764-ncwcg"
Sep 10 19:28:42.228594 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:42.228540    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rm9m2\" (UniqueName: \"kubernetes.io/projected/caf8ba4f-361d-467b-b6a5-26dd763612a3-kube-api-access-rm9m2\") pod \"e2e-mt-84cb5814-openshift-default-68f49b5764-ncwcg\" (UID: \"caf8ba4f-361d-467b-b6a5-26dd763612a3\") " pod="openshift-ingress/e2e-mt-84cb5814-openshift-default-68f49b5764-ncwcg"
Sep 10 19:28:42.228929 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:42.228749    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caf8ba4f-361d-467b-b6a5-26dd763612a3-workload-certs\") pod \"e2e-mt-84cb5814-openshift-default-68f49b5764-ncwcg\" (UID: \"caf8ba4f-361d-467b-b6a5-26dd763612a3\") " pod="openshift-ingress/e2e-mt-84cb5814-openshift-default-68f49b5764-ncwcg"
Sep 10 19:28:42.228929 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:42.228819    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caf8ba4f-361d-467b-b6a5-26dd763612a3-workload-socket\") pod \"e2e-mt-84cb5814-openshift-default-68f49b5764-ncwcg\" (UID: \"caf8ba4f-361d-467b-b6a5-26dd763612a3\") " pod="openshift-ingress/e2e-mt-84cb5814-openshift-default-68f49b5764-ncwcg"
Sep 10 19:28:42.229000 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:42.228929    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caf8ba4f-361d-467b-b6a5-26dd763612a3-credential-socket\") pod \"e2e-mt-84cb5814-openshift-default-68f49b5764-ncwcg\" (UID: \"caf8ba4f-361d-467b-b6a5-26dd763612a3\") " pod="openshift-ingress/e2e-mt-84cb5814-openshift-default-68f49b5764-ncwcg"
Sep 10 19:28:42.229083 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:42.229060    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caf8ba4f-361d-467b-b6a5-26dd763612a3-istio-data\") pod \"e2e-mt-84cb5814-openshift-default-68f49b5764-ncwcg\" (UID: \"caf8ba4f-361d-467b-b6a5-26dd763612a3\") " pod="openshift-ingress/e2e-mt-84cb5814-openshift-default-68f49b5764-ncwcg"
Sep 10 19:28:42.229197 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:42.229182    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caf8ba4f-361d-467b-b6a5-26dd763612a3-istiod-ca-cert\") pod \"e2e-mt-84cb5814-openshift-default-68f49b5764-ncwcg\" (UID: \"caf8ba4f-361d-467b-b6a5-26dd763612a3\") " pod="openshift-ingress/e2e-mt-84cb5814-openshift-default-68f49b5764-ncwcg"
Sep 10 19:28:42.230981 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:42.230957    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caf8ba4f-361d-467b-b6a5-26dd763612a3-istio-envoy\") pod \"e2e-mt-84cb5814-openshift-default-68f49b5764-ncwcg\" (UID: \"caf8ba4f-361d-467b-b6a5-26dd763612a3\") " pod="openshift-ingress/e2e-mt-84cb5814-openshift-default-68f49b5764-ncwcg"
Sep 10 19:28:42.231393 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:42.231374    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caf8ba4f-361d-467b-b6a5-26dd763612a3-istio-podinfo\") pod \"e2e-mt-84cb5814-openshift-default-68f49b5764-ncwcg\" (UID: \"caf8ba4f-361d-467b-b6a5-26dd763612a3\") " pod="openshift-ingress/e2e-mt-84cb5814-openshift-default-68f49b5764-ncwcg"
Sep 10 19:28:42.233230 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:42.233214    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-f15c09-openshift-default-7b7497d674-g7xs6"
Sep 10 19:28:42.233416 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:42.233397    2546 patch_prober.go:28] interesting pod/e2e-shared-b-w3-f15c09-openshift-default-7b7497d674-g7xs6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Sep 10 19:28:42.233459 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:42.233437    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-f15c09-openshift-default-7b7497d674-g7xs6" podUID="e2f23b51-72c3-45c6-82cf-da36af76ec82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Sep 10 19:28:42.237959 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:42.237931    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/caf8ba4f-361d-467b-b6a5-26dd763612a3-istio-token\") pod \"e2e-mt-84cb5814-openshift-default-68f49b5764-ncwcg\" (UID: \"caf8ba4f-361d-467b-b6a5-26dd763612a3\") " pod="openshift-ingress/e2e-mt-84cb5814-openshift-default-68f49b5764-ncwcg"
Sep 10 19:28:42.238289 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:42.238267    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rm9m2\" (UniqueName: \"kubernetes.io/projected/caf8ba4f-361d-467b-b6a5-26dd763612a3-kube-api-access-rm9m2\") pod \"e2e-mt-84cb5814-openshift-default-68f49b5764-ncwcg\" (UID: \"caf8ba4f-361d-467b-b6a5-26dd763612a3\") " pod="openshift-ingress/e2e-mt-84cb5814-openshift-default-68f49b5764-ncwcg"
Sep 10 19:28:42.346707 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:42.346595    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-84cb5814-openshift-default-68f49b5764-ncwcg"
Sep 10 19:28:42.390801 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:42.390322    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7525-a7ed95" podUID="872d7dfb-d478-4c64-8e9e-527f5eef54aa" containerName="test-curl-7525-a7ed95" containerID="cri-o://c1c97e702cabe3618e5361222e9c066ace367ede135686aab3975ccd61848ab5" gracePeriod=2
Sep 10 19:28:42.590181 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:42.590152    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7525-b3b7d3"
Sep 10 19:28:42.654314 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:42.654210    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-84cb5814-openshift-default-68f49b5764-ncwcg"]
Sep 10 19:28:42.656503 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:28:42.656467    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcaf8ba4f_361d_467b_b6a5_26dd763612a3.slice/crio-a9b1ca35fcf9e775344ce57388aeec8751f276a4c499666022ccb09b53624e4b WatchSource:0}: Error finding container a9b1ca35fcf9e775344ce57388aeec8751f276a4c499666022ccb09b53624e4b: Status 404 returned error can't find the container with id a9b1ca35fcf9e775344ce57388aeec8751f276a4c499666022ccb09b53624e4b
Sep 10 19:28:42.661845 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:42.659662    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 19:28:42.661845 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:42.659759    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 19:28:42.661845 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:42.659807    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 19:28:42.733830 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:42.733659    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f4tzs\" (UniqueName: \"kubernetes.io/projected/61357968-ce02-4902-aa63-25573ca4cf0f-kube-api-access-f4tzs\") pod \"61357968-ce02-4902-aa63-25573ca4cf0f\" (UID: \"61357968-ce02-4902-aa63-25573ca4cf0f\") "
Sep 10 19:28:42.735766 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:42.735739    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61357968-ce02-4902-aa63-25573ca4cf0f-kube-api-access-f4tzs" (OuterVolumeSpecName: "kube-api-access-f4tzs") pod "61357968-ce02-4902-aa63-25573ca4cf0f" (UID: "61357968-ce02-4902-aa63-25573ca4cf0f"). InnerVolumeSpecName "kube-api-access-f4tzs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 19:28:42.835074 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:42.835029    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4tzs\" (UniqueName: \"kubernetes.io/projected/61357968-ce02-4902-aa63-25573ca4cf0f-kube-api-access-f4tzs\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:28:43.234259 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:43.234216    2546 patch_prober.go:28] interesting pod/e2e-shared-b-w3-f15c09-openshift-default-7b7497d674-g7xs6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Sep 10 19:28:43.234455 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:43.234297    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-f15c09-openshift-default-7b7497d674-g7xs6" podUID="e2f23b51-72c3-45c6-82cf-da36af76ec82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Sep 10 19:28:43.285174 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:43.285141    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="61357968-ce02-4902-aa63-25573ca4cf0f" path="/var/lib/kubelet/pods/61357968-ce02-4902-aa63-25573ca4cf0f/volumes"
Sep 10 19:28:43.396397 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:43.396331    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-84cb5814-openshift-default-68f49b5764-ncwcg" event={"ID":"caf8ba4f-361d-467b-b6a5-26dd763612a3","Type":"ContainerStarted","Data":"61386ca02c5de93564ecff8e97a82357191d65f4bed67e20b7c039f5ed8f994b"}
Sep 10 19:28:43.396829 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:43.396428    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-84cb5814-openshift-default-68f49b5764-ncwcg" event={"ID":"caf8ba4f-361d-467b-b6a5-26dd763612a3","Type":"ContainerStarted","Data":"a9b1ca35fcf9e775344ce57388aeec8751f276a4c499666022ccb09b53624e4b"}
Sep 10 19:28:43.397565 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:43.397533    2546 generic.go:356] "Generic (PLEG): container finished" podID="61357968-ce02-4902-aa63-25573ca4cf0f" containerID="23ba869a0571c63e633ea9e252ce4460f24f1f5cd33c1e2eaf89980dadc19d81" exitCode=137
Sep 10 19:28:43.397565 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:43.397563    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7525-b3b7d3"
Sep 10 19:28:43.397754 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:43.397613    2546 scope.go:117] "RemoveContainer" containerID="23ba869a0571c63e633ea9e252ce4460f24f1f5cd33c1e2eaf89980dadc19d81"
Sep 10 19:28:43.406135 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:43.406111    2546 scope.go:117] "RemoveContainer" containerID="23ba869a0571c63e633ea9e252ce4460f24f1f5cd33c1e2eaf89980dadc19d81"
Sep 10 19:28:43.406435 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:28:43.406417    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23ba869a0571c63e633ea9e252ce4460f24f1f5cd33c1e2eaf89980dadc19d81\": container with ID starting with 23ba869a0571c63e633ea9e252ce4460f24f1f5cd33c1e2eaf89980dadc19d81 not found: ID does not exist" containerID="23ba869a0571c63e633ea9e252ce4460f24f1f5cd33c1e2eaf89980dadc19d81"
Sep 10 19:28:43.406485 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:43.406443    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23ba869a0571c63e633ea9e252ce4460f24f1f5cd33c1e2eaf89980dadc19d81"} err="failed to get container status \"23ba869a0571c63e633ea9e252ce4460f24f1f5cd33c1e2eaf89980dadc19d81\": rpc error: code = NotFound desc = could not find container \"23ba869a0571c63e633ea9e252ce4460f24f1f5cd33c1e2eaf89980dadc19d81\": container with ID starting with 23ba869a0571c63e633ea9e252ce4460f24f1f5cd33c1e2eaf89980dadc19d81 not found: ID does not exist"
Sep 10 19:28:43.421647 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:43.421602    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-84cb5814-openshift-default-68f49b5764-ncwcg" podStartSLOduration=1.4215857920000001 podStartE2EDuration="1.421585792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:28:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:28:43.419259713 +0000 UTC m=+1086.648309729" watchObservedRunningTime="2026-09-10 19:28:43.421585792 +0000 UTC m=+1086.650635817"
Sep 10 19:28:44.233604 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:44.233565    2546 patch_prober.go:28] interesting pod/e2e-shared-b-w3-f15c09-openshift-default-7b7497d674-g7xs6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Sep 10 19:28:44.233777 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:44.233617    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-f15c09-openshift-default-7b7497d674-g7xs6" podUID="e2f23b51-72c3-45c6-82cf-da36af76ec82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Sep 10 19:28:44.347259 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:44.347218    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-84cb5814-openshift-default-68f49b5764-ncwcg"
Sep 10 19:28:44.352715 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:44.352690    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-84cb5814-openshift-default-68f49b5764-ncwcg"
Sep 10 19:28:44.405465 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:44.405438    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-84cb5814-openshift-default-68f49b5764-ncwcg"
Sep 10 19:28:44.406402 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:44.406362    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-84cb5814-openshift-default-68f49b5764-ncwcg"
Sep 10 19:28:44.628476 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:44.628453    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7525-a7ed95"
Sep 10 19:28:44.754733 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:44.754691    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ndlv2\" (UniqueName: \"kubernetes.io/projected/872d7dfb-d478-4c64-8e9e-527f5eef54aa-kube-api-access-ndlv2\") pod \"872d7dfb-d478-4c64-8e9e-527f5eef54aa\" (UID: \"872d7dfb-d478-4c64-8e9e-527f5eef54aa\") "
Sep 10 19:28:44.756948 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:44.756908    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/872d7dfb-d478-4c64-8e9e-527f5eef54aa-kube-api-access-ndlv2" (OuterVolumeSpecName: "kube-api-access-ndlv2") pod "872d7dfb-d478-4c64-8e9e-527f5eef54aa" (UID: "872d7dfb-d478-4c64-8e9e-527f5eef54aa"). InnerVolumeSpecName "kube-api-access-ndlv2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 19:28:44.856122 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:44.856070    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndlv2\" (UniqueName: \"kubernetes.io/projected/872d7dfb-d478-4c64-8e9e-527f5eef54aa-kube-api-access-ndlv2\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:28:45.234157 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:45.234050    2546 patch_prober.go:28] interesting pod/e2e-shared-b-w3-f15c09-openshift-default-7b7497d674-g7xs6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Sep 10 19:28:45.234157 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:45.234126    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-f15c09-openshift-default-7b7497d674-g7xs6" podUID="e2f23b51-72c3-45c6-82cf-da36af76ec82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Sep 10 19:28:45.285954 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:45.285905    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="872d7dfb-d478-4c64-8e9e-527f5eef54aa" path="/var/lib/kubelet/pods/872d7dfb-d478-4c64-8e9e-527f5eef54aa/volumes"
Sep 10 19:28:45.411899 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:45.411866    2546 generic.go:356] "Generic (PLEG): container finished" podID="872d7dfb-d478-4c64-8e9e-527f5eef54aa" containerID="c1c97e702cabe3618e5361222e9c066ace367ede135686aab3975ccd61848ab5" exitCode=137
Sep 10 19:28:45.412269 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:45.411923    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7525-a7ed95"
Sep 10 19:28:45.412417 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:45.412388    2546 scope.go:117] "RemoveContainer" containerID="c1c97e702cabe3618e5361222e9c066ace367ede135686aab3975ccd61848ab5"
Sep 10 19:28:45.421467 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:45.421446    2546 scope.go:117] "RemoveContainer" containerID="c1c97e702cabe3618e5361222e9c066ace367ede135686aab3975ccd61848ab5"
Sep 10 19:28:45.421768 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:28:45.421747    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1c97e702cabe3618e5361222e9c066ace367ede135686aab3975ccd61848ab5\": container with ID starting with c1c97e702cabe3618e5361222e9c066ace367ede135686aab3975ccd61848ab5 not found: ID does not exist" containerID="c1c97e702cabe3618e5361222e9c066ace367ede135686aab3975ccd61848ab5"
Sep 10 19:28:45.421815 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:45.421777    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1c97e702cabe3618e5361222e9c066ace367ede135686aab3975ccd61848ab5"} err="failed to get container status \"c1c97e702cabe3618e5361222e9c066ace367ede135686aab3975ccd61848ab5\": rpc error: code = NotFound desc = could not find container \"c1c97e702cabe3618e5361222e9c066ace367ede135686aab3975ccd61848ab5\": container with ID starting with c1c97e702cabe3618e5361222e9c066ace367ede135686aab3975ccd61848ab5 not found: ID does not exist"
Sep 10 19:28:46.234026 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:46.233990    2546 patch_prober.go:28] interesting pod/e2e-shared-b-w3-f15c09-openshift-default-7b7497d674-g7xs6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Sep 10 19:28:46.234221 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:46.234047    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-f15c09-openshift-default-7b7497d674-g7xs6" podUID="e2f23b51-72c3-45c6-82cf-da36af76ec82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Sep 10 19:28:47.237737 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:47.237705    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-f15c09-openshift-default-7b7497d674-g7xs6"
Sep 10 19:28:47.238114 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:47.237999    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-f15c09-openshift-default-7b7497d674-g7xs6"
Sep 10 19:28:47.238710 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:47.238689    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-f15c09-openshift-default-7b7497d674-g7xs6"
Sep 10 19:28:52.332434 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:28:52.332398    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-88n8f"]
Sep 10 19:29:01.635473 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:01.635434    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-b64b1c/auth-test-model-b64b1c-kserve-5b7d7fb44c-86v4p"]
Sep 10 19:29:01.641566 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:01.641545    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-b64b1c/auth-test-model-b64b1c-kserve-5b7d7fb44c-86v4p"
Sep 10 19:29:01.646067 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:01.645663    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-b64b1c\"/\"auth-test-model-b64b1c-kserve-self-signed-certs\""
Sep 10 19:29:01.646067 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:01.645905    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-b64b1c\"/\"openshift-service-ca.crt\""
Sep 10 19:29:01.646307 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:01.646269    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-b64b1c\"/\"kube-root-ca.crt\""
Sep 10 19:29:01.646537 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:01.646515    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-b64b1c\"/\"default-dockercfg-5k7v8\""
Sep 10 19:29:01.648313 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:01.648289    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-b64b1c/auth-test-model-b64b1c-kserve-5b7d7fb44c-86v4p"]
Sep 10 19:29:01.746836 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:01.746793    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1994f618-a7f6-4b01-8856-f4fa930568ae-model-cache\") pod \"auth-test-model-b64b1c-kserve-5b7d7fb44c-86v4p\" (UID: \"1994f618-a7f6-4b01-8856-f4fa930568ae\") " pod="ai-tenant-e2e-shared-a-w3-b64b1c/auth-test-model-b64b1c-kserve-5b7d7fb44c-86v4p"
Sep 10 19:29:01.746836 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:01.746842    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1994f618-a7f6-4b01-8856-f4fa930568ae-tmp-dir\") pod \"auth-test-model-b64b1c-kserve-5b7d7fb44c-86v4p\" (UID: \"1994f618-a7f6-4b01-8856-f4fa930568ae\") " pod="ai-tenant-e2e-shared-a-w3-b64b1c/auth-test-model-b64b1c-kserve-5b7d7fb44c-86v4p"
Sep 10 19:29:01.747100 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:01.746986    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1994f618-a7f6-4b01-8856-f4fa930568ae-dshm\") pod \"auth-test-model-b64b1c-kserve-5b7d7fb44c-86v4p\" (UID: \"1994f618-a7f6-4b01-8856-f4fa930568ae\") " pod="ai-tenant-e2e-shared-a-w3-b64b1c/auth-test-model-b64b1c-kserve-5b7d7fb44c-86v4p"
Sep 10 19:29:01.747100 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:01.747031    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1994f618-a7f6-4b01-8856-f4fa930568ae-home\") pod \"auth-test-model-b64b1c-kserve-5b7d7fb44c-86v4p\" (UID: \"1994f618-a7f6-4b01-8856-f4fa930568ae\") " pod="ai-tenant-e2e-shared-a-w3-b64b1c/auth-test-model-b64b1c-kserve-5b7d7fb44c-86v4p"
Sep 10 19:29:01.747187 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:01.747096    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1994f618-a7f6-4b01-8856-f4fa930568ae-tls-certs\") pod \"auth-test-model-b64b1c-kserve-5b7d7fb44c-86v4p\" (UID: \"1994f618-a7f6-4b01-8856-f4fa930568ae\") " pod="ai-tenant-e2e-shared-a-w3-b64b1c/auth-test-model-b64b1c-kserve-5b7d7fb44c-86v4p"
Sep 10 19:29:01.747187 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:01.747172    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfhk9\" (UniqueName: \"kubernetes.io/projected/1994f618-a7f6-4b01-8856-f4fa930568ae-kube-api-access-vfhk9\") pod \"auth-test-model-b64b1c-kserve-5b7d7fb44c-86v4p\" (UID: \"1994f618-a7f6-4b01-8856-f4fa930568ae\") " pod="ai-tenant-e2e-shared-a-w3-b64b1c/auth-test-model-b64b1c-kserve-5b7d7fb44c-86v4p"
Sep 10 19:29:01.848467 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:01.848434    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1994f618-a7f6-4b01-8856-f4fa930568ae-model-cache\") pod \"auth-test-model-b64b1c-kserve-5b7d7fb44c-86v4p\" (UID: \"1994f618-a7f6-4b01-8856-f4fa930568ae\") " pod="ai-tenant-e2e-shared-a-w3-b64b1c/auth-test-model-b64b1c-kserve-5b7d7fb44c-86v4p"
Sep 10 19:29:01.848467 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:01.848468    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1994f618-a7f6-4b01-8856-f4fa930568ae-tmp-dir\") pod \"auth-test-model-b64b1c-kserve-5b7d7fb44c-86v4p\" (UID: \"1994f618-a7f6-4b01-8856-f4fa930568ae\") " pod="ai-tenant-e2e-shared-a-w3-b64b1c/auth-test-model-b64b1c-kserve-5b7d7fb44c-86v4p"
Sep 10 19:29:01.848698 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:01.848523    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1994f618-a7f6-4b01-8856-f4fa930568ae-dshm\") pod \"auth-test-model-b64b1c-kserve-5b7d7fb44c-86v4p\" (UID: \"1994f618-a7f6-4b01-8856-f4fa930568ae\") " pod="ai-tenant-e2e-shared-a-w3-b64b1c/auth-test-model-b64b1c-kserve-5b7d7fb44c-86v4p"
Sep 10 19:29:01.848698 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:01.848544    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1994f618-a7f6-4b01-8856-f4fa930568ae-home\") pod \"auth-test-model-b64b1c-kserve-5b7d7fb44c-86v4p\" (UID: \"1994f618-a7f6-4b01-8856-f4fa930568ae\") " pod="ai-tenant-e2e-shared-a-w3-b64b1c/auth-test-model-b64b1c-kserve-5b7d7fb44c-86v4p"
Sep 10 19:29:01.848698 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:01.848566    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1994f618-a7f6-4b01-8856-f4fa930568ae-tls-certs\") pod \"auth-test-model-b64b1c-kserve-5b7d7fb44c-86v4p\" (UID: \"1994f618-a7f6-4b01-8856-f4fa930568ae\") " pod="ai-tenant-e2e-shared-a-w3-b64b1c/auth-test-model-b64b1c-kserve-5b7d7fb44c-86v4p"
Sep 10 19:29:01.848698 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:01.848651    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfhk9\" (UniqueName: \"kubernetes.io/projected/1994f618-a7f6-4b01-8856-f4fa930568ae-kube-api-access-vfhk9\") pod \"auth-test-model-b64b1c-kserve-5b7d7fb44c-86v4p\" (UID: \"1994f618-a7f6-4b01-8856-f4fa930568ae\") " pod="ai-tenant-e2e-shared-a-w3-b64b1c/auth-test-model-b64b1c-kserve-5b7d7fb44c-86v4p"
Sep 10 19:29:01.848954 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:01.848929    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1994f618-a7f6-4b01-8856-f4fa930568ae-model-cache\") pod \"auth-test-model-b64b1c-kserve-5b7d7fb44c-86v4p\" (UID: \"1994f618-a7f6-4b01-8856-f4fa930568ae\") " pod="ai-tenant-e2e-shared-a-w3-b64b1c/auth-test-model-b64b1c-kserve-5b7d7fb44c-86v4p"
Sep 10 19:29:01.849056 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:01.848956    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1994f618-a7f6-4b01-8856-f4fa930568ae-tmp-dir\") pod \"auth-test-model-b64b1c-kserve-5b7d7fb44c-86v4p\" (UID: \"1994f618-a7f6-4b01-8856-f4fa930568ae\") " pod="ai-tenant-e2e-shared-a-w3-b64b1c/auth-test-model-b64b1c-kserve-5b7d7fb44c-86v4p"
Sep 10 19:29:01.849056 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:01.849046    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1994f618-a7f6-4b01-8856-f4fa930568ae-home\") pod \"auth-test-model-b64b1c-kserve-5b7d7fb44c-86v4p\" (UID: \"1994f618-a7f6-4b01-8856-f4fa930568ae\") " pod="ai-tenant-e2e-shared-a-w3-b64b1c/auth-test-model-b64b1c-kserve-5b7d7fb44c-86v4p"
Sep 10 19:29:01.851036 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:01.851006    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1994f618-a7f6-4b01-8856-f4fa930568ae-dshm\") pod \"auth-test-model-b64b1c-kserve-5b7d7fb44c-86v4p\" (UID: \"1994f618-a7f6-4b01-8856-f4fa930568ae\") " pod="ai-tenant-e2e-shared-a-w3-b64b1c/auth-test-model-b64b1c-kserve-5b7d7fb44c-86v4p"
Sep 10 19:29:01.851278 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:01.851260    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1994f618-a7f6-4b01-8856-f4fa930568ae-tls-certs\") pod \"auth-test-model-b64b1c-kserve-5b7d7fb44c-86v4p\" (UID: \"1994f618-a7f6-4b01-8856-f4fa930568ae\") " pod="ai-tenant-e2e-shared-a-w3-b64b1c/auth-test-model-b64b1c-kserve-5b7d7fb44c-86v4p"
Sep 10 19:29:01.857245 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:01.857223    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfhk9\" (UniqueName: \"kubernetes.io/projected/1994f618-a7f6-4b01-8856-f4fa930568ae-kube-api-access-vfhk9\") pod \"auth-test-model-b64b1c-kserve-5b7d7fb44c-86v4p\" (UID: \"1994f618-a7f6-4b01-8856-f4fa930568ae\") " pod="ai-tenant-e2e-shared-a-w3-b64b1c/auth-test-model-b64b1c-kserve-5b7d7fb44c-86v4p"
Sep 10 19:29:01.954125 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:01.954040    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-b64b1c/auth-test-model-b64b1c-kserve-5b7d7fb44c-86v4p"
Sep 10 19:29:02.232747 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:02.232701    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-b64b1c/auth-test-model-b64b1c-kserve-5b7d7fb44c-86v4p"]
Sep 10 19:29:02.233437 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:29:02.233398    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1994f618_a7f6_4b01_8856_f4fa930568ae.slice/crio-820cadfd6bd25a58c916405ea0ef68b5f195cb8cc1a1d5f1bb1d28f1fb40c839 WatchSource:0}: Error finding container 820cadfd6bd25a58c916405ea0ef68b5f195cb8cc1a1d5f1bb1d28f1fb40c839: Status 404 returned error can't find the container with id 820cadfd6bd25a58c916405ea0ef68b5f195cb8cc1a1d5f1bb1d28f1fb40c839
Sep 10 19:29:02.516406 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:02.516282    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-b64b1c/auth-test-model-b64b1c-kserve-5b7d7fb44c-86v4p" event={"ID":"1994f618-a7f6-4b01-8856-f4fa930568ae","Type":"ContainerStarted","Data":"c05f32058e5cff033080abd9072b7342a08ee724b2582ce2b2054c5a7b90e5c8"}
Sep 10 19:29:02.516406 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:02.516327    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-b64b1c/auth-test-model-b64b1c-kserve-5b7d7fb44c-86v4p" event={"ID":"1994f618-a7f6-4b01-8856-f4fa930568ae","Type":"ContainerStarted","Data":"820cadfd6bd25a58c916405ea0ef68b5f195cb8cc1a1d5f1bb1d28f1fb40c839"}
Sep 10 19:29:02.536875 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:02.536822    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-b64b1c/auth-test-model-b64b1c-kserve-5b7d7fb44c-86v4p" podStartSLOduration=1.536804993 podStartE2EDuration="1.536804993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:29:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:29:02.535394287 +0000 UTC m=+1105.764444313" watchObservedRunningTime="2026-09-10 19:29:02.536804993 +0000 UTC m=+1105.765855025"
Sep 10 19:29:03.522262 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:03.522228    2546 generic.go:356] "Generic (PLEG): container finished" podID="caf8ba4f-361d-467b-b6a5-26dd763612a3" containerID="61386ca02c5de93564ecff8e97a82357191d65f4bed67e20b7c039f5ed8f994b" exitCode=0
Sep 10 19:29:03.522738 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:03.522298    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-84cb5814-openshift-default-68f49b5764-ncwcg" event={"ID":"caf8ba4f-361d-467b-b6a5-26dd763612a3","Type":"ContainerDied","Data":"61386ca02c5de93564ecff8e97a82357191d65f4bed67e20b7c039f5ed8f994b"}
Sep 10 19:29:03.522830 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:03.522810    2546 scope.go:117] "RemoveContainer" containerID="61386ca02c5de93564ecff8e97a82357191d65f4bed67e20b7c039f5ed8f994b"
Sep 10 19:29:03.523918 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:03.523887    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 19:29:03.523969 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:03.523959    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 19:29:03.524003 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:03.523992    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 19:29:04.530490 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:04.530456    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-84cb5814-openshift-default-68f49b5764-ncwcg" event={"ID":"caf8ba4f-361d-467b-b6a5-26dd763612a3","Type":"ContainerStarted","Data":"0d53510ee61b8338718e8bd3ceca5ff6a29bf5523b94115253e6c2152fb89ec0"}
Sep 10 19:29:05.346851 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:05.346808    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-84cb5814-openshift-default-68f49b5764-ncwcg"
Sep 10 19:29:05.347161 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:05.347062    2546 patch_prober.go:28] interesting pod/e2e-mt-84cb5814-openshift-default-68f49b5764-ncwcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Sep 10 19:29:05.347161 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:05.347122    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-84cb5814-openshift-default-68f49b5764-ncwcg" podUID="caf8ba4f-361d-467b-b6a5-26dd763612a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Sep 10 19:29:05.926406 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:05.926365    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-88n8f"]
Sep 10 19:29:06.348007 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:06.347922    2546 patch_prober.go:28] interesting pod/e2e-mt-84cb5814-openshift-default-68f49b5764-ncwcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Sep 10 19:29:06.348007 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:06.347978    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-84cb5814-openshift-default-68f49b5764-ncwcg" podUID="caf8ba4f-361d-467b-b6a5-26dd763612a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Sep 10 19:29:07.347853 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:07.347813    2546 patch_prober.go:28] interesting pod/e2e-mt-84cb5814-openshift-default-68f49b5764-ncwcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Sep 10 19:29:07.348264 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:07.347875    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-84cb5814-openshift-default-68f49b5764-ncwcg" podUID="caf8ba4f-361d-467b-b6a5-26dd763612a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Sep 10 19:29:08.347149 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:08.347113    2546 patch_prober.go:28] interesting pod/e2e-mt-84cb5814-openshift-default-68f49b5764-ncwcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Sep 10 19:29:08.347352 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:08.347180    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-84cb5814-openshift-default-68f49b5764-ncwcg" podUID="caf8ba4f-361d-467b-b6a5-26dd763612a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Sep 10 19:29:09.347877 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:09.347846    2546 patch_prober.go:28] interesting pod/e2e-mt-84cb5814-openshift-default-68f49b5764-ncwcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Sep 10 19:29:09.348277 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:09.347898    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-84cb5814-openshift-default-68f49b5764-ncwcg" podUID="caf8ba4f-361d-467b-b6a5-26dd763612a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Sep 10 19:29:10.347593 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:10.347556    2546 patch_prober.go:28] interesting pod/e2e-mt-84cb5814-openshift-default-68f49b5764-ncwcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Sep 10 19:29:10.347785 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:10.347614    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-84cb5814-openshift-default-68f49b5764-ncwcg" podUID="caf8ba4f-361d-467b-b6a5-26dd763612a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Sep 10 19:29:11.351116 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:11.351087    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-84cb5814-openshift-default-68f49b5764-ncwcg"
Sep 10 19:29:11.351556 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:11.351380    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-84cb5814-openshift-default-68f49b5764-ncwcg"
Sep 10 19:29:11.351999 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:11.351983    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-84cb5814-openshift-default-68f49b5764-ncwcg"
Sep 10 19:29:11.954810 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:11.954774    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-b64b1c/auth-test-model-b64b1c-kserve-5b7d7fb44c-86v4p"
Sep 10 19:29:11.955028 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:11.954832    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-b64b1c/auth-test-model-b64b1c-kserve-5b7d7fb44c-86v4p"
Sep 10 19:29:11.967779 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:11.967750    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-b64b1c/auth-test-model-b64b1c-kserve-5b7d7fb44c-86v4p"
Sep 10 19:29:12.590954 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:12.590926    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-b64b1c/auth-test-model-b64b1c-kserve-5b7d7fb44c-86v4p"
Sep 10 19:29:21.650236 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:21.650203    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-84cb5814/e2e-lifecycle-model-84cb5814-kserve-7c6648479f-nvq4r"]
Sep 10 19:29:21.653753 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:21.653735    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-84cb5814/e2e-lifecycle-model-84cb5814-kserve-7c6648479f-nvq4r"
Sep 10 19:29:21.658221 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:21.657972    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-84cb5814\"/\"e2e-lifecycle-model-84cb5814-kserve-self-signed-certs\""
Sep 10 19:29:21.658221 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:21.658001    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-84cb5814\"/\"kube-root-ca.crt\""
Sep 10 19:29:21.658221 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:21.658000    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-84cb5814\"/\"openshift-service-ca.crt\""
Sep 10 19:29:21.658787 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:21.658768    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-84cb5814\"/\"default-dockercfg-v9gkb\""
Sep 10 19:29:21.721253 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:21.721212    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-84cb5814/e2e-lifecycle-model-84cb5814-kserve-7c6648479f-nvq4r"]
Sep 10 19:29:21.751270 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:21.751231    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a04a73d6-3ef5-4a52-bff1-7b27bdfcf842-dshm\") pod \"e2e-lifecycle-model-84cb5814-kserve-7c6648479f-nvq4r\" (UID: \"a04a73d6-3ef5-4a52-bff1-7b27bdfcf842\") " pod="ai-tenant-e2e-mt-84cb5814/e2e-lifecycle-model-84cb5814-kserve-7c6648479f-nvq4r"
Sep 10 19:29:21.751270 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:21.751271    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzknc\" (UniqueName: \"kubernetes.io/projected/a04a73d6-3ef5-4a52-bff1-7b27bdfcf842-kube-api-access-dzknc\") pod \"e2e-lifecycle-model-84cb5814-kserve-7c6648479f-nvq4r\" (UID: \"a04a73d6-3ef5-4a52-bff1-7b27bdfcf842\") " pod="ai-tenant-e2e-mt-84cb5814/e2e-lifecycle-model-84cb5814-kserve-7c6648479f-nvq4r"
Sep 10 19:29:21.751527 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:21.751434    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a04a73d6-3ef5-4a52-bff1-7b27bdfcf842-model-cache\") pod \"e2e-lifecycle-model-84cb5814-kserve-7c6648479f-nvq4r\" (UID: \"a04a73d6-3ef5-4a52-bff1-7b27bdfcf842\") " pod="ai-tenant-e2e-mt-84cb5814/e2e-lifecycle-model-84cb5814-kserve-7c6648479f-nvq4r"
Sep 10 19:29:21.751527 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:21.751482    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a04a73d6-3ef5-4a52-bff1-7b27bdfcf842-tmp-dir\") pod \"e2e-lifecycle-model-84cb5814-kserve-7c6648479f-nvq4r\" (UID: \"a04a73d6-3ef5-4a52-bff1-7b27bdfcf842\") " pod="ai-tenant-e2e-mt-84cb5814/e2e-lifecycle-model-84cb5814-kserve-7c6648479f-nvq4r"
Sep 10 19:29:21.751647 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:21.751591    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a04a73d6-3ef5-4a52-bff1-7b27bdfcf842-home\") pod \"e2e-lifecycle-model-84cb5814-kserve-7c6648479f-nvq4r\" (UID: \"a04a73d6-3ef5-4a52-bff1-7b27bdfcf842\") " pod="ai-tenant-e2e-mt-84cb5814/e2e-lifecycle-model-84cb5814-kserve-7c6648479f-nvq4r"
Sep 10 19:29:21.751702 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:21.751675    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a04a73d6-3ef5-4a52-bff1-7b27bdfcf842-tls-certs\") pod \"e2e-lifecycle-model-84cb5814-kserve-7c6648479f-nvq4r\" (UID: \"a04a73d6-3ef5-4a52-bff1-7b27bdfcf842\") " pod="ai-tenant-e2e-mt-84cb5814/e2e-lifecycle-model-84cb5814-kserve-7c6648479f-nvq4r"
Sep 10 19:29:21.852297 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:21.852260    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a04a73d6-3ef5-4a52-bff1-7b27bdfcf842-dshm\") pod \"e2e-lifecycle-model-84cb5814-kserve-7c6648479f-nvq4r\" (UID: \"a04a73d6-3ef5-4a52-bff1-7b27bdfcf842\") " pod="ai-tenant-e2e-mt-84cb5814/e2e-lifecycle-model-84cb5814-kserve-7c6648479f-nvq4r"
Sep 10 19:29:21.852297 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:21.852297    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzknc\" (UniqueName: \"kubernetes.io/projected/a04a73d6-3ef5-4a52-bff1-7b27bdfcf842-kube-api-access-dzknc\") pod \"e2e-lifecycle-model-84cb5814-kserve-7c6648479f-nvq4r\" (UID: \"a04a73d6-3ef5-4a52-bff1-7b27bdfcf842\") " pod="ai-tenant-e2e-mt-84cb5814/e2e-lifecycle-model-84cb5814-kserve-7c6648479f-nvq4r"
Sep 10 19:29:21.852542 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:21.852374    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a04a73d6-3ef5-4a52-bff1-7b27bdfcf842-model-cache\") pod \"e2e-lifecycle-model-84cb5814-kserve-7c6648479f-nvq4r\" (UID: \"a04a73d6-3ef5-4a52-bff1-7b27bdfcf842\") " pod="ai-tenant-e2e-mt-84cb5814/e2e-lifecycle-model-84cb5814-kserve-7c6648479f-nvq4r"
Sep 10 19:29:21.852542 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:21.852408    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a04a73d6-3ef5-4a52-bff1-7b27bdfcf842-tmp-dir\") pod \"e2e-lifecycle-model-84cb5814-kserve-7c6648479f-nvq4r\" (UID: \"a04a73d6-3ef5-4a52-bff1-7b27bdfcf842\") " pod="ai-tenant-e2e-mt-84cb5814/e2e-lifecycle-model-84cb5814-kserve-7c6648479f-nvq4r"
Sep 10 19:29:21.852542 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:21.852470    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a04a73d6-3ef5-4a52-bff1-7b27bdfcf842-home\") pod \"e2e-lifecycle-model-84cb5814-kserve-7c6648479f-nvq4r\" (UID: \"a04a73d6-3ef5-4a52-bff1-7b27bdfcf842\") " pod="ai-tenant-e2e-mt-84cb5814/e2e-lifecycle-model-84cb5814-kserve-7c6648479f-nvq4r"
Sep 10 19:29:21.852542 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:21.852530    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a04a73d6-3ef5-4a52-bff1-7b27bdfcf842-tls-certs\") pod \"e2e-lifecycle-model-84cb5814-kserve-7c6648479f-nvq4r\" (UID: \"a04a73d6-3ef5-4a52-bff1-7b27bdfcf842\") " pod="ai-tenant-e2e-mt-84cb5814/e2e-lifecycle-model-84cb5814-kserve-7c6648479f-nvq4r"
Sep 10 19:29:21.852833 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:21.852804    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a04a73d6-3ef5-4a52-bff1-7b27bdfcf842-model-cache\") pod \"e2e-lifecycle-model-84cb5814-kserve-7c6648479f-nvq4r\" (UID: \"a04a73d6-3ef5-4a52-bff1-7b27bdfcf842\") " pod="ai-tenant-e2e-mt-84cb5814/e2e-lifecycle-model-84cb5814-kserve-7c6648479f-nvq4r"
Sep 10 19:29:21.852833 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:21.852830    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a04a73d6-3ef5-4a52-bff1-7b27bdfcf842-home\") pod \"e2e-lifecycle-model-84cb5814-kserve-7c6648479f-nvq4r\" (UID: \"a04a73d6-3ef5-4a52-bff1-7b27bdfcf842\") " pod="ai-tenant-e2e-mt-84cb5814/e2e-lifecycle-model-84cb5814-kserve-7c6648479f-nvq4r"
Sep 10 19:29:21.853011 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:21.852851    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a04a73d6-3ef5-4a52-bff1-7b27bdfcf842-tmp-dir\") pod \"e2e-lifecycle-model-84cb5814-kserve-7c6648479f-nvq4r\" (UID: \"a04a73d6-3ef5-4a52-bff1-7b27bdfcf842\") " pod="ai-tenant-e2e-mt-84cb5814/e2e-lifecycle-model-84cb5814-kserve-7c6648479f-nvq4r"
Sep 10 19:29:21.854738 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:21.854706    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a04a73d6-3ef5-4a52-bff1-7b27bdfcf842-dshm\") pod \"e2e-lifecycle-model-84cb5814-kserve-7c6648479f-nvq4r\" (UID: \"a04a73d6-3ef5-4a52-bff1-7b27bdfcf842\") " pod="ai-tenant-e2e-mt-84cb5814/e2e-lifecycle-model-84cb5814-kserve-7c6648479f-nvq4r"
Sep 10 19:29:21.854839 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:21.854800    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a04a73d6-3ef5-4a52-bff1-7b27bdfcf842-tls-certs\") pod \"e2e-lifecycle-model-84cb5814-kserve-7c6648479f-nvq4r\" (UID: \"a04a73d6-3ef5-4a52-bff1-7b27bdfcf842\") " pod="ai-tenant-e2e-mt-84cb5814/e2e-lifecycle-model-84cb5814-kserve-7c6648479f-nvq4r"
Sep 10 19:29:21.865147 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:21.865127    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzknc\" (UniqueName: \"kubernetes.io/projected/a04a73d6-3ef5-4a52-bff1-7b27bdfcf842-kube-api-access-dzknc\") pod \"e2e-lifecycle-model-84cb5814-kserve-7c6648479f-nvq4r\" (UID: \"a04a73d6-3ef5-4a52-bff1-7b27bdfcf842\") " pod="ai-tenant-e2e-mt-84cb5814/e2e-lifecycle-model-84cb5814-kserve-7c6648479f-nvq4r"
Sep 10 19:29:21.964691 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:21.964606    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-84cb5814/e2e-lifecycle-model-84cb5814-kserve-7c6648479f-nvq4r"
Sep 10 19:29:22.248386 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:22.248357    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-84cb5814/e2e-lifecycle-model-84cb5814-kserve-7c6648479f-nvq4r"]
Sep 10 19:29:22.251223 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:29:22.251192    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda04a73d6_3ef5_4a52_bff1_7b27bdfcf842.slice/crio-2ff9e283f87875b49f1e15cda2e8edf15191cc8f42f851aa2e36a4568b9d8989 WatchSource:0}: Error finding container 2ff9e283f87875b49f1e15cda2e8edf15191cc8f42f851aa2e36a4568b9d8989: Status 404 returned error can't find the container with id 2ff9e283f87875b49f1e15cda2e8edf15191cc8f42f851aa2e36a4568b9d8989
Sep 10 19:29:22.634545 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:22.634506    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-84cb5814/e2e-lifecycle-model-84cb5814-kserve-7c6648479f-nvq4r" event={"ID":"a04a73d6-3ef5-4a52-bff1-7b27bdfcf842","Type":"ContainerStarted","Data":"88b8322312ae01a7b25e20f2cad6326714a97037464959e9a3e9ee3c18f09231"}
Sep 10 19:29:22.634717 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:22.634558    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-84cb5814/e2e-lifecycle-model-84cb5814-kserve-7c6648479f-nvq4r" event={"ID":"a04a73d6-3ef5-4a52-bff1-7b27bdfcf842","Type":"ContainerStarted","Data":"2ff9e283f87875b49f1e15cda2e8edf15191cc8f42f851aa2e36a4568b9d8989"}
Sep 10 19:29:22.656908 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:22.656847    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-84cb5814/e2e-lifecycle-model-84cb5814-kserve-7c6648479f-nvq4r" podStartSLOduration=1.656825257 podStartE2EDuration="1.656825257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:29:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:29:22.653518247 +0000 UTC m=+1125.882568263" watchObservedRunningTime="2026-09-10 19:29:22.656825257 +0000 UTC m=+1125.885875284"
Sep 10 19:29:26.428409 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:26.428369    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-f15c09/auth-test-model-f15c09-kserve-6985648bdc-h7p6q"]
Sep 10 19:29:26.431933 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:26.431909    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-f15c09/auth-test-model-f15c09-kserve-6985648bdc-h7p6q"
Sep 10 19:29:26.434881 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:26.434860    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-f15c09\"/\"auth-test-model-f15c09-kserve-self-signed-certs\""
Sep 10 19:29:26.435974 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:26.435952    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-f15c09\"/\"kube-root-ca.crt\""
Sep 10 19:29:26.435974 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:26.435966    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-f15c09\"/\"openshift-service-ca.crt\""
Sep 10 19:29:26.436097 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:26.436015    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-f15c09\"/\"default-dockercfg-rcjfv\""
Sep 10 19:29:26.441040 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:26.441020    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-f15c09/auth-test-model-f15c09-kserve-6985648bdc-h7p6q"]
Sep 10 19:29:26.606212 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:26.606170    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96d85a15-afed-43f2-9b29-9b3bbe2af9d4-dshm\") pod \"auth-test-model-f15c09-kserve-6985648bdc-h7p6q\" (UID: \"96d85a15-afed-43f2-9b29-9b3bbe2af9d4\") " pod="ai-tenant-e2e-shared-b-w3-f15c09/auth-test-model-f15c09-kserve-6985648bdc-h7p6q"
Sep 10 19:29:26.606431 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:26.606226    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96d85a15-afed-43f2-9b29-9b3bbe2af9d4-home\") pod \"auth-test-model-f15c09-kserve-6985648bdc-h7p6q\" (UID: \"96d85a15-afed-43f2-9b29-9b3bbe2af9d4\") " pod="ai-tenant-e2e-shared-b-w3-f15c09/auth-test-model-f15c09-kserve-6985648bdc-h7p6q"
Sep 10 19:29:26.606431 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:26.606295    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnq94\" (UniqueName: \"kubernetes.io/projected/96d85a15-afed-43f2-9b29-9b3bbe2af9d4-kube-api-access-xnq94\") pod \"auth-test-model-f15c09-kserve-6985648bdc-h7p6q\" (UID: \"96d85a15-afed-43f2-9b29-9b3bbe2af9d4\") " pod="ai-tenant-e2e-shared-b-w3-f15c09/auth-test-model-f15c09-kserve-6985648bdc-h7p6q"
Sep 10 19:29:26.606431 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:26.606329    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96d85a15-afed-43f2-9b29-9b3bbe2af9d4-tls-certs\") pod \"auth-test-model-f15c09-kserve-6985648bdc-h7p6q\" (UID: \"96d85a15-afed-43f2-9b29-9b3bbe2af9d4\") " pod="ai-tenant-e2e-shared-b-w3-f15c09/auth-test-model-f15c09-kserve-6985648bdc-h7p6q"
Sep 10 19:29:26.606556 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:26.606433    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96d85a15-afed-43f2-9b29-9b3bbe2af9d4-tmp-dir\") pod \"auth-test-model-f15c09-kserve-6985648bdc-h7p6q\" (UID: \"96d85a15-afed-43f2-9b29-9b3bbe2af9d4\") " pod="ai-tenant-e2e-shared-b-w3-f15c09/auth-test-model-f15c09-kserve-6985648bdc-h7p6q"
Sep 10 19:29:26.606556 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:26.606497    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96d85a15-afed-43f2-9b29-9b3bbe2af9d4-model-cache\") pod \"auth-test-model-f15c09-kserve-6985648bdc-h7p6q\" (UID: \"96d85a15-afed-43f2-9b29-9b3bbe2af9d4\") " pod="ai-tenant-e2e-shared-b-w3-f15c09/auth-test-model-f15c09-kserve-6985648bdc-h7p6q"
Sep 10 19:29:26.707272 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:26.707182    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96d85a15-afed-43f2-9b29-9b3bbe2af9d4-model-cache\") pod \"auth-test-model-f15c09-kserve-6985648bdc-h7p6q\" (UID: \"96d85a15-afed-43f2-9b29-9b3bbe2af9d4\") " pod="ai-tenant-e2e-shared-b-w3-f15c09/auth-test-model-f15c09-kserve-6985648bdc-h7p6q"
Sep 10 19:29:26.707272 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:26.707259    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96d85a15-afed-43f2-9b29-9b3bbe2af9d4-dshm\") pod \"auth-test-model-f15c09-kserve-6985648bdc-h7p6q\" (UID: \"96d85a15-afed-43f2-9b29-9b3bbe2af9d4\") " pod="ai-tenant-e2e-shared-b-w3-f15c09/auth-test-model-f15c09-kserve-6985648bdc-h7p6q"
Sep 10 19:29:26.707512 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:26.707306    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96d85a15-afed-43f2-9b29-9b3bbe2af9d4-home\") pod \"auth-test-model-f15c09-kserve-6985648bdc-h7p6q\" (UID: \"96d85a15-afed-43f2-9b29-9b3bbe2af9d4\") " pod="ai-tenant-e2e-shared-b-w3-f15c09/auth-test-model-f15c09-kserve-6985648bdc-h7p6q"
Sep 10 19:29:26.707512 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:26.707368    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnq94\" (UniqueName: \"kubernetes.io/projected/96d85a15-afed-43f2-9b29-9b3bbe2af9d4-kube-api-access-xnq94\") pod \"auth-test-model-f15c09-kserve-6985648bdc-h7p6q\" (UID: \"96d85a15-afed-43f2-9b29-9b3bbe2af9d4\") " pod="ai-tenant-e2e-shared-b-w3-f15c09/auth-test-model-f15c09-kserve-6985648bdc-h7p6q"
Sep 10 19:29:26.707512 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:26.707417    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96d85a15-afed-43f2-9b29-9b3bbe2af9d4-tls-certs\") pod \"auth-test-model-f15c09-kserve-6985648bdc-h7p6q\" (UID: \"96d85a15-afed-43f2-9b29-9b3bbe2af9d4\") " pod="ai-tenant-e2e-shared-b-w3-f15c09/auth-test-model-f15c09-kserve-6985648bdc-h7p6q"
Sep 10 19:29:26.707512 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:26.707466    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96d85a15-afed-43f2-9b29-9b3bbe2af9d4-tmp-dir\") pod \"auth-test-model-f15c09-kserve-6985648bdc-h7p6q\" (UID: \"96d85a15-afed-43f2-9b29-9b3bbe2af9d4\") " pod="ai-tenant-e2e-shared-b-w3-f15c09/auth-test-model-f15c09-kserve-6985648bdc-h7p6q"
Sep 10 19:29:26.707727 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:26.707704    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96d85a15-afed-43f2-9b29-9b3bbe2af9d4-home\") pod \"auth-test-model-f15c09-kserve-6985648bdc-h7p6q\" (UID: \"96d85a15-afed-43f2-9b29-9b3bbe2af9d4\") " pod="ai-tenant-e2e-shared-b-w3-f15c09/auth-test-model-f15c09-kserve-6985648bdc-h7p6q"
Sep 10 19:29:26.707793 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:26.707739    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96d85a15-afed-43f2-9b29-9b3bbe2af9d4-model-cache\") pod \"auth-test-model-f15c09-kserve-6985648bdc-h7p6q\" (UID: \"96d85a15-afed-43f2-9b29-9b3bbe2af9d4\") " pod="ai-tenant-e2e-shared-b-w3-f15c09/auth-test-model-f15c09-kserve-6985648bdc-h7p6q"
Sep 10 19:29:26.707848 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:26.707792    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96d85a15-afed-43f2-9b29-9b3bbe2af9d4-tmp-dir\") pod \"auth-test-model-f15c09-kserve-6985648bdc-h7p6q\" (UID: \"96d85a15-afed-43f2-9b29-9b3bbe2af9d4\") " pod="ai-tenant-e2e-shared-b-w3-f15c09/auth-test-model-f15c09-kserve-6985648bdc-h7p6q"
Sep 10 19:29:26.709662 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:26.709643    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96d85a15-afed-43f2-9b29-9b3bbe2af9d4-dshm\") pod \"auth-test-model-f15c09-kserve-6985648bdc-h7p6q\" (UID: \"96d85a15-afed-43f2-9b29-9b3bbe2af9d4\") " pod="ai-tenant-e2e-shared-b-w3-f15c09/auth-test-model-f15c09-kserve-6985648bdc-h7p6q"
Sep 10 19:29:26.709979 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:26.709960    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96d85a15-afed-43f2-9b29-9b3bbe2af9d4-tls-certs\") pod \"auth-test-model-f15c09-kserve-6985648bdc-h7p6q\" (UID: \"96d85a15-afed-43f2-9b29-9b3bbe2af9d4\") " pod="ai-tenant-e2e-shared-b-w3-f15c09/auth-test-model-f15c09-kserve-6985648bdc-h7p6q"
Sep 10 19:29:26.717717 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:26.717684    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnq94\" (UniqueName: \"kubernetes.io/projected/96d85a15-afed-43f2-9b29-9b3bbe2af9d4-kube-api-access-xnq94\") pod \"auth-test-model-f15c09-kserve-6985648bdc-h7p6q\" (UID: \"96d85a15-afed-43f2-9b29-9b3bbe2af9d4\") " pod="ai-tenant-e2e-shared-b-w3-f15c09/auth-test-model-f15c09-kserve-6985648bdc-h7p6q"
Sep 10 19:29:26.742529 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:26.742495    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-f15c09/auth-test-model-f15c09-kserve-6985648bdc-h7p6q"
Sep 10 19:29:27.021524 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:27.021495    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-f15c09/auth-test-model-f15c09-kserve-6985648bdc-h7p6q"]
Sep 10 19:29:27.023129 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:29:27.023099    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod96d85a15_afed_43f2_9b29_9b3bbe2af9d4.slice/crio-8588869ea3fb5a6027cf2e1ec6948875d213f54bb8d6ad908340a5f0ac32eea9 WatchSource:0}: Error finding container 8588869ea3fb5a6027cf2e1ec6948875d213f54bb8d6ad908340a5f0ac32eea9: Status 404 returned error can't find the container with id 8588869ea3fb5a6027cf2e1ec6948875d213f54bb8d6ad908340a5f0ac32eea9
Sep 10 19:29:27.434725 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:27.434689    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-88n8f"]
Sep 10 19:29:27.663084 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:27.663047    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-f15c09/auth-test-model-f15c09-kserve-6985648bdc-h7p6q" event={"ID":"96d85a15-afed-43f2-9b29-9b3bbe2af9d4","Type":"ContainerStarted","Data":"e973785bd025fd20f591be0ec98009aa9ffa1fa9ca49176b15f6db10e5a2fa8b"}
Sep 10 19:29:27.663272 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:27.663096    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-f15c09/auth-test-model-f15c09-kserve-6985648bdc-h7p6q" event={"ID":"96d85a15-afed-43f2-9b29-9b3bbe2af9d4","Type":"ContainerStarted","Data":"8588869ea3fb5a6027cf2e1ec6948875d213f54bb8d6ad908340a5f0ac32eea9"}
Sep 10 19:29:27.685924 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:27.685821    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-f15c09/auth-test-model-f15c09-kserve-6985648bdc-h7p6q" podStartSLOduration=1.6858028269999998 podStartE2EDuration="1.685802827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:29:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:29:27.683603776 +0000 UTC m=+1130.912653802" watchObservedRunningTime="2026-09-10 19:29:27.685802827 +0000 UTC m=+1130.914852852"
Sep 10 19:29:31.965699 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:31.965661    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-84cb5814/e2e-lifecycle-model-84cb5814-kserve-7c6648479f-nvq4r"
Sep 10 19:29:31.965699 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:31.965715    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-84cb5814/e2e-lifecycle-model-84cb5814-kserve-7c6648479f-nvq4r"
Sep 10 19:29:31.978423 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:31.978390    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-84cb5814/e2e-lifecycle-model-84cb5814-kserve-7c6648479f-nvq4r"
Sep 10 19:29:32.702048 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:32.702011    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-84cb5814/e2e-lifecycle-model-84cb5814-kserve-7c6648479f-nvq4r"
Sep 10 19:29:36.743117 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:36.743083    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-f15c09/auth-test-model-f15c09-kserve-6985648bdc-h7p6q"
Sep 10 19:29:36.743589 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:36.743264    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-f15c09/auth-test-model-f15c09-kserve-6985648bdc-h7p6q"
Sep 10 19:29:36.755680 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:36.755658    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-f15c09/auth-test-model-f15c09-kserve-6985648bdc-h7p6q"
Sep 10 19:29:37.727963 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:37.727933    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-f15c09/auth-test-model-f15c09-kserve-6985648bdc-h7p6q"
Sep 10 19:29:43.042230 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:43.042195    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-88n8f"]
Sep 10 19:29:59.543199 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:29:59.543099    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-88n8f"]
Sep 10 19:30:00.217835 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:30:00.217793    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f15c09-29817810-7f8p2"]
Sep 10 19:30:00.223889 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:30:00.223855    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f15c09-29817810-7f8p2"
Sep 10 19:30:00.226991 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:30:00.226965    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kfmmz\""
Sep 10 19:30:00.235569 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:30:00.235544    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f15c09-29817810-7f8p2"
Sep 10 19:30:00.242071 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:30:00.242034    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f15c09-29817810-7f8p2"]
Sep 10 19:30:00.269412 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:30:00.269333    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-84cb5814-29817810-89x6r"]
Sep 10 19:30:00.274032 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:30:00.274005    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-84cb5814-29817810-89x6r"
Sep 10 19:30:00.288585 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:30:00.288388    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-84cb5814-29817810-89x6r"
Sep 10 19:30:00.288729 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:30:00.288627    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-84cb5814-29817810-89x6r"]
Sep 10 19:30:00.522189 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:30:00.522156    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f15c09-29817810-7f8p2"]
Sep 10 19:30:00.524006 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:30:00.523984    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf87606e_cc4f_4fe3_a023_0cfa667edee6.slice/crio-ceabc5b2084f547af63a9899f260a1c514d3c872535199a64ab6695ce8923717 WatchSource:0}: Error finding container ceabc5b2084f547af63a9899f260a1c514d3c872535199a64ab6695ce8923717: Status 404 returned error can't find the container with id ceabc5b2084f547af63a9899f260a1c514d3c872535199a64ab6695ce8923717
Sep 10 19:30:00.579910 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:30:00.579877    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-84cb5814-29817810-89x6r"]
Sep 10 19:30:00.580893 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:30:00.580864    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1899c7a1_1cee_4c5a_af45_fcbcef2a7d1f.slice/crio-421f13a980ff334a8b5ddcdcc7161d638e5e8b7850a40ab51d64dddabdcdd61d WatchSource:0}: Error finding container 421f13a980ff334a8b5ddcdcc7161d638e5e8b7850a40ab51d64dddabdcdd61d: Status 404 returned error can't find the container with id 421f13a980ff334a8b5ddcdcc7161d638e5e8b7850a40ab51d64dddabdcdd61d
Sep 10 19:30:00.838991 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:30:00.838901    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-84cb5814-29817810-89x6r" event={"ID":"1899c7a1-1cee-4c5a-af45-fcbcef2a7d1f","Type":"ContainerStarted","Data":"421f13a980ff334a8b5ddcdcc7161d638e5e8b7850a40ab51d64dddabdcdd61d"}
Sep 10 19:30:00.840119 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:30:00.840089    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f15c09-29817810-7f8p2" event={"ID":"bf87606e-cc4f-4fe3-a023-0cfa667edee6","Type":"ContainerStarted","Data":"ceabc5b2084f547af63a9899f260a1c514d3c872535199a64ab6695ce8923717"}
Sep 10 19:30:03.861058 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:30:03.861018    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-84cb5814-29817810-89x6r" event={"ID":"1899c7a1-1cee-4c5a-af45-fcbcef2a7d1f","Type":"ContainerStarted","Data":"e58ab58ee4f59578b1fec31a6b4ef55bdb5aa59f6040079c8dd5fe70b1451759"}
Sep 10 19:30:03.863464 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:30:03.863438    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f15c09-29817810-7f8p2" event={"ID":"bf87606e-cc4f-4fe3-a023-0cfa667edee6","Type":"ContainerStarted","Data":"18e98e09473fe27910dc0922629ef152a9146e6d743c60a08434698da03e2283"}
Sep 10 19:30:03.879835 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:30:03.879783    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-84cb5814-29817810-89x6r" podStartSLOduration=1.5518141669999999 podStartE2EDuration="3.87976651s" totalImagesPullingTime="2.327952343s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:30:03.878957582 +0000 UTC m=+1167.108007609" watchObservedRunningTime="2026-09-10 19:30:03.87976651 +0000 UTC m=+1167.108816536"
Sep 10 19:30:03.898552 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:30:03.898501    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f15c09-29817810-7f8p2" podStartSLOduration=1.518496829 podStartE2EDuration="3.898484261s" totalImagesPullingTime="2.379987432s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:30:03.895947179 +0000 UTC m=+1167.124997203" watchObservedRunningTime="2026-09-10 19:30:03.898484261 +0000 UTC m=+1167.127534286"
Sep 10 19:30:04.870841 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:30:04.870802    2546 generic.go:356] "Generic (PLEG): container finished" podID="1899c7a1-1cee-4c5a-af45-fcbcef2a7d1f" containerID="e58ab58ee4f59578b1fec31a6b4ef55bdb5aa59f6040079c8dd5fe70b1451759" exitCode=0
Sep 10 19:30:04.871300 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:30:04.870899    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-84cb5814-29817810-89x6r" event={"ID":"1899c7a1-1cee-4c5a-af45-fcbcef2a7d1f","Type":"ContainerDied","Data":"e58ab58ee4f59578b1fec31a6b4ef55bdb5aa59f6040079c8dd5fe70b1451759"}
Sep 10 19:30:04.872511 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:30:04.872487    2546 generic.go:356] "Generic (PLEG): container finished" podID="bf87606e-cc4f-4fe3-a023-0cfa667edee6" containerID="18e98e09473fe27910dc0922629ef152a9146e6d743c60a08434698da03e2283" exitCode=0
Sep 10 19:30:04.872632 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:30:04.872536    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f15c09-29817810-7f8p2" event={"ID":"bf87606e-cc4f-4fe3-a023-0cfa667edee6","Type":"ContainerDied","Data":"18e98e09473fe27910dc0922629ef152a9146e6d743c60a08434698da03e2283"}
Sep 10 19:30:05.887960 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:30:05.887926    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-84cb5814-29817810-89x6r"]
Sep 10 19:30:06.065413 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:30:06.065388    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f15c09-29817810-7f8p2"
Sep 10 19:30:06.070252 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:30:06.070229    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-84cb5814-29817810-89x6r"
Sep 10 19:30:06.887450 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:30:06.887420    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-84cb5814-29817810-89x6r"
Sep 10 19:30:06.887648 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:30:06.887421    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-84cb5814-29817810-89x6r" event={"ID":"1899c7a1-1cee-4c5a-af45-fcbcef2a7d1f","Type":"ContainerDied","Data":"421f13a980ff334a8b5ddcdcc7161d638e5e8b7850a40ab51d64dddabdcdd61d"}
Sep 10 19:30:06.887648 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:30:06.887563    2546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="421f13a980ff334a8b5ddcdcc7161d638e5e8b7850a40ab51d64dddabdcdd61d"
Sep 10 19:30:06.888777 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:30:06.888755    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f15c09-29817810-7f8p2" event={"ID":"bf87606e-cc4f-4fe3-a023-0cfa667edee6","Type":"ContainerDied","Data":"ceabc5b2084f547af63a9899f260a1c514d3c872535199a64ab6695ce8923717"}
Sep 10 19:30:06.889118 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:30:06.888795    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f15c09-29817810-7f8p2"
Sep 10 19:30:06.889118 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:30:06.888797    2546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ceabc5b2084f547af63a9899f260a1c514d3c872535199a64ab6695ce8923717"
Sep 10 19:30:06.952905 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:30:06.952858    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-84cb5814-29817810-89x6r"]
Sep 10 19:30:06.972160 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:30:06.972127    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-84cb5814-29817810-89x6r"]
Sep 10 19:30:07.281394 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:30:07.281304    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1899c7a1-1cee-4c5a-af45-fcbcef2a7d1f" path="/var/lib/kubelet/pods/1899c7a1-1cee-4c5a-af45-fcbcef2a7d1f/volumes"
Sep 10 19:30:14.201324 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:30:14.201280    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-84cb5814/e2e-lifecycle-model-84cb5814-kserve-7c6648479f-nvq4r"]
Sep 10 19:30:18.236959 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:30:18.236923    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-88n8f"]
Sep 10 19:30:29.220996 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:30:29.220955    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-84cb5814/e2e-lifecycle-model-84cb5814-kserve-7c6648479f-nvq4r" podUID="a04a73d6-3ef5-4a52-bff1-7b27bdfcf842" containerName="main" containerID="cri-o://88b8322312ae01a7b25e20f2cad6326714a97037464959e9a3e9ee3c18f09231" gracePeriod=45
Sep 10 19:30:29.481289 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:30:29.481231    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-84cb5814/e2e-lifecycle-model-84cb5814-kserve-7c6648479f-nvq4r"
Sep 10 19:30:29.633373 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:30:29.633332    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a04a73d6-3ef5-4a52-bff1-7b27bdfcf842-model-cache\") pod \"a04a73d6-3ef5-4a52-bff1-7b27bdfcf842\" (UID: \"a04a73d6-3ef5-4a52-bff1-7b27bdfcf842\") "
Sep 10 19:30:29.633579 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:30:29.633393    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dzknc\" (UniqueName: \"kubernetes.io/projected/a04a73d6-3ef5-4a52-bff1-7b27bdfcf842-kube-api-access-dzknc\") pod \"a04a73d6-3ef5-4a52-bff1-7b27bdfcf842\" (UID: \"a04a73d6-3ef5-4a52-bff1-7b27bdfcf842\") "
Sep 10 19:30:29.633579 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:30:29.633440    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a04a73d6-3ef5-4a52-bff1-7b27bdfcf842-dshm\") pod \"a04a73d6-3ef5-4a52-bff1-7b27bdfcf842\" (UID: \"a04a73d6-3ef5-4a52-bff1-7b27bdfcf842\") "
Sep 10 19:30:29.633579 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:30:29.633476    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a04a73d6-3ef5-4a52-bff1-7b27bdfcf842-tls-certs\") pod \"a04a73d6-3ef5-4a52-bff1-7b27bdfcf842\" (UID: \"a04a73d6-3ef5-4a52-bff1-7b27bdfcf842\") "
Sep 10 19:30:29.633702 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:30:29.633563    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a04a73d6-3ef5-4a52-bff1-7b27bdfcf842-model-cache" (OuterVolumeSpecName: "model-cache") pod "a04a73d6-3ef5-4a52-bff1-7b27bdfcf842" (UID: "a04a73d6-3ef5-4a52-bff1-7b27bdfcf842"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:30:29.633702 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:30:29.633585    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a04a73d6-3ef5-4a52-bff1-7b27bdfcf842-tmp-dir\") pod \"a04a73d6-3ef5-4a52-bff1-7b27bdfcf842\" (UID: \"a04a73d6-3ef5-4a52-bff1-7b27bdfcf842\") "
Sep 10 19:30:29.633702 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:30:29.633635    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a04a73d6-3ef5-4a52-bff1-7b27bdfcf842-home\") pod \"a04a73d6-3ef5-4a52-bff1-7b27bdfcf842\" (UID: \"a04a73d6-3ef5-4a52-bff1-7b27bdfcf842\") "
Sep 10 19:30:29.633823 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:30:29.633786    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a04a73d6-3ef5-4a52-bff1-7b27bdfcf842-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a04a73d6-3ef5-4a52-bff1-7b27bdfcf842" (UID: "a04a73d6-3ef5-4a52-bff1-7b27bdfcf842"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:30:29.633906 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:30:29.633883    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a04a73d6-3ef5-4a52-bff1-7b27bdfcf842-home" (OuterVolumeSpecName: "home") pod "a04a73d6-3ef5-4a52-bff1-7b27bdfcf842" (UID: "a04a73d6-3ef5-4a52-bff1-7b27bdfcf842"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:30:29.634112 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:30:29.634093    2546 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a04a73d6-3ef5-4a52-bff1-7b27bdfcf842-model-cache\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:30:29.634204 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:30:29.634117    2546 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a04a73d6-3ef5-4a52-bff1-7b27bdfcf842-tmp-dir\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:30:29.634204 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:30:29.634129    2546 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a04a73d6-3ef5-4a52-bff1-7b27bdfcf842-home\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:30:29.635730 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:30:29.635705    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a04a73d6-3ef5-4a52-bff1-7b27bdfcf842-dshm" (OuterVolumeSpecName: "dshm") pod "a04a73d6-3ef5-4a52-bff1-7b27bdfcf842" (UID: "a04a73d6-3ef5-4a52-bff1-7b27bdfcf842"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:30:29.635846 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:30:29.635790    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a04a73d6-3ef5-4a52-bff1-7b27bdfcf842-kube-api-access-dzknc" (OuterVolumeSpecName: "kube-api-access-dzknc") pod "a04a73d6-3ef5-4a52-bff1-7b27bdfcf842" (UID: "a04a73d6-3ef5-4a52-bff1-7b27bdfcf842"). InnerVolumeSpecName "kube-api-access-dzknc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 19:30:29.635891 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:30:29.635856    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a04a73d6-3ef5-4a52-bff1-7b27bdfcf842-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a04a73d6-3ef5-4a52-bff1-7b27bdfcf842" (UID: "a04a73d6-3ef5-4a52-bff1-7b27bdfcf842"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 19:30:29.735399 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:30:29.735310    2546 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a04a73d6-3ef5-4a52-bff1-7b27bdfcf842-dshm\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:30:29.735399 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:30:29.735372    2546 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a04a73d6-3ef5-4a52-bff1-7b27bdfcf842-tls-certs\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:30:29.735399 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:30:29.735384    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzknc\" (UniqueName: \"kubernetes.io/projected/a04a73d6-3ef5-4a52-bff1-7b27bdfcf842-kube-api-access-dzknc\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:30:30.007929 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:30:30.007842    2546 generic.go:356] "Generic (PLEG): container finished" podID="a04a73d6-3ef5-4a52-bff1-7b27bdfcf842" containerID="88b8322312ae01a7b25e20f2cad6326714a97037464959e9a3e9ee3c18f09231" exitCode=0
Sep 10 19:30:30.007929 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:30:30.007898    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-84cb5814/e2e-lifecycle-model-84cb5814-kserve-7c6648479f-nvq4r"
Sep 10 19:30:30.008111 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:30:30.007924    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-84cb5814/e2e-lifecycle-model-84cb5814-kserve-7c6648479f-nvq4r" event={"ID":"a04a73d6-3ef5-4a52-bff1-7b27bdfcf842","Type":"ContainerDied","Data":"88b8322312ae01a7b25e20f2cad6326714a97037464959e9a3e9ee3c18f09231"}
Sep 10 19:30:30.008111 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:30:30.007966    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-84cb5814/e2e-lifecycle-model-84cb5814-kserve-7c6648479f-nvq4r" event={"ID":"a04a73d6-3ef5-4a52-bff1-7b27bdfcf842","Type":"ContainerDied","Data":"2ff9e283f87875b49f1e15cda2e8edf15191cc8f42f851aa2e36a4568b9d8989"}
Sep 10 19:30:30.008111 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:30:30.007984    2546 scope.go:117] "RemoveContainer" containerID="88b8322312ae01a7b25e20f2cad6326714a97037464959e9a3e9ee3c18f09231"
Sep 10 19:30:30.016834 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:30:30.016816    2546 scope.go:117] "RemoveContainer" containerID="88b8322312ae01a7b25e20f2cad6326714a97037464959e9a3e9ee3c18f09231"
Sep 10 19:30:30.017097 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:30:30.017079    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88b8322312ae01a7b25e20f2cad6326714a97037464959e9a3e9ee3c18f09231\": container with ID starting with 88b8322312ae01a7b25e20f2cad6326714a97037464959e9a3e9ee3c18f09231 not found: ID does not exist" containerID="88b8322312ae01a7b25e20f2cad6326714a97037464959e9a3e9ee3c18f09231"
Sep 10 19:30:30.017150 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:30:30.017104    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88b8322312ae01a7b25e20f2cad6326714a97037464959e9a3e9ee3c18f09231"} err="failed to get container status \"88b8322312ae01a7b25e20f2cad6326714a97037464959e9a3e9ee3c18f09231\": rpc error: code = NotFound desc = could not find container \"88b8322312ae01a7b25e20f2cad6326714a97037464959e9a3e9ee3c18f09231\": container with ID starting with 88b8322312ae01a7b25e20f2cad6326714a97037464959e9a3e9ee3c18f09231 not found: ID does not exist"
Sep 10 19:30:30.031330 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:30:30.031306    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-84cb5814/e2e-lifecycle-model-84cb5814-kserve-7c6648479f-nvq4r"]
Sep 10 19:30:30.033127 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:30:30.033110    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-84cb5814/e2e-lifecycle-model-84cb5814-kserve-7c6648479f-nvq4r"]
Sep 10 19:30:31.281047 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:30:31.281013    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a04a73d6-3ef5-4a52-bff1-7b27bdfcf842" path="/var/lib/kubelet/pods/a04a73d6-3ef5-4a52-bff1-7b27bdfcf842/volumes"
Sep 10 19:30:32.528721 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:30:32.528686    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-88n8f"]
Sep 10 19:30:44.224174 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:30:44.224138    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-88n8f"]
Sep 10 19:31:02.935432 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:31:02.935382    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-88n8f"]
Sep 10 19:31:19.419117 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:31:19.419078    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-88n8f"]
Sep 10 19:31:20.949828 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:31:20.949785    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-84cb5814-openshift-default-68f49b5764-ncwcg"]
Sep 10 19:31:20.950234 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:31:20.950067    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-84cb5814-openshift-default-68f49b5764-ncwcg" podUID="caf8ba4f-361d-467b-b6a5-26dd763612a3" containerName="istio-proxy" containerID="cri-o://0d53510ee61b8338718e8bd3ceca5ff6a29bf5523b94115253e6c2152fb89ec0" gracePeriod=30
Sep 10 19:31:26.202008 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:31:26.201982    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-84cb5814-openshift-default-68f49b5764-ncwcg"
Sep 10 19:31:26.302177 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:31:26.302086    2546 generic.go:356] "Generic (PLEG): container finished" podID="caf8ba4f-361d-467b-b6a5-26dd763612a3" containerID="0d53510ee61b8338718e8bd3ceca5ff6a29bf5523b94115253e6c2152fb89ec0" exitCode=0
Sep 10 19:31:26.302177 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:31:26.302133    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-84cb5814-openshift-default-68f49b5764-ncwcg"
Sep 10 19:31:26.302177 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:31:26.302163    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-84cb5814-openshift-default-68f49b5764-ncwcg" event={"ID":"caf8ba4f-361d-467b-b6a5-26dd763612a3","Type":"ContainerDied","Data":"0d53510ee61b8338718e8bd3ceca5ff6a29bf5523b94115253e6c2152fb89ec0"}
Sep 10 19:31:26.302453 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:31:26.302209    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-84cb5814-openshift-default-68f49b5764-ncwcg" event={"ID":"caf8ba4f-361d-467b-b6a5-26dd763612a3","Type":"ContainerDied","Data":"a9b1ca35fcf9e775344ce57388aeec8751f276a4c499666022ccb09b53624e4b"}
Sep 10 19:31:26.302453 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:31:26.302232    2546 scope.go:117] "RemoveContainer" containerID="0d53510ee61b8338718e8bd3ceca5ff6a29bf5523b94115253e6c2152fb89ec0"
Sep 10 19:31:26.302697 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:31:26.302675    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caf8ba4f-361d-467b-b6a5-26dd763612a3-istio-data\") pod \"caf8ba4f-361d-467b-b6a5-26dd763612a3\" (UID: \"caf8ba4f-361d-467b-b6a5-26dd763612a3\") "
Sep 10 19:31:26.302741 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:31:26.302716    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/caf8ba4f-361d-467b-b6a5-26dd763612a3-istio-token\") pod \"caf8ba4f-361d-467b-b6a5-26dd763612a3\" (UID: \"caf8ba4f-361d-467b-b6a5-26dd763612a3\") "
Sep 10 19:31:26.302791 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:31:26.302744    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caf8ba4f-361d-467b-b6a5-26dd763612a3-istiod-ca-cert\") pod \"caf8ba4f-361d-467b-b6a5-26dd763612a3\" (UID: \"caf8ba4f-361d-467b-b6a5-26dd763612a3\") "
Sep 10 19:31:26.302791 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:31:26.302767    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caf8ba4f-361d-467b-b6a5-26dd763612a3-workload-socket\") pod \"caf8ba4f-361d-467b-b6a5-26dd763612a3\" (UID: \"caf8ba4f-361d-467b-b6a5-26dd763612a3\") "
Sep 10 19:31:26.302906 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:31:26.302819    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caf8ba4f-361d-467b-b6a5-26dd763612a3-istio-envoy\") pod \"caf8ba4f-361d-467b-b6a5-26dd763612a3\" (UID: \"caf8ba4f-361d-467b-b6a5-26dd763612a3\") "
Sep 10 19:31:26.302906 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:31:26.302854    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rm9m2\" (UniqueName: \"kubernetes.io/projected/caf8ba4f-361d-467b-b6a5-26dd763612a3-kube-api-access-rm9m2\") pod \"caf8ba4f-361d-467b-b6a5-26dd763612a3\" (UID: \"caf8ba4f-361d-467b-b6a5-26dd763612a3\") "
Sep 10 19:31:26.303011 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:31:26.302943    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caf8ba4f-361d-467b-b6a5-26dd763612a3-istio-podinfo\") pod \"caf8ba4f-361d-467b-b6a5-26dd763612a3\" (UID: \"caf8ba4f-361d-467b-b6a5-26dd763612a3\") "
Sep 10 19:31:26.303011 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:31:26.302960    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caf8ba4f-361d-467b-b6a5-26dd763612a3-istio-data" (OuterVolumeSpecName: "istio-data") pod "caf8ba4f-361d-467b-b6a5-26dd763612a3" (UID: "caf8ba4f-361d-467b-b6a5-26dd763612a3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:31:26.303118 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:31:26.303014    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caf8ba4f-361d-467b-b6a5-26dd763612a3-credential-socket\") pod \"caf8ba4f-361d-467b-b6a5-26dd763612a3\" (UID: \"caf8ba4f-361d-467b-b6a5-26dd763612a3\") "
Sep 10 19:31:26.303118 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:31:26.303081    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caf8ba4f-361d-467b-b6a5-26dd763612a3-workload-certs\") pod \"caf8ba4f-361d-467b-b6a5-26dd763612a3\" (UID: \"caf8ba4f-361d-467b-b6a5-26dd763612a3\") "
Sep 10 19:31:26.303235 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:31:26.303180    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caf8ba4f-361d-467b-b6a5-26dd763612a3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "caf8ba4f-361d-467b-b6a5-26dd763612a3" (UID: "caf8ba4f-361d-467b-b6a5-26dd763612a3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:31:26.303235 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:31:26.303191    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/caf8ba4f-361d-467b-b6a5-26dd763612a3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "caf8ba4f-361d-467b-b6a5-26dd763612a3" (UID: "caf8ba4f-361d-467b-b6a5-26dd763612a3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 19:31:26.303429 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:31:26.303408    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caf8ba4f-361d-467b-b6a5-26dd763612a3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "caf8ba4f-361d-467b-b6a5-26dd763612a3" (UID: "caf8ba4f-361d-467b-b6a5-26dd763612a3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:31:26.303514 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:31:26.303495    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caf8ba4f-361d-467b-b6a5-26dd763612a3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "caf8ba4f-361d-467b-b6a5-26dd763612a3" (UID: "caf8ba4f-361d-467b-b6a5-26dd763612a3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:31:26.303658 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:31:26.303614    2546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caf8ba4f-361d-467b-b6a5-26dd763612a3-credential-socket\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:31:26.303658 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:31:26.303635    2546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caf8ba4f-361d-467b-b6a5-26dd763612a3-workload-certs\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:31:26.303658 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:31:26.303651    2546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caf8ba4f-361d-467b-b6a5-26dd763612a3-istio-data\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:31:26.303828 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:31:26.303664    2546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caf8ba4f-361d-467b-b6a5-26dd763612a3-istiod-ca-cert\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:31:26.303828 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:31:26.303677    2546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caf8ba4f-361d-467b-b6a5-26dd763612a3-workload-socket\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:31:26.305257 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:31:26.305222    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/caf8ba4f-361d-467b-b6a5-26dd763612a3-istio-token" (OuterVolumeSpecName: "istio-token") pod "caf8ba4f-361d-467b-b6a5-26dd763612a3" (UID: "caf8ba4f-361d-467b-b6a5-26dd763612a3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 19:31:26.305513 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:31:26.305490    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/caf8ba4f-361d-467b-b6a5-26dd763612a3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "caf8ba4f-361d-467b-b6a5-26dd763612a3" (UID: "caf8ba4f-361d-467b-b6a5-26dd763612a3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 19:31:26.305813 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:31:26.305793    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caf8ba4f-361d-467b-b6a5-26dd763612a3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "caf8ba4f-361d-467b-b6a5-26dd763612a3" (UID: "caf8ba4f-361d-467b-b6a5-26dd763612a3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:31:26.305813 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:31:26.305793    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/caf8ba4f-361d-467b-b6a5-26dd763612a3-kube-api-access-rm9m2" (OuterVolumeSpecName: "kube-api-access-rm9m2") pod "caf8ba4f-361d-467b-b6a5-26dd763612a3" (UID: "caf8ba4f-361d-467b-b6a5-26dd763612a3"). InnerVolumeSpecName "kube-api-access-rm9m2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 19:31:26.324276 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:31:26.324249    2546 scope.go:117] "RemoveContainer" containerID="61386ca02c5de93564ecff8e97a82357191d65f4bed67e20b7c039f5ed8f994b"
Sep 10 19:31:26.332824 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:31:26.332802    2546 scope.go:117] "RemoveContainer" containerID="0d53510ee61b8338718e8bd3ceca5ff6a29bf5523b94115253e6c2152fb89ec0"
Sep 10 19:31:26.333119 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:31:26.333100    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d53510ee61b8338718e8bd3ceca5ff6a29bf5523b94115253e6c2152fb89ec0\": container with ID starting with 0d53510ee61b8338718e8bd3ceca5ff6a29bf5523b94115253e6c2152fb89ec0 not found: ID does not exist" containerID="0d53510ee61b8338718e8bd3ceca5ff6a29bf5523b94115253e6c2152fb89ec0"
Sep 10 19:31:26.333185 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:31:26.333134    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d53510ee61b8338718e8bd3ceca5ff6a29bf5523b94115253e6c2152fb89ec0"} err="failed to get container status \"0d53510ee61b8338718e8bd3ceca5ff6a29bf5523b94115253e6c2152fb89ec0\": rpc error: code = NotFound desc = could not find container \"0d53510ee61b8338718e8bd3ceca5ff6a29bf5523b94115253e6c2152fb89ec0\": container with ID starting with 0d53510ee61b8338718e8bd3ceca5ff6a29bf5523b94115253e6c2152fb89ec0 not found: ID does not exist"
Sep 10 19:31:26.333185 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:31:26.333163    2546 scope.go:117] "RemoveContainer" containerID="61386ca02c5de93564ecff8e97a82357191d65f4bed67e20b7c039f5ed8f994b"
Sep 10 19:31:26.333485 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:31:26.333464    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61386ca02c5de93564ecff8e97a82357191d65f4bed67e20b7c039f5ed8f994b\": container with ID starting with 61386ca02c5de93564ecff8e97a82357191d65f4bed67e20b7c039f5ed8f994b not found: ID does not exist" containerID="61386ca02c5de93564ecff8e97a82357191d65f4bed67e20b7c039f5ed8f994b"
Sep 10 19:31:26.333536 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:31:26.333491    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61386ca02c5de93564ecff8e97a82357191d65f4bed67e20b7c039f5ed8f994b"} err="failed to get container status \"61386ca02c5de93564ecff8e97a82357191d65f4bed67e20b7c039f5ed8f994b\": rpc error: code = NotFound desc = could not find container \"61386ca02c5de93564ecff8e97a82357191d65f4bed67e20b7c039f5ed8f994b\": container with ID starting with 61386ca02c5de93564ecff8e97a82357191d65f4bed67e20b7c039f5ed8f994b not found: ID does not exist"
Sep 10 19:31:26.404813 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:31:26.404774    2546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/caf8ba4f-361d-467b-b6a5-26dd763612a3-istio-token\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:31:26.404813 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:31:26.404803    2546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caf8ba4f-361d-467b-b6a5-26dd763612a3-istio-envoy\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:31:26.404813 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:31:26.404812    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rm9m2\" (UniqueName: \"kubernetes.io/projected/caf8ba4f-361d-467b-b6a5-26dd763612a3-kube-api-access-rm9m2\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:31:26.404813 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:31:26.404822    2546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caf8ba4f-361d-467b-b6a5-26dd763612a3-istio-podinfo\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:31:26.625555 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:31:26.625526    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-84cb5814-openshift-default-68f49b5764-ncwcg"]
Sep 10 19:31:26.629661 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:31:26.629635    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-84cb5814-openshift-default-68f49b5764-ncwcg"]
Sep 10 19:31:27.281031 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:31:27.280993    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="caf8ba4f-361d-467b-b6a5-26dd763612a3" path="/var/lib/kubelet/pods/caf8ba4f-361d-467b-b6a5-26dd763612a3/volumes"
Sep 10 19:31:35.415794 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:31:35.415708    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-88n8f"]
Sep 10 19:31:46.061631 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:31:46.061594    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7528-e027fe"]
Sep 10 19:31:46.065242 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:31:46.065211    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a04a73d6-3ef5-4a52-bff1-7b27bdfcf842" containerName="main"
Sep 10 19:31:46.065242 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:31:46.065238    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="a04a73d6-3ef5-4a52-bff1-7b27bdfcf842" containerName="main"
Sep 10 19:31:46.065242 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:31:46.065250    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf87606e-cc4f-4fe3-a023-0cfa667edee6" containerName="cleanup"
Sep 10 19:31:46.065470 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:31:46.065273    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf87606e-cc4f-4fe3-a023-0cfa667edee6" containerName="cleanup"
Sep 10 19:31:46.065470 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:31:46.065296    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="caf8ba4f-361d-467b-b6a5-26dd763612a3" containerName="istio-proxy"
Sep 10 19:31:46.065470 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:31:46.065301    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="caf8ba4f-361d-467b-b6a5-26dd763612a3" containerName="istio-proxy"
Sep 10 19:31:46.065470 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:31:46.065307    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1899c7a1-1cee-4c5a-af45-fcbcef2a7d1f" containerName="cleanup"
Sep 10 19:31:46.065470 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:31:46.065312    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="1899c7a1-1cee-4c5a-af45-fcbcef2a7d1f" containerName="cleanup"
Sep 10 19:31:46.065470 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:31:46.065319    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="caf8ba4f-361d-467b-b6a5-26dd763612a3" containerName="istio-proxy"
Sep 10 19:31:46.065470 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:31:46.065325    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="caf8ba4f-361d-467b-b6a5-26dd763612a3" containerName="istio-proxy"
Sep 10 19:31:46.065470 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:31:46.065406    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf87606e-cc4f-4fe3-a023-0cfa667edee6" containerName="cleanup"
Sep 10 19:31:46.065470 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:31:46.065417    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="1899c7a1-1cee-4c5a-af45-fcbcef2a7d1f" containerName="cleanup"
Sep 10 19:31:46.065470 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:31:46.065424    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="caf8ba4f-361d-467b-b6a5-26dd763612a3" containerName="istio-proxy"
Sep 10 19:31:46.065470 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:31:46.065429    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="a04a73d6-3ef5-4a52-bff1-7b27bdfcf842" containerName="main"
Sep 10 19:31:46.065470 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:31:46.065434    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="caf8ba4f-361d-467b-b6a5-26dd763612a3" containerName="istio-proxy"
Sep 10 19:31:46.067831 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:31:46.067815    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7528-e027fe"
Sep 10 19:31:46.070713 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:31:46.070689    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-sbhkd\""
Sep 10 19:31:46.075375 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:31:46.075068    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7528-e027fe"]
Sep 10 19:31:46.205730 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:31:46.205688    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7nzb\" (UniqueName: \"kubernetes.io/projected/f5a147f3-b3fc-48d0-8ebb-1e12352cd8fd-kube-api-access-q7nzb\") pod \"test-curl-7528-e027fe\" (UID: \"f5a147f3-b3fc-48d0-8ebb-1e12352cd8fd\") " pod="opendatahub/test-curl-7528-e027fe"
Sep 10 19:31:46.306605 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:31:46.306562    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7nzb\" (UniqueName: \"kubernetes.io/projected/f5a147f3-b3fc-48d0-8ebb-1e12352cd8fd-kube-api-access-q7nzb\") pod \"test-curl-7528-e027fe\" (UID: \"f5a147f3-b3fc-48d0-8ebb-1e12352cd8fd\") " pod="opendatahub/test-curl-7528-e027fe"
Sep 10 19:31:46.316745 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:31:46.316675    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7nzb\" (UniqueName: \"kubernetes.io/projected/f5a147f3-b3fc-48d0-8ebb-1e12352cd8fd-kube-api-access-q7nzb\") pod \"test-curl-7528-e027fe\" (UID: \"f5a147f3-b3fc-48d0-8ebb-1e12352cd8fd\") " pod="opendatahub/test-curl-7528-e027fe"
Sep 10 19:31:46.377333 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:31:46.377300    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7528-e027fe"
Sep 10 19:31:46.647612 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:31:46.647488    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7528-e027fe"]
Sep 10 19:31:46.649826 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:31:46.649798    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf5a147f3_b3fc_48d0_8ebb_1e12352cd8fd.slice/crio-b6810e0be467a10734f38e36be0d7bf0d0291b10fd71e4d7b11d7f89c501f842 WatchSource:0}: Error finding container b6810e0be467a10734f38e36be0d7bf0d0291b10fd71e4d7b11d7f89c501f842: Status 404 returned error can't find the container with id b6810e0be467a10734f38e36be0d7bf0d0291b10fd71e4d7b11d7f89c501f842
Sep 10 19:31:47.420775 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:31:47.420734    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7528-e027fe" event={"ID":"f5a147f3-b3fc-48d0-8ebb-1e12352cd8fd","Type":"ContainerStarted","Data":"84b94b264697853e946f7103bce63ae21ff2e932e3f4390c1a4c2827cd981be4"}
Sep 10 19:31:47.421177 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:31:47.420784    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7528-e027fe" event={"ID":"f5a147f3-b3fc-48d0-8ebb-1e12352cd8fd","Type":"ContainerStarted","Data":"b6810e0be467a10734f38e36be0d7bf0d0291b10fd71e4d7b11d7f89c501f842"}
Sep 10 19:31:47.441302 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:31:47.441245    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7528-e027fe" podStartSLOduration=1.441229613 podStartE2EDuration="1.441229613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:31:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:31:47.440998375 +0000 UTC m=+1270.670048400" watchObservedRunningTime="2026-09-10 19:31:47.441229613 +0000 UTC m=+1270.670279638"
Sep 10 19:31:48.086896 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:31:48.086858    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7528-e027fe"]
Sep 10 19:31:48.093021 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:31:48.092978    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7528-e027fe"]
Sep 10 19:31:48.426257 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:31:48.426221    2546 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7528-e027fe" container="test-curl-7528-e027fe"
Sep 10 19:31:49.430757 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:31:49.430717    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7528-e027fe" podUID="f5a147f3-b3fc-48d0-8ebb-1e12352cd8fd" containerName="test-curl-7528-e027fe" containerID="cri-o://84b94b264697853e946f7103bce63ae21ff2e932e3f4390c1a4c2827cd981be4" gracePeriod=2
Sep 10 19:31:49.484123 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:31:49.484086    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-b64b1c/auth-test-model-b64b1c-kserve-5b7d7fb44c-86v4p"]
Sep 10 19:31:50.406127 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:31:50.406093    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-f15c09/auth-test-model-f15c09-kserve-6985648bdc-h7p6q"]
Sep 10 19:31:50.435376 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:31:50.435328    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-88n8f"]
Sep 10 19:31:51.571209 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:31:51.571184    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7528-e027fe"
Sep 10 19:31:51.759376 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:31:51.759263    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q7nzb\" (UniqueName: \"kubernetes.io/projected/f5a147f3-b3fc-48d0-8ebb-1e12352cd8fd-kube-api-access-q7nzb\") pod \"f5a147f3-b3fc-48d0-8ebb-1e12352cd8fd\" (UID: \"f5a147f3-b3fc-48d0-8ebb-1e12352cd8fd\") "
Sep 10 19:31:51.761440 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:31:51.761407    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5a147f3-b3fc-48d0-8ebb-1e12352cd8fd-kube-api-access-q7nzb" (OuterVolumeSpecName: "kube-api-access-q7nzb") pod "f5a147f3-b3fc-48d0-8ebb-1e12352cd8fd" (UID: "f5a147f3-b3fc-48d0-8ebb-1e12352cd8fd"). InnerVolumeSpecName "kube-api-access-q7nzb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 19:31:51.860304 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:31:51.860259    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7nzb\" (UniqueName: \"kubernetes.io/projected/f5a147f3-b3fc-48d0-8ebb-1e12352cd8fd-kube-api-access-q7nzb\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:31:52.448331 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:31:52.448289    2546 generic.go:356] "Generic (PLEG): container finished" podID="f5a147f3-b3fc-48d0-8ebb-1e12352cd8fd" containerID="84b94b264697853e946f7103bce63ae21ff2e932e3f4390c1a4c2827cd981be4" exitCode=137
Sep 10 19:31:52.448331 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:31:52.448327    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7528-e027fe"
Sep 10 19:31:52.448611 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:31:52.448386    2546 scope.go:117] "RemoveContainer" containerID="84b94b264697853e946f7103bce63ae21ff2e932e3f4390c1a4c2827cd981be4"
Sep 10 19:31:52.456825 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:31:52.456802    2546 scope.go:117] "RemoveContainer" containerID="84b94b264697853e946f7103bce63ae21ff2e932e3f4390c1a4c2827cd981be4"
Sep 10 19:31:52.457126 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:31:52.457110    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84b94b264697853e946f7103bce63ae21ff2e932e3f4390c1a4c2827cd981be4\": container with ID starting with 84b94b264697853e946f7103bce63ae21ff2e932e3f4390c1a4c2827cd981be4 not found: ID does not exist" containerID="84b94b264697853e946f7103bce63ae21ff2e932e3f4390c1a4c2827cd981be4"
Sep 10 19:31:52.457187 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:31:52.457135    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84b94b264697853e946f7103bce63ae21ff2e932e3f4390c1a4c2827cd981be4"} err="failed to get container status \"84b94b264697853e946f7103bce63ae21ff2e932e3f4390c1a4c2827cd981be4\": rpc error: code = NotFound desc = could not find container \"84b94b264697853e946f7103bce63ae21ff2e932e3f4390c1a4c2827cd981be4\": container with ID starting with 84b94b264697853e946f7103bce63ae21ff2e932e3f4390c1a4c2827cd981be4 not found: ID does not exist"
Sep 10 19:31:52.459123 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:31:52.459097    2546 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7528-e027fe\" is forbidden: User \"system:node:ip-10-0-132-222.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-222.ec2.internal' and this object" podUID="f5a147f3-b3fc-48d0-8ebb-1e12352cd8fd" pod="opendatahub/test-curl-7528-e027fe"
Sep 10 19:31:53.280879 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:31:53.280842    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5a147f3-b3fc-48d0-8ebb-1e12352cd8fd" path="/var/lib/kubelet/pods/f5a147f3-b3fc-48d0-8ebb-1e12352cd8fd/volumes"
Sep 10 19:32:01.516492 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:01.516458    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-88n8f"]
Sep 10 19:32:04.504128 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:04.504049    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-b64b1c/auth-test-model-b64b1c-kserve-5b7d7fb44c-86v4p" podUID="1994f618-a7f6-4b01-8856-f4fa930568ae" containerName="main" containerID="cri-o://c05f32058e5cff033080abd9072b7342a08ee724b2582ce2b2054c5a7b90e5c8" gracePeriod=45
Sep 10 19:32:04.763643 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:04.763576    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-b64b1c/auth-test-model-b64b1c-kserve-5b7d7fb44c-86v4p"
Sep 10 19:32:04.893748 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:04.893713    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1994f618-a7f6-4b01-8856-f4fa930568ae-home\") pod \"1994f618-a7f6-4b01-8856-f4fa930568ae\" (UID: \"1994f618-a7f6-4b01-8856-f4fa930568ae\") "
Sep 10 19:32:04.893960 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:04.893779    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vfhk9\" (UniqueName: \"kubernetes.io/projected/1994f618-a7f6-4b01-8856-f4fa930568ae-kube-api-access-vfhk9\") pod \"1994f618-a7f6-4b01-8856-f4fa930568ae\" (UID: \"1994f618-a7f6-4b01-8856-f4fa930568ae\") "
Sep 10 19:32:04.893960 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:04.893806    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1994f618-a7f6-4b01-8856-f4fa930568ae-tls-certs\") pod \"1994f618-a7f6-4b01-8856-f4fa930568ae\" (UID: \"1994f618-a7f6-4b01-8856-f4fa930568ae\") "
Sep 10 19:32:04.893960 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:04.893828    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1994f618-a7f6-4b01-8856-f4fa930568ae-tmp-dir\") pod \"1994f618-a7f6-4b01-8856-f4fa930568ae\" (UID: \"1994f618-a7f6-4b01-8856-f4fa930568ae\") "
Sep 10 19:32:04.893960 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:04.893874    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1994f618-a7f6-4b01-8856-f4fa930568ae-model-cache\") pod \"1994f618-a7f6-4b01-8856-f4fa930568ae\" (UID: \"1994f618-a7f6-4b01-8856-f4fa930568ae\") "
Sep 10 19:32:04.893960 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:04.893945    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1994f618-a7f6-4b01-8856-f4fa930568ae-dshm\") pod \"1994f618-a7f6-4b01-8856-f4fa930568ae\" (UID: \"1994f618-a7f6-4b01-8856-f4fa930568ae\") "
Sep 10 19:32:04.894222 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:04.894052    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1994f618-a7f6-4b01-8856-f4fa930568ae-home" (OuterVolumeSpecName: "home") pod "1994f618-a7f6-4b01-8856-f4fa930568ae" (UID: "1994f618-a7f6-4b01-8856-f4fa930568ae"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:32:04.894222 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:04.894163    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1994f618-a7f6-4b01-8856-f4fa930568ae-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1994f618-a7f6-4b01-8856-f4fa930568ae" (UID: "1994f618-a7f6-4b01-8856-f4fa930568ae"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:32:04.894222 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:04.894178    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1994f618-a7f6-4b01-8856-f4fa930568ae-model-cache" (OuterVolumeSpecName: "model-cache") pod "1994f618-a7f6-4b01-8856-f4fa930568ae" (UID: "1994f618-a7f6-4b01-8856-f4fa930568ae"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:32:04.894377 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:04.894329    2546 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1994f618-a7f6-4b01-8856-f4fa930568ae-home\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:32:04.894452 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:04.894375    2546 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1994f618-a7f6-4b01-8856-f4fa930568ae-tmp-dir\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:32:04.894452 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:04.894390    2546 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1994f618-a7f6-4b01-8856-f4fa930568ae-model-cache\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:32:04.896131 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:04.896108    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1994f618-a7f6-4b01-8856-f4fa930568ae-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1994f618-a7f6-4b01-8856-f4fa930568ae" (UID: "1994f618-a7f6-4b01-8856-f4fa930568ae"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 19:32:04.896255 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:04.896177    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1994f618-a7f6-4b01-8856-f4fa930568ae-dshm" (OuterVolumeSpecName: "dshm") pod "1994f618-a7f6-4b01-8856-f4fa930568ae" (UID: "1994f618-a7f6-4b01-8856-f4fa930568ae"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:32:04.896255 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:04.896194    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1994f618-a7f6-4b01-8856-f4fa930568ae-kube-api-access-vfhk9" (OuterVolumeSpecName: "kube-api-access-vfhk9") pod "1994f618-a7f6-4b01-8856-f4fa930568ae" (UID: "1994f618-a7f6-4b01-8856-f4fa930568ae"). InnerVolumeSpecName "kube-api-access-vfhk9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 19:32:04.995786 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:04.995742    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfhk9\" (UniqueName: \"kubernetes.io/projected/1994f618-a7f6-4b01-8856-f4fa930568ae-kube-api-access-vfhk9\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:32:04.995786 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:04.995780    2546 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1994f618-a7f6-4b01-8856-f4fa930568ae-tls-certs\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:32:04.995786 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:04.995790    2546 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1994f618-a7f6-4b01-8856-f4fa930568ae-dshm\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:32:05.427194 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:05.427147    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-f15c09/auth-test-model-f15c09-kserve-6985648bdc-h7p6q" podUID="96d85a15-afed-43f2-9b29-9b3bbe2af9d4" containerName="main" containerID="cri-o://e973785bd025fd20f591be0ec98009aa9ffa1fa9ca49176b15f6db10e5a2fa8b" gracePeriod=45
Sep 10 19:32:05.516510 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:05.516476    2546 generic.go:356] "Generic (PLEG): container finished" podID="1994f618-a7f6-4b01-8856-f4fa930568ae" containerID="c05f32058e5cff033080abd9072b7342a08ee724b2582ce2b2054c5a7b90e5c8" exitCode=0
Sep 10 19:32:05.516903 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:05.516527    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-b64b1c/auth-test-model-b64b1c-kserve-5b7d7fb44c-86v4p"
Sep 10 19:32:05.516903 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:05.516542    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-b64b1c/auth-test-model-b64b1c-kserve-5b7d7fb44c-86v4p" event={"ID":"1994f618-a7f6-4b01-8856-f4fa930568ae","Type":"ContainerDied","Data":"c05f32058e5cff033080abd9072b7342a08ee724b2582ce2b2054c5a7b90e5c8"}
Sep 10 19:32:05.516903 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:05.516600    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-b64b1c/auth-test-model-b64b1c-kserve-5b7d7fb44c-86v4p" event={"ID":"1994f618-a7f6-4b01-8856-f4fa930568ae","Type":"ContainerDied","Data":"820cadfd6bd25a58c916405ea0ef68b5f195cb8cc1a1d5f1bb1d28f1fb40c839"}
Sep 10 19:32:05.516903 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:05.516625    2546 scope.go:117] "RemoveContainer" containerID="c05f32058e5cff033080abd9072b7342a08ee724b2582ce2b2054c5a7b90e5c8"
Sep 10 19:32:05.535042 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:05.535014    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-b64b1c/auth-test-model-b64b1c-kserve-5b7d7fb44c-86v4p"]
Sep 10 19:32:05.539228 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:05.539147    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-b64b1c/auth-test-model-b64b1c-kserve-5b7d7fb44c-86v4p"]
Sep 10 19:32:05.540050 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:05.540028    2546 scope.go:117] "RemoveContainer" containerID="c05f32058e5cff033080abd9072b7342a08ee724b2582ce2b2054c5a7b90e5c8"
Sep 10 19:32:05.540447 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:32:05.540425    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c05f32058e5cff033080abd9072b7342a08ee724b2582ce2b2054c5a7b90e5c8\": container with ID starting with c05f32058e5cff033080abd9072b7342a08ee724b2582ce2b2054c5a7b90e5c8 not found: ID does not exist" containerID="c05f32058e5cff033080abd9072b7342a08ee724b2582ce2b2054c5a7b90e5c8"
Sep 10 19:32:05.540523 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:05.540457    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c05f32058e5cff033080abd9072b7342a08ee724b2582ce2b2054c5a7b90e5c8"} err="failed to get container status \"c05f32058e5cff033080abd9072b7342a08ee724b2582ce2b2054c5a7b90e5c8\": rpc error: code = NotFound desc = could not find container \"c05f32058e5cff033080abd9072b7342a08ee724b2582ce2b2054c5a7b90e5c8\": container with ID starting with c05f32058e5cff033080abd9072b7342a08ee724b2582ce2b2054c5a7b90e5c8 not found: ID does not exist"
Sep 10 19:32:05.687756 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:05.687731    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-f15c09/auth-test-model-f15c09-kserve-6985648bdc-h7p6q"
Sep 10 19:32:05.802866 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:05.802769    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96d85a15-afed-43f2-9b29-9b3bbe2af9d4-dshm\") pod \"96d85a15-afed-43f2-9b29-9b3bbe2af9d4\" (UID: \"96d85a15-afed-43f2-9b29-9b3bbe2af9d4\") "
Sep 10 19:32:05.802866 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:05.802850    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96d85a15-afed-43f2-9b29-9b3bbe2af9d4-model-cache\") pod \"96d85a15-afed-43f2-9b29-9b3bbe2af9d4\" (UID: \"96d85a15-afed-43f2-9b29-9b3bbe2af9d4\") "
Sep 10 19:32:05.803117 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:05.802883    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xnq94\" (UniqueName: \"kubernetes.io/projected/96d85a15-afed-43f2-9b29-9b3bbe2af9d4-kube-api-access-xnq94\") pod \"96d85a15-afed-43f2-9b29-9b3bbe2af9d4\" (UID: \"96d85a15-afed-43f2-9b29-9b3bbe2af9d4\") "
Sep 10 19:32:05.803117 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:05.802931    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96d85a15-afed-43f2-9b29-9b3bbe2af9d4-tmp-dir\") pod \"96d85a15-afed-43f2-9b29-9b3bbe2af9d4\" (UID: \"96d85a15-afed-43f2-9b29-9b3bbe2af9d4\") "
Sep 10 19:32:05.803117 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:05.802972    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96d85a15-afed-43f2-9b29-9b3bbe2af9d4-tls-certs\") pod \"96d85a15-afed-43f2-9b29-9b3bbe2af9d4\" (UID: \"96d85a15-afed-43f2-9b29-9b3bbe2af9d4\") "
Sep 10 19:32:05.803117 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:05.803035    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96d85a15-afed-43f2-9b29-9b3bbe2af9d4-home\") pod \"96d85a15-afed-43f2-9b29-9b3bbe2af9d4\" (UID: \"96d85a15-afed-43f2-9b29-9b3bbe2af9d4\") "
Sep 10 19:32:05.803314 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:05.803179    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96d85a15-afed-43f2-9b29-9b3bbe2af9d4-model-cache" (OuterVolumeSpecName: "model-cache") pod "96d85a15-afed-43f2-9b29-9b3bbe2af9d4" (UID: "96d85a15-afed-43f2-9b29-9b3bbe2af9d4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:32:05.803314 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:05.803301    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96d85a15-afed-43f2-9b29-9b3bbe2af9d4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "96d85a15-afed-43f2-9b29-9b3bbe2af9d4" (UID: "96d85a15-afed-43f2-9b29-9b3bbe2af9d4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:32:05.803458 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:05.803329    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96d85a15-afed-43f2-9b29-9b3bbe2af9d4-home" (OuterVolumeSpecName: "home") pod "96d85a15-afed-43f2-9b29-9b3bbe2af9d4" (UID: "96d85a15-afed-43f2-9b29-9b3bbe2af9d4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:32:05.803515 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:05.803497    2546 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96d85a15-afed-43f2-9b29-9b3bbe2af9d4-model-cache\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:32:05.803515 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:05.803510    2546 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96d85a15-afed-43f2-9b29-9b3bbe2af9d4-tmp-dir\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:32:05.803617 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:05.803519    2546 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96d85a15-afed-43f2-9b29-9b3bbe2af9d4-home\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:32:05.805235 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:05.805206    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96d85a15-afed-43f2-9b29-9b3bbe2af9d4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "96d85a15-afed-43f2-9b29-9b3bbe2af9d4" (UID: "96d85a15-afed-43f2-9b29-9b3bbe2af9d4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 19:32:05.805235 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:05.805227    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96d85a15-afed-43f2-9b29-9b3bbe2af9d4-dshm" (OuterVolumeSpecName: "dshm") pod "96d85a15-afed-43f2-9b29-9b3bbe2af9d4" (UID: "96d85a15-afed-43f2-9b29-9b3bbe2af9d4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:32:05.805405 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:05.805275    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96d85a15-afed-43f2-9b29-9b3bbe2af9d4-kube-api-access-xnq94" (OuterVolumeSpecName: "kube-api-access-xnq94") pod "96d85a15-afed-43f2-9b29-9b3bbe2af9d4" (UID: "96d85a15-afed-43f2-9b29-9b3bbe2af9d4"). InnerVolumeSpecName "kube-api-access-xnq94". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 19:32:05.904242 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:05.904198    2546 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96d85a15-afed-43f2-9b29-9b3bbe2af9d4-dshm\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:32:05.904242 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:05.904236    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnq94\" (UniqueName: \"kubernetes.io/projected/96d85a15-afed-43f2-9b29-9b3bbe2af9d4-kube-api-access-xnq94\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:32:05.904242 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:05.904249    2546 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96d85a15-afed-43f2-9b29-9b3bbe2af9d4-tls-certs\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:32:06.524085 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:06.524046    2546 generic.go:356] "Generic (PLEG): container finished" podID="96d85a15-afed-43f2-9b29-9b3bbe2af9d4" containerID="e973785bd025fd20f591be0ec98009aa9ffa1fa9ca49176b15f6db10e5a2fa8b" exitCode=0
Sep 10 19:32:06.524715 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:06.524100    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-f15c09/auth-test-model-f15c09-kserve-6985648bdc-h7p6q"
Sep 10 19:32:06.524715 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:06.524112    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-f15c09/auth-test-model-f15c09-kserve-6985648bdc-h7p6q" event={"ID":"96d85a15-afed-43f2-9b29-9b3bbe2af9d4","Type":"ContainerDied","Data":"e973785bd025fd20f591be0ec98009aa9ffa1fa9ca49176b15f6db10e5a2fa8b"}
Sep 10 19:32:06.524715 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:06.524142    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-f15c09/auth-test-model-f15c09-kserve-6985648bdc-h7p6q" event={"ID":"96d85a15-afed-43f2-9b29-9b3bbe2af9d4","Type":"ContainerDied","Data":"8588869ea3fb5a6027cf2e1ec6948875d213f54bb8d6ad908340a5f0ac32eea9"}
Sep 10 19:32:06.524715 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:06.524160    2546 scope.go:117] "RemoveContainer" containerID="e973785bd025fd20f591be0ec98009aa9ffa1fa9ca49176b15f6db10e5a2fa8b"
Sep 10 19:32:06.535037 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:06.535014    2546 scope.go:117] "RemoveContainer" containerID="e973785bd025fd20f591be0ec98009aa9ffa1fa9ca49176b15f6db10e5a2fa8b"
Sep 10 19:32:06.535366 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:32:06.535331    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e973785bd025fd20f591be0ec98009aa9ffa1fa9ca49176b15f6db10e5a2fa8b\": container with ID starting with e973785bd025fd20f591be0ec98009aa9ffa1fa9ca49176b15f6db10e5a2fa8b not found: ID does not exist" containerID="e973785bd025fd20f591be0ec98009aa9ffa1fa9ca49176b15f6db10e5a2fa8b"
Sep 10 19:32:06.535442 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:06.535380    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e973785bd025fd20f591be0ec98009aa9ffa1fa9ca49176b15f6db10e5a2fa8b"} err="failed to get container status \"e973785bd025fd20f591be0ec98009aa9ffa1fa9ca49176b15f6db10e5a2fa8b\": rpc error: code = NotFound desc = could not find container \"e973785bd025fd20f591be0ec98009aa9ffa1fa9ca49176b15f6db10e5a2fa8b\": container with ID starting with e973785bd025fd20f591be0ec98009aa9ffa1fa9ca49176b15f6db10e5a2fa8b not found: ID does not exist"
Sep 10 19:32:06.547043 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:06.547008    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-f15c09/auth-test-model-f15c09-kserve-6985648bdc-h7p6q"]
Sep 10 19:32:06.550732 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:06.550704    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-f15c09/auth-test-model-f15c09-kserve-6985648bdc-h7p6q"]
Sep 10 19:32:07.285982 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:07.285948    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1994f618-a7f6-4b01-8856-f4fa930568ae" path="/var/lib/kubelet/pods/1994f618-a7f6-4b01-8856-f4fa930568ae/volumes"
Sep 10 19:32:07.286367 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:07.286334    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="96d85a15-afed-43f2-9b29-9b3bbe2af9d4" path="/var/lib/kubelet/pods/96d85a15-afed-43f2-9b29-9b3bbe2af9d4/volumes"
Sep 10 19:32:11.228928 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:11.228882    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-88n8f"]
Sep 10 19:32:16.134811 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:16.134775    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-cd7dca-openshift-default-654f7c496d-ngm26"]
Sep 10 19:32:16.135196 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:16.135135    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1994f618-a7f6-4b01-8856-f4fa930568ae" containerName="main"
Sep 10 19:32:16.135196 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:16.135147    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="1994f618-a7f6-4b01-8856-f4fa930568ae" containerName="main"
Sep 10 19:32:16.135196 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:16.135156    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5a147f3-b3fc-48d0-8ebb-1e12352cd8fd" containerName="test-curl-7528-e027fe"
Sep 10 19:32:16.135196 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:16.135162    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5a147f3-b3fc-48d0-8ebb-1e12352cd8fd" containerName="test-curl-7528-e027fe"
Sep 10 19:32:16.135196 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:16.135183    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96d85a15-afed-43f2-9b29-9b3bbe2af9d4" containerName="main"
Sep 10 19:32:16.135196 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:16.135189    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="96d85a15-afed-43f2-9b29-9b3bbe2af9d4" containerName="main"
Sep 10 19:32:16.135429 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:16.135244    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="1994f618-a7f6-4b01-8856-f4fa930568ae" containerName="main"
Sep 10 19:32:16.135429 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:16.135251    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="96d85a15-afed-43f2-9b29-9b3bbe2af9d4" containerName="main"
Sep 10 19:32:16.135429 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:16.135260    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5a147f3-b3fc-48d0-8ebb-1e12352cd8fd" containerName="test-curl-7528-e027fe"
Sep 10 19:32:16.139513 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:16.139494    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-cd7dca-openshift-default-654f7c496d-ngm26"
Sep 10 19:32:16.143114 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:16.143089    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-cd7dca-openshift-default-dockercfg-qnfpk\""
Sep 10 19:32:16.156180 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:16.156156    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-cd7dca-openshift-default-654f7c496d-ngm26"]
Sep 10 19:32:16.204089 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:16.204031    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d817b15-2af4-4c24-a737-e2cfa443ed47-workload-certs\") pod \"e2e-api-a-cd7dca-openshift-default-654f7c496d-ngm26\" (UID: \"4d817b15-2af4-4c24-a737-e2cfa443ed47\") " pod="openshift-ingress/e2e-api-a-cd7dca-openshift-default-654f7c496d-ngm26"
Sep 10 19:32:16.204439 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:16.204411    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9tdp\" (UniqueName: \"kubernetes.io/projected/4d817b15-2af4-4c24-a737-e2cfa443ed47-kube-api-access-r9tdp\") pod \"e2e-api-a-cd7dca-openshift-default-654f7c496d-ngm26\" (UID: \"4d817b15-2af4-4c24-a737-e2cfa443ed47\") " pod="openshift-ingress/e2e-api-a-cd7dca-openshift-default-654f7c496d-ngm26"
Sep 10 19:32:16.204612 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:16.204590    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d817b15-2af4-4c24-a737-e2cfa443ed47-credential-socket\") pod \"e2e-api-a-cd7dca-openshift-default-654f7c496d-ngm26\" (UID: \"4d817b15-2af4-4c24-a737-e2cfa443ed47\") " pod="openshift-ingress/e2e-api-a-cd7dca-openshift-default-654f7c496d-ngm26"
Sep 10 19:32:16.204793 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:16.204778    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d817b15-2af4-4c24-a737-e2cfa443ed47-istio-data\") pod \"e2e-api-a-cd7dca-openshift-default-654f7c496d-ngm26\" (UID: \"4d817b15-2af4-4c24-a737-e2cfa443ed47\") " pod="openshift-ingress/e2e-api-a-cd7dca-openshift-default-654f7c496d-ngm26"
Sep 10 19:32:16.204928 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:16.204913    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d817b15-2af4-4c24-a737-e2cfa443ed47-istio-token\") pod \"e2e-api-a-cd7dca-openshift-default-654f7c496d-ngm26\" (UID: \"4d817b15-2af4-4c24-a737-e2cfa443ed47\") " pod="openshift-ingress/e2e-api-a-cd7dca-openshift-default-654f7c496d-ngm26"
Sep 10 19:32:16.205104 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:16.205091    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d817b15-2af4-4c24-a737-e2cfa443ed47-istiod-ca-cert\") pod \"e2e-api-a-cd7dca-openshift-default-654f7c496d-ngm26\" (UID: \"4d817b15-2af4-4c24-a737-e2cfa443ed47\") " pod="openshift-ingress/e2e-api-a-cd7dca-openshift-default-654f7c496d-ngm26"
Sep 10 19:32:16.205286 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:16.205249    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d817b15-2af4-4c24-a737-e2cfa443ed47-istio-podinfo\") pod \"e2e-api-a-cd7dca-openshift-default-654f7c496d-ngm26\" (UID: \"4d817b15-2af4-4c24-a737-e2cfa443ed47\") " pod="openshift-ingress/e2e-api-a-cd7dca-openshift-default-654f7c496d-ngm26"
Sep 10 19:32:16.205450 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:16.205436    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d817b15-2af4-4c24-a737-e2cfa443ed47-workload-socket\") pod \"e2e-api-a-cd7dca-openshift-default-654f7c496d-ngm26\" (UID: \"4d817b15-2af4-4c24-a737-e2cfa443ed47\") " pod="openshift-ingress/e2e-api-a-cd7dca-openshift-default-654f7c496d-ngm26"
Sep 10 19:32:16.205669 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:16.205655    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d817b15-2af4-4c24-a737-e2cfa443ed47-istio-envoy\") pod \"e2e-api-a-cd7dca-openshift-default-654f7c496d-ngm26\" (UID: \"4d817b15-2af4-4c24-a737-e2cfa443ed47\") " pod="openshift-ingress/e2e-api-a-cd7dca-openshift-default-654f7c496d-ngm26"
Sep 10 19:32:16.307016 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:16.306984    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d817b15-2af4-4c24-a737-e2cfa443ed47-istio-envoy\") pod \"e2e-api-a-cd7dca-openshift-default-654f7c496d-ngm26\" (UID: \"4d817b15-2af4-4c24-a737-e2cfa443ed47\") " pod="openshift-ingress/e2e-api-a-cd7dca-openshift-default-654f7c496d-ngm26"
Sep 10 19:32:16.307223 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:16.307026    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d817b15-2af4-4c24-a737-e2cfa443ed47-workload-certs\") pod \"e2e-api-a-cd7dca-openshift-default-654f7c496d-ngm26\" (UID: \"4d817b15-2af4-4c24-a737-e2cfa443ed47\") " pod="openshift-ingress/e2e-api-a-cd7dca-openshift-default-654f7c496d-ngm26"
Sep 10 19:32:16.307223 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:16.307044    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9tdp\" (UniqueName: \"kubernetes.io/projected/4d817b15-2af4-4c24-a737-e2cfa443ed47-kube-api-access-r9tdp\") pod \"e2e-api-a-cd7dca-openshift-default-654f7c496d-ngm26\" (UID: \"4d817b15-2af4-4c24-a737-e2cfa443ed47\") " pod="openshift-ingress/e2e-api-a-cd7dca-openshift-default-654f7c496d-ngm26"
Sep 10 19:32:16.307223 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:16.307070    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d817b15-2af4-4c24-a737-e2cfa443ed47-credential-socket\") pod \"e2e-api-a-cd7dca-openshift-default-654f7c496d-ngm26\" (UID: \"4d817b15-2af4-4c24-a737-e2cfa443ed47\") " pod="openshift-ingress/e2e-api-a-cd7dca-openshift-default-654f7c496d-ngm26"
Sep 10 19:32:16.307442 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:16.307210    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d817b15-2af4-4c24-a737-e2cfa443ed47-istio-data\") pod \"e2e-api-a-cd7dca-openshift-default-654f7c496d-ngm26\" (UID: \"4d817b15-2af4-4c24-a737-e2cfa443ed47\") " pod="openshift-ingress/e2e-api-a-cd7dca-openshift-default-654f7c496d-ngm26"
Sep 10 19:32:16.307442 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:16.307255    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d817b15-2af4-4c24-a737-e2cfa443ed47-istio-token\") pod \"e2e-api-a-cd7dca-openshift-default-654f7c496d-ngm26\" (UID: \"4d817b15-2af4-4c24-a737-e2cfa443ed47\") " pod="openshift-ingress/e2e-api-a-cd7dca-openshift-default-654f7c496d-ngm26"
Sep 10 19:32:16.307442 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:16.307322    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d817b15-2af4-4c24-a737-e2cfa443ed47-istiod-ca-cert\") pod \"e2e-api-a-cd7dca-openshift-default-654f7c496d-ngm26\" (UID: \"4d817b15-2af4-4c24-a737-e2cfa443ed47\") " pod="openshift-ingress/e2e-api-a-cd7dca-openshift-default-654f7c496d-ngm26"
Sep 10 19:32:16.307442 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:16.307391    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d817b15-2af4-4c24-a737-e2cfa443ed47-istio-podinfo\") pod \"e2e-api-a-cd7dca-openshift-default-654f7c496d-ngm26\" (UID: \"4d817b15-2af4-4c24-a737-e2cfa443ed47\") " pod="openshift-ingress/e2e-api-a-cd7dca-openshift-default-654f7c496d-ngm26"
Sep 10 19:32:16.307442 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:16.307422    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d817b15-2af4-4c24-a737-e2cfa443ed47-workload-socket\") pod \"e2e-api-a-cd7dca-openshift-default-654f7c496d-ngm26\" (UID: \"4d817b15-2af4-4c24-a737-e2cfa443ed47\") " pod="openshift-ingress/e2e-api-a-cd7dca-openshift-default-654f7c496d-ngm26"
Sep 10 19:32:16.307703 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:16.307520    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d817b15-2af4-4c24-a737-e2cfa443ed47-workload-certs\") pod \"e2e-api-a-cd7dca-openshift-default-654f7c496d-ngm26\" (UID: \"4d817b15-2af4-4c24-a737-e2cfa443ed47\") " pod="openshift-ingress/e2e-api-a-cd7dca-openshift-default-654f7c496d-ngm26"
Sep 10 19:32:16.307703 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:16.307555    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d817b15-2af4-4c24-a737-e2cfa443ed47-istio-data\") pod \"e2e-api-a-cd7dca-openshift-default-654f7c496d-ngm26\" (UID: \"4d817b15-2af4-4c24-a737-e2cfa443ed47\") " pod="openshift-ingress/e2e-api-a-cd7dca-openshift-default-654f7c496d-ngm26"
Sep 10 19:32:16.307804 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:16.307753    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d817b15-2af4-4c24-a737-e2cfa443ed47-credential-socket\") pod \"e2e-api-a-cd7dca-openshift-default-654f7c496d-ngm26\" (UID: \"4d817b15-2af4-4c24-a737-e2cfa443ed47\") " pod="openshift-ingress/e2e-api-a-cd7dca-openshift-default-654f7c496d-ngm26"
Sep 10 19:32:16.307866 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:16.307813    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d817b15-2af4-4c24-a737-e2cfa443ed47-workload-socket\") pod \"e2e-api-a-cd7dca-openshift-default-654f7c496d-ngm26\" (UID: \"4d817b15-2af4-4c24-a737-e2cfa443ed47\") " pod="openshift-ingress/e2e-api-a-cd7dca-openshift-default-654f7c496d-ngm26"
Sep 10 19:32:16.308133 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:16.308110    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d817b15-2af4-4c24-a737-e2cfa443ed47-istiod-ca-cert\") pod \"e2e-api-a-cd7dca-openshift-default-654f7c496d-ngm26\" (UID: \"4d817b15-2af4-4c24-a737-e2cfa443ed47\") " pod="openshift-ingress/e2e-api-a-cd7dca-openshift-default-654f7c496d-ngm26"
Sep 10 19:32:16.309419 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:16.309401    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d817b15-2af4-4c24-a737-e2cfa443ed47-istio-envoy\") pod \"e2e-api-a-cd7dca-openshift-default-654f7c496d-ngm26\" (UID: \"4d817b15-2af4-4c24-a737-e2cfa443ed47\") " pod="openshift-ingress/e2e-api-a-cd7dca-openshift-default-654f7c496d-ngm26"
Sep 10 19:32:16.309736 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:16.309716    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d817b15-2af4-4c24-a737-e2cfa443ed47-istio-podinfo\") pod \"e2e-api-a-cd7dca-openshift-default-654f7c496d-ngm26\" (UID: \"4d817b15-2af4-4c24-a737-e2cfa443ed47\") " pod="openshift-ingress/e2e-api-a-cd7dca-openshift-default-654f7c496d-ngm26"
Sep 10 19:32:16.316254 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:16.316230    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d817b15-2af4-4c24-a737-e2cfa443ed47-istio-token\") pod \"e2e-api-a-cd7dca-openshift-default-654f7c496d-ngm26\" (UID: \"4d817b15-2af4-4c24-a737-e2cfa443ed47\") " pod="openshift-ingress/e2e-api-a-cd7dca-openshift-default-654f7c496d-ngm26"
Sep 10 19:32:16.316438 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:16.316418    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9tdp\" (UniqueName: \"kubernetes.io/projected/4d817b15-2af4-4c24-a737-e2cfa443ed47-kube-api-access-r9tdp\") pod \"e2e-api-a-cd7dca-openshift-default-654f7c496d-ngm26\" (UID: \"4d817b15-2af4-4c24-a737-e2cfa443ed47\") " pod="openshift-ingress/e2e-api-a-cd7dca-openshift-default-654f7c496d-ngm26"
Sep 10 19:32:16.450242 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:16.450137    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-cd7dca-openshift-default-654f7c496d-ngm26"
Sep 10 19:32:16.752651 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:16.752627    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-cd7dca-openshift-default-654f7c496d-ngm26"]
Sep 10 19:32:16.755102 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:32:16.755073    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d817b15_2af4_4c24_a737_e2cfa443ed47.slice/crio-2a7dc869116a3e2252ccbf06ffa4b67582fc34d1fb9bceb672c374a5ef7586d3 WatchSource:0}: Error finding container 2a7dc869116a3e2252ccbf06ffa4b67582fc34d1fb9bceb672c374a5ef7586d3: Status 404 returned error can't find the container with id 2a7dc869116a3e2252ccbf06ffa4b67582fc34d1fb9bceb672c374a5ef7586d3
Sep 10 19:32:16.757709 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:16.757664    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 19:32:16.757816 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:16.757755    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 19:32:16.757864 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:16.757812    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 19:32:17.583383 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:17.583331    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-cd7dca-openshift-default-654f7c496d-ngm26" event={"ID":"4d817b15-2af4-4c24-a737-e2cfa443ed47","Type":"ContainerStarted","Data":"05a850f6dee6b955483d7cab0ec3d480ca95eff6e909185d1e91e2e89a45cbd2"}
Sep 10 19:32:17.583786 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:17.583439    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-cd7dca-openshift-default-654f7c496d-ngm26" event={"ID":"4d817b15-2af4-4c24-a737-e2cfa443ed47","Type":"ContainerStarted","Data":"2a7dc869116a3e2252ccbf06ffa4b67582fc34d1fb9bceb672c374a5ef7586d3"}
Sep 10 19:32:17.607891 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:17.607841    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-cd7dca-openshift-default-654f7c496d-ngm26" podStartSLOduration=1.607822916 podStartE2EDuration="1.607822916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:32:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:32:17.605242822 +0000 UTC m=+1300.834292872" watchObservedRunningTime="2026-09-10 19:32:17.607822916 +0000 UTC m=+1300.836872941"
Sep 10 19:32:18.450388 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:18.450325    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-cd7dca-openshift-default-654f7c496d-ngm26"
Sep 10 19:32:18.456390 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:18.456355    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-cd7dca-openshift-default-654f7c496d-ngm26"
Sep 10 19:32:18.588762 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:18.588733    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-cd7dca-openshift-default-654f7c496d-ngm26"
Sep 10 19:32:18.589670 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:18.589649    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-cd7dca-openshift-default-654f7c496d-ngm26"
Sep 10 19:32:27.321103 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:27.321068    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-88n8f"]
Sep 10 19:32:37.687177 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:37.687088    2546 generic.go:356] "Generic (PLEG): container finished" podID="4d817b15-2af4-4c24-a737-e2cfa443ed47" containerID="05a850f6dee6b955483d7cab0ec3d480ca95eff6e909185d1e91e2e89a45cbd2" exitCode=0
Sep 10 19:32:37.687177 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:37.687160    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-cd7dca-openshift-default-654f7c496d-ngm26" event={"ID":"4d817b15-2af4-4c24-a737-e2cfa443ed47","Type":"ContainerDied","Data":"05a850f6dee6b955483d7cab0ec3d480ca95eff6e909185d1e91e2e89a45cbd2"}
Sep 10 19:32:37.687703 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:37.687618    2546 scope.go:117] "RemoveContainer" containerID="05a850f6dee6b955483d7cab0ec3d480ca95eff6e909185d1e91e2e89a45cbd2"
Sep 10 19:32:37.688748 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:37.688721    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 19:32:37.688808 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:37.688787    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 19:32:37.688847 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:37.688818    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 19:32:38.694266 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:38.694230    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-cd7dca-openshift-default-654f7c496d-ngm26" event={"ID":"4d817b15-2af4-4c24-a737-e2cfa443ed47","Type":"ContainerStarted","Data":"ba415640f1b7512b14ce52c0b55b76cf8b3d2f773c1bbd9e1da5ec81ccf7e231"}
Sep 10 19:32:39.450813 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:39.450770    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-cd7dca-openshift-default-654f7c496d-ngm26"
Sep 10 19:32:39.451050 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:39.451026    2546 patch_prober.go:28] interesting pod/e2e-api-a-cd7dca-openshift-default-654f7c496d-ngm26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Sep 10 19:32:39.451099 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:39.451076    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cd7dca-openshift-default-654f7c496d-ngm26" podUID="4d817b15-2af4-4c24-a737-e2cfa443ed47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Sep 10 19:32:40.451022 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:40.450983    2546 patch_prober.go:28] interesting pod/e2e-api-a-cd7dca-openshift-default-654f7c496d-ngm26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Sep 10 19:32:40.451466 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:40.451038    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cd7dca-openshift-default-654f7c496d-ngm26" podUID="4d817b15-2af4-4c24-a737-e2cfa443ed47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Sep 10 19:32:41.451633 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:41.451596    2546 patch_prober.go:28] interesting pod/e2e-api-a-cd7dca-openshift-default-654f7c496d-ngm26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Sep 10 19:32:41.452060 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:41.451654    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cd7dca-openshift-default-654f7c496d-ngm26" podUID="4d817b15-2af4-4c24-a737-e2cfa443ed47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Sep 10 19:32:42.450607 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:42.450573    2546 patch_prober.go:28] interesting pod/e2e-api-a-cd7dca-openshift-default-654f7c496d-ngm26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Sep 10 19:32:42.450805 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:42.450645    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cd7dca-openshift-default-654f7c496d-ngm26" podUID="4d817b15-2af4-4c24-a737-e2cfa443ed47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Sep 10 19:32:43.451242 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:43.451198    2546 patch_prober.go:28] interesting pod/e2e-api-a-cd7dca-openshift-default-654f7c496d-ngm26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Sep 10 19:32:43.451702 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:43.451272    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cd7dca-openshift-default-654f7c496d-ngm26" podUID="4d817b15-2af4-4c24-a737-e2cfa443ed47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Sep 10 19:32:44.455398 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:44.455359    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-cd7dca-openshift-default-654f7c496d-ngm26"
Sep 10 19:32:44.455774 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:44.455639    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-cd7dca-openshift-default-654f7c496d-ngm26"
Sep 10 19:32:44.456352 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:44.456321    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-cd7dca-openshift-default-654f7c496d-ngm26"
Sep 10 19:32:44.725531 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:44.725431    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-f15c09/sub-test-model-f15c09-kserve-68c8bbd867-52rx6"]
Sep 10 19:32:44.729054 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:44.729034    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-f15c09/sub-test-model-f15c09-kserve-68c8bbd867-52rx6"
Sep 10 19:32:44.731554 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:44.731526    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-f15c09\"/\"openshift-service-ca.crt\""
Sep 10 19:32:44.731701 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:44.731575    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-f15c09\"/\"kube-root-ca.crt\""
Sep 10 19:32:44.731701 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:44.731587    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-f15c09\"/\"sub-test-model-f15c09-kserve-self-signed-certs\""
Sep 10 19:32:44.731701 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:44.731668    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-f15c09\"/\"default-dockercfg-rcjfv\""
Sep 10 19:32:44.738925 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:44.738899    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-f15c09/sub-test-model-f15c09-kserve-68c8bbd867-52rx6"]
Sep 10 19:32:44.795180 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:44.795136    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc71d573-ac0c-4021-afdc-e418fd592da9-home\") pod \"sub-test-model-f15c09-kserve-68c8bbd867-52rx6\" (UID: \"bc71d573-ac0c-4021-afdc-e418fd592da9\") " pod="ai-tenant-e2e-shared-b-w3-f15c09/sub-test-model-f15c09-kserve-68c8bbd867-52rx6"
Sep 10 19:32:44.795447 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:44.795286    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxhgk\" (UniqueName: \"kubernetes.io/projected/bc71d573-ac0c-4021-afdc-e418fd592da9-kube-api-access-dxhgk\") pod \"sub-test-model-f15c09-kserve-68c8bbd867-52rx6\" (UID: \"bc71d573-ac0c-4021-afdc-e418fd592da9\") " pod="ai-tenant-e2e-shared-b-w3-f15c09/sub-test-model-f15c09-kserve-68c8bbd867-52rx6"
Sep 10 19:32:44.795447 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:44.795403    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc71d573-ac0c-4021-afdc-e418fd592da9-dshm\") pod \"sub-test-model-f15c09-kserve-68c8bbd867-52rx6\" (UID: \"bc71d573-ac0c-4021-afdc-e418fd592da9\") " pod="ai-tenant-e2e-shared-b-w3-f15c09/sub-test-model-f15c09-kserve-68c8bbd867-52rx6"
Sep 10 19:32:44.795447 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:44.795432    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc71d573-ac0c-4021-afdc-e418fd592da9-model-cache\") pod \"sub-test-model-f15c09-kserve-68c8bbd867-52rx6\" (UID: \"bc71d573-ac0c-4021-afdc-e418fd592da9\") " pod="ai-tenant-e2e-shared-b-w3-f15c09/sub-test-model-f15c09-kserve-68c8bbd867-52rx6"
Sep 10 19:32:44.795638 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:44.795455    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc71d573-ac0c-4021-afdc-e418fd592da9-tls-certs\") pod \"sub-test-model-f15c09-kserve-68c8bbd867-52rx6\" (UID: \"bc71d573-ac0c-4021-afdc-e418fd592da9\") " pod="ai-tenant-e2e-shared-b-w3-f15c09/sub-test-model-f15c09-kserve-68c8bbd867-52rx6"
Sep 10 19:32:44.795638 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:44.795537    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc71d573-ac0c-4021-afdc-e418fd592da9-tmp-dir\") pod \"sub-test-model-f15c09-kserve-68c8bbd867-52rx6\" (UID: \"bc71d573-ac0c-4021-afdc-e418fd592da9\") " pod="ai-tenant-e2e-shared-b-w3-f15c09/sub-test-model-f15c09-kserve-68c8bbd867-52rx6"
Sep 10 19:32:44.896661 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:44.896622    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc71d573-ac0c-4021-afdc-e418fd592da9-dshm\") pod \"sub-test-model-f15c09-kserve-68c8bbd867-52rx6\" (UID: \"bc71d573-ac0c-4021-afdc-e418fd592da9\") " pod="ai-tenant-e2e-shared-b-w3-f15c09/sub-test-model-f15c09-kserve-68c8bbd867-52rx6"
Sep 10 19:32:44.896661 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:44.896664    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc71d573-ac0c-4021-afdc-e418fd592da9-model-cache\") pod \"sub-test-model-f15c09-kserve-68c8bbd867-52rx6\" (UID: \"bc71d573-ac0c-4021-afdc-e418fd592da9\") " pod="ai-tenant-e2e-shared-b-w3-f15c09/sub-test-model-f15c09-kserve-68c8bbd867-52rx6"
Sep 10 19:32:44.896928 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:44.896685    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc71d573-ac0c-4021-afdc-e418fd592da9-tls-certs\") pod \"sub-test-model-f15c09-kserve-68c8bbd867-52rx6\" (UID: \"bc71d573-ac0c-4021-afdc-e418fd592da9\") " pod="ai-tenant-e2e-shared-b-w3-f15c09/sub-test-model-f15c09-kserve-68c8bbd867-52rx6"
Sep 10 19:32:44.896928 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:44.896725    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc71d573-ac0c-4021-afdc-e418fd592da9-tmp-dir\") pod \"sub-test-model-f15c09-kserve-68c8bbd867-52rx6\" (UID: \"bc71d573-ac0c-4021-afdc-e418fd592da9\") " pod="ai-tenant-e2e-shared-b-w3-f15c09/sub-test-model-f15c09-kserve-68c8bbd867-52rx6"
Sep 10 19:32:44.896928 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:44.896767    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc71d573-ac0c-4021-afdc-e418fd592da9-home\") pod \"sub-test-model-f15c09-kserve-68c8bbd867-52rx6\" (UID: \"bc71d573-ac0c-4021-afdc-e418fd592da9\") " pod="ai-tenant-e2e-shared-b-w3-f15c09/sub-test-model-f15c09-kserve-68c8bbd867-52rx6"
Sep 10 19:32:44.896928 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:44.896828    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxhgk\" (UniqueName: \"kubernetes.io/projected/bc71d573-ac0c-4021-afdc-e418fd592da9-kube-api-access-dxhgk\") pod \"sub-test-model-f15c09-kserve-68c8bbd867-52rx6\" (UID: \"bc71d573-ac0c-4021-afdc-e418fd592da9\") " pod="ai-tenant-e2e-shared-b-w3-f15c09/sub-test-model-f15c09-kserve-68c8bbd867-52rx6"
Sep 10 19:32:44.897176 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:44.897155    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc71d573-ac0c-4021-afdc-e418fd592da9-model-cache\") pod \"sub-test-model-f15c09-kserve-68c8bbd867-52rx6\" (UID: \"bc71d573-ac0c-4021-afdc-e418fd592da9\") " pod="ai-tenant-e2e-shared-b-w3-f15c09/sub-test-model-f15c09-kserve-68c8bbd867-52rx6"
Sep 10 19:32:44.897234 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:44.897218    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc71d573-ac0c-4021-afdc-e418fd592da9-home\") pod \"sub-test-model-f15c09-kserve-68c8bbd867-52rx6\" (UID: \"bc71d573-ac0c-4021-afdc-e418fd592da9\") " pod="ai-tenant-e2e-shared-b-w3-f15c09/sub-test-model-f15c09-kserve-68c8bbd867-52rx6"
Sep 10 19:32:44.897518 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:44.897496    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc71d573-ac0c-4021-afdc-e418fd592da9-tmp-dir\") pod \"sub-test-model-f15c09-kserve-68c8bbd867-52rx6\" (UID: \"bc71d573-ac0c-4021-afdc-e418fd592da9\") " pod="ai-tenant-e2e-shared-b-w3-f15c09/sub-test-model-f15c09-kserve-68c8bbd867-52rx6"
Sep 10 19:32:44.899039 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:44.899011    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc71d573-ac0c-4021-afdc-e418fd592da9-dshm\") pod \"sub-test-model-f15c09-kserve-68c8bbd867-52rx6\" (UID: \"bc71d573-ac0c-4021-afdc-e418fd592da9\") " pod="ai-tenant-e2e-shared-b-w3-f15c09/sub-test-model-f15c09-kserve-68c8bbd867-52rx6"
Sep 10 19:32:44.899692 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:44.899667    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc71d573-ac0c-4021-afdc-e418fd592da9-tls-certs\") pod \"sub-test-model-f15c09-kserve-68c8bbd867-52rx6\" (UID: \"bc71d573-ac0c-4021-afdc-e418fd592da9\") " pod="ai-tenant-e2e-shared-b-w3-f15c09/sub-test-model-f15c09-kserve-68c8bbd867-52rx6"
Sep 10 19:32:44.906535 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:44.906497    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxhgk\" (UniqueName: \"kubernetes.io/projected/bc71d573-ac0c-4021-afdc-e418fd592da9-kube-api-access-dxhgk\") pod \"sub-test-model-f15c09-kserve-68c8bbd867-52rx6\" (UID: \"bc71d573-ac0c-4021-afdc-e418fd592da9\") " pod="ai-tenant-e2e-shared-b-w3-f15c09/sub-test-model-f15c09-kserve-68c8bbd867-52rx6"
Sep 10 19:32:45.039089 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:45.038985    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-f15c09/sub-test-model-f15c09-kserve-68c8bbd867-52rx6"
Sep 10 19:32:45.331154 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:45.331121    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-f15c09/sub-test-model-f15c09-kserve-68c8bbd867-52rx6"]
Sep 10 19:32:45.336106 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:32:45.336077    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbc71d573_ac0c_4021_afdc_e418fd592da9.slice/crio-3e6552c850122b9e907565b15dd59df8c239170e59b4753649ae9b11a814385f WatchSource:0}: Error finding container 3e6552c850122b9e907565b15dd59df8c239170e59b4753649ae9b11a814385f: Status 404 returned error can't find the container with id 3e6552c850122b9e907565b15dd59df8c239170e59b4753649ae9b11a814385f
Sep 10 19:32:45.735527 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:45.735488    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-f15c09/sub-test-model-f15c09-kserve-68c8bbd867-52rx6" event={"ID":"bc71d573-ac0c-4021-afdc-e418fd592da9","Type":"ContainerStarted","Data":"2c8a674d173cce21e58620f0cb5c39cf9f4da60aee0258737df45bb7755faa03"}
Sep 10 19:32:45.736038 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:45.735541    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-f15c09/sub-test-model-f15c09-kserve-68c8bbd867-52rx6" event={"ID":"bc71d573-ac0c-4021-afdc-e418fd592da9","Type":"ContainerStarted","Data":"3e6552c850122b9e907565b15dd59df8c239170e59b4753649ae9b11a814385f"}
Sep 10 19:32:45.753323 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:45.753267    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-f15c09/sub-test-model-f15c09-kserve-68c8bbd867-52rx6" podStartSLOduration=1.753250722 podStartE2EDuration="1.753250722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:32:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:32:45.752249382 +0000 UTC m=+1328.981299409" watchObservedRunningTime="2026-09-10 19:32:45.753250722 +0000 UTC m=+1328.982300748"
Sep 10 19:32:49.293613 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:49.293568    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-66e9b4-54ffc6b84c-275sc"]
Sep 10 19:32:49.297941 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:49.297916    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-66e9b4-54ffc6b84c-275sc"
Sep 10 19:32:49.300819 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:49.300785    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-66e9b4-serving-cert\""
Sep 10 19:32:49.314144 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:49.314116    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-66e9b4-54ffc6b84c-275sc"]
Sep 10 19:32:49.342756 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:49.342722    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a20a9d7-2d6d-4dc7-b6d4-de7e20cbba45-metrics-certs\") pod \"maas-api-e2e-api-b-66e9b4-54ffc6b84c-275sc\" (UID: \"0a20a9d7-2d6d-4dc7-b6d4-de7e20cbba45\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-66e9b4-54ffc6b84c-275sc"
Sep 10 19:32:49.343018 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:49.342993    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a20a9d7-2d6d-4dc7-b6d4-de7e20cbba45-maas-api-tls\") pod \"maas-api-e2e-api-b-66e9b4-54ffc6b84c-275sc\" (UID: \"0a20a9d7-2d6d-4dc7-b6d4-de7e20cbba45\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-66e9b4-54ffc6b84c-275sc"
Sep 10 19:32:49.343273 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:49.343247    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a20a9d7-2d6d-4dc7-b6d4-de7e20cbba45-openshift-service-ca\") pod \"maas-api-e2e-api-b-66e9b4-54ffc6b84c-275sc\" (UID: \"0a20a9d7-2d6d-4dc7-b6d4-de7e20cbba45\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-66e9b4-54ffc6b84c-275sc"
Sep 10 19:32:49.343397 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:49.343304    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfprr\" (UniqueName: \"kubernetes.io/projected/0a20a9d7-2d6d-4dc7-b6d4-de7e20cbba45-kube-api-access-wfprr\") pod \"maas-api-e2e-api-b-66e9b4-54ffc6b84c-275sc\" (UID: \"0a20a9d7-2d6d-4dc7-b6d4-de7e20cbba45\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-66e9b4-54ffc6b84c-275sc"
Sep 10 19:32:49.444722 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:49.444680    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a20a9d7-2d6d-4dc7-b6d4-de7e20cbba45-maas-api-tls\") pod \"maas-api-e2e-api-b-66e9b4-54ffc6b84c-275sc\" (UID: \"0a20a9d7-2d6d-4dc7-b6d4-de7e20cbba45\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-66e9b4-54ffc6b84c-275sc"
Sep 10 19:32:49.444945 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:49.444740    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a20a9d7-2d6d-4dc7-b6d4-de7e20cbba45-openshift-service-ca\") pod \"maas-api-e2e-api-b-66e9b4-54ffc6b84c-275sc\" (UID: \"0a20a9d7-2d6d-4dc7-b6d4-de7e20cbba45\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-66e9b4-54ffc6b84c-275sc"
Sep 10 19:32:49.444945 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:49.444768    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfprr\" (UniqueName: \"kubernetes.io/projected/0a20a9d7-2d6d-4dc7-b6d4-de7e20cbba45-kube-api-access-wfprr\") pod \"maas-api-e2e-api-b-66e9b4-54ffc6b84c-275sc\" (UID: \"0a20a9d7-2d6d-4dc7-b6d4-de7e20cbba45\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-66e9b4-54ffc6b84c-275sc"
Sep 10 19:32:49.444945 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:49.444788    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a20a9d7-2d6d-4dc7-b6d4-de7e20cbba45-metrics-certs\") pod \"maas-api-e2e-api-b-66e9b4-54ffc6b84c-275sc\" (UID: \"0a20a9d7-2d6d-4dc7-b6d4-de7e20cbba45\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-66e9b4-54ffc6b84c-275sc"
Sep 10 19:32:49.444945 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:32:49.444843    2546 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-66e9b4-serving-cert: secret "maas-api-e2e-api-b-66e9b4-serving-cert" not found
Sep 10 19:32:49.444945 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:32:49.444930    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a20a9d7-2d6d-4dc7-b6d4-de7e20cbba45-maas-api-tls podName:0a20a9d7-2d6d-4dc7-b6d4-de7e20cbba45 nodeName:}" failed. No retries permitted until 2026-09-10 19:32:49.94490843 +0000 UTC m=+1333.173958454 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0a20a9d7-2d6d-4dc7-b6d4-de7e20cbba45-maas-api-tls") pod "maas-api-e2e-api-b-66e9b4-54ffc6b84c-275sc" (UID: "0a20a9d7-2d6d-4dc7-b6d4-de7e20cbba45") : secret "maas-api-e2e-api-b-66e9b4-serving-cert" not found
Sep 10 19:32:49.445551 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:49.445529    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a20a9d7-2d6d-4dc7-b6d4-de7e20cbba45-openshift-service-ca\") pod \"maas-api-e2e-api-b-66e9b4-54ffc6b84c-275sc\" (UID: \"0a20a9d7-2d6d-4dc7-b6d4-de7e20cbba45\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-66e9b4-54ffc6b84c-275sc"
Sep 10 19:32:49.447300 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:49.447277    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a20a9d7-2d6d-4dc7-b6d4-de7e20cbba45-metrics-certs\") pod \"maas-api-e2e-api-b-66e9b4-54ffc6b84c-275sc\" (UID: \"0a20a9d7-2d6d-4dc7-b6d4-de7e20cbba45\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-66e9b4-54ffc6b84c-275sc"
Sep 10 19:32:49.459299 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:49.459270    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfprr\" (UniqueName: \"kubernetes.io/projected/0a20a9d7-2d6d-4dc7-b6d4-de7e20cbba45-kube-api-access-wfprr\") pod \"maas-api-e2e-api-b-66e9b4-54ffc6b84c-275sc\" (UID: \"0a20a9d7-2d6d-4dc7-b6d4-de7e20cbba45\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-66e9b4-54ffc6b84c-275sc"
Sep 10 19:32:49.948669 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:49.948625    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a20a9d7-2d6d-4dc7-b6d4-de7e20cbba45-maas-api-tls\") pod \"maas-api-e2e-api-b-66e9b4-54ffc6b84c-275sc\" (UID: \"0a20a9d7-2d6d-4dc7-b6d4-de7e20cbba45\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-66e9b4-54ffc6b84c-275sc"
Sep 10 19:32:49.951303 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:49.951266    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a20a9d7-2d6d-4dc7-b6d4-de7e20cbba45-maas-api-tls\") pod \"maas-api-e2e-api-b-66e9b4-54ffc6b84c-275sc\" (UID: \"0a20a9d7-2d6d-4dc7-b6d4-de7e20cbba45\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-66e9b4-54ffc6b84c-275sc"
Sep 10 19:32:50.138740 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:50.138701    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-88n8f"]
Sep 10 19:32:50.212594 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:50.212496    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-66e9b4-54ffc6b84c-275sc"
Sep 10 19:32:50.492092 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:50.492064    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-66e9b4-54ffc6b84c-275sc"]
Sep 10 19:32:50.494347 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:32:50.494305    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a20a9d7_2d6d_4dc7_b6d4_de7e20cbba45.slice/crio-b6e464b362e1bd488a6dffa959b57e7e6ea7d9107cd61d22a10549ca81a2d96d WatchSource:0}: Error finding container b6e464b362e1bd488a6dffa959b57e7e6ea7d9107cd61d22a10549ca81a2d96d: Status 404 returned error can't find the container with id b6e464b362e1bd488a6dffa959b57e7e6ea7d9107cd61d22a10549ca81a2d96d
Sep 10 19:32:50.764438 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:50.764335    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-66e9b4-54ffc6b84c-275sc" event={"ID":"0a20a9d7-2d6d-4dc7-b6d4-de7e20cbba45","Type":"ContainerStarted","Data":"b6e464b362e1bd488a6dffa959b57e7e6ea7d9107cd61d22a10549ca81a2d96d"}
Sep 10 19:32:51.771712 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:51.771674    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-66e9b4-54ffc6b84c-275sc" event={"ID":"0a20a9d7-2d6d-4dc7-b6d4-de7e20cbba45","Type":"ContainerStarted","Data":"bf6bde66375e685706fedcea51bb85ef2d1cd098e70288772f4c02c07ce9aa9e"}
Sep 10 19:32:51.772110 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:51.771797    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-66e9b4-54ffc6b84c-275sc"
Sep 10 19:32:51.791215 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:51.791156    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-66e9b4-54ffc6b84c-275sc" podStartSLOduration=2.48001083 podStartE2EDuration="2.79114056s" totalImagesPullingTime="311.12973ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:32:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:32:51.78919386 +0000 UTC m=+1335.018243886" watchObservedRunningTime="2026-09-10 19:32:51.79114056 +0000 UTC m=+1335.020190585"
Sep 10 19:32:55.039631 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:55.039596    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-f15c09/sub-test-model-f15c09-kserve-68c8bbd867-52rx6"
Sep 10 19:32:55.039631 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:55.039637    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-f15c09/sub-test-model-f15c09-kserve-68c8bbd867-52rx6"
Sep 10 19:32:55.053013 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:55.052973    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-f15c09/sub-test-model-f15c09-kserve-68c8bbd867-52rx6"
Sep 10 19:32:55.805743 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:55.805715    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-f15c09/sub-test-model-f15c09-kserve-68c8bbd867-52rx6"
Sep 10 19:32:57.784083 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:32:57.784054    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-66e9b4-54ffc6b84c-275sc"
Sep 10 19:33:05.915622 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:05.915531    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-88n8f"]
Sep 10 19:33:30.824488 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:30.824447    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-88n8f"]
Sep 10 19:33:36.050878 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:36.050844    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7528-496639"]
Sep 10 19:33:36.055513 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:36.055493    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7528-496639"
Sep 10 19:33:36.059204 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:36.059149    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7528-496639"]
Sep 10 19:33:36.059394 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:36.059225    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-sbhkd\""
Sep 10 19:33:36.215395 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:36.215324    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lf8m4\" (UniqueName: \"kubernetes.io/projected/b01a33f3-aeec-4e44-a714-c81dada6136f-kube-api-access-lf8m4\") pod \"test-curl-7528-496639\" (UID: \"b01a33f3-aeec-4e44-a714-c81dada6136f\") " pod="opendatahub/test-curl-7528-496639"
Sep 10 19:33:36.316581 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:36.316480    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lf8m4\" (UniqueName: \"kubernetes.io/projected/b01a33f3-aeec-4e44-a714-c81dada6136f-kube-api-access-lf8m4\") pod \"test-curl-7528-496639\" (UID: \"b01a33f3-aeec-4e44-a714-c81dada6136f\") " pod="opendatahub/test-curl-7528-496639"
Sep 10 19:33:36.325488 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:36.325454    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lf8m4\" (UniqueName: \"kubernetes.io/projected/b01a33f3-aeec-4e44-a714-c81dada6136f-kube-api-access-lf8m4\") pod \"test-curl-7528-496639\" (UID: \"b01a33f3-aeec-4e44-a714-c81dada6136f\") " pod="opendatahub/test-curl-7528-496639"
Sep 10 19:33:36.367953 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:36.367920    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7528-496639"
Sep 10 19:33:36.638744 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:36.638705    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7528-496639"]
Sep 10 19:33:36.641294 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:33:36.641265    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb01a33f3_aeec_4e44_a714_c81dada6136f.slice/crio-64925fac99ab189b19041c67364079d7f9312b8f6cf53882d61028ee6b39a38c WatchSource:0}: Error finding container 64925fac99ab189b19041c67364079d7f9312b8f6cf53882d61028ee6b39a38c: Status 404 returned error can't find the container with id 64925fac99ab189b19041c67364079d7f9312b8f6cf53882d61028ee6b39a38c
Sep 10 19:33:37.010037 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:37.009945    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7528-496639" event={"ID":"b01a33f3-aeec-4e44-a714-c81dada6136f","Type":"ContainerStarted","Data":"6b743696f88d411f48564f6608a3c5d613ce4cdbdef6d89438fb44a6616d1c30"}
Sep 10 19:33:37.010037 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:37.009995    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7528-496639" event={"ID":"b01a33f3-aeec-4e44-a714-c81dada6136f","Type":"ContainerStarted","Data":"64925fac99ab189b19041c67364079d7f9312b8f6cf53882d61028ee6b39a38c"}
Sep 10 19:33:37.027726 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:37.027675    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7528-496639" podStartSLOduration=1.027659533 podStartE2EDuration="1.027659533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:33:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:33:37.025834115 +0000 UTC m=+1380.254884135" watchObservedRunningTime="2026-09-10 19:33:37.027659533 +0000 UTC m=+1380.256709605"
Sep 10 19:33:37.630191 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:37.630152    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7528-496639"]
Sep 10 19:33:37.634703 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:37.634676    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7528-496639"]
Sep 10 19:33:38.015131 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:33:38.015098    2546 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7528-496639" container="test-curl-7528-496639"
Sep 10 19:33:38.678683 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:38.678643    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-cd7dca-openshift-default-654f7c496d-ngm26"]
Sep 10 19:33:38.679095 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:38.678850    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-cd7dca-openshift-default-654f7c496d-ngm26" podUID="4d817b15-2af4-4c24-a737-e2cfa443ed47" containerName="istio-proxy" containerID="cri-o://ba415640f1b7512b14ce52c0b55b76cf8b3d2f773c1bbd9e1da5ec81ccf7e231" gracePeriod=30
Sep 10 19:33:39.020446 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:39.020350    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7528-496639" podUID="b01a33f3-aeec-4e44-a714-c81dada6136f" containerName="test-curl-7528-496639" containerID="cri-o://6b743696f88d411f48564f6608a3c5d613ce4cdbdef6d89438fb44a6616d1c30" gracePeriod=2
Sep 10 19:33:39.024931 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:33:39.024901    2546 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7528-496639\" is forbidden: User \"system:node:ip-10-0-132-222.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-222.ec2.internal' and this object" podUID="b01a33f3-aeec-4e44-a714-c81dada6136f" pod="opendatahub/test-curl-7528-496639"
Sep 10 19:33:40.625235 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:40.625203    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-f15c09/sub-test-model-f15c09-kserve-68c8bbd867-52rx6"]
Sep 10 19:33:41.165074 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:41.165042    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7528-496639"
Sep 10 19:33:41.267587 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:41.267544    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lf8m4\" (UniqueName: \"kubernetes.io/projected/b01a33f3-aeec-4e44-a714-c81dada6136f-kube-api-access-lf8m4\") pod \"b01a33f3-aeec-4e44-a714-c81dada6136f\" (UID: \"b01a33f3-aeec-4e44-a714-c81dada6136f\") "
Sep 10 19:33:41.269736 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:41.269704    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b01a33f3-aeec-4e44-a714-c81dada6136f-kube-api-access-lf8m4" (OuterVolumeSpecName: "kube-api-access-lf8m4") pod "b01a33f3-aeec-4e44-a714-c81dada6136f" (UID: "b01a33f3-aeec-4e44-a714-c81dada6136f"). InnerVolumeSpecName "kube-api-access-lf8m4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 19:33:41.281177 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:41.281151    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b01a33f3-aeec-4e44-a714-c81dada6136f" path="/var/lib/kubelet/pods/b01a33f3-aeec-4e44-a714-c81dada6136f/volumes"
Sep 10 19:33:41.368686 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:41.368649    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lf8m4\" (UniqueName: \"kubernetes.io/projected/b01a33f3-aeec-4e44-a714-c81dada6136f-kube-api-access-lf8m4\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:33:42.036688 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:42.036649    2546 generic.go:356] "Generic (PLEG): container finished" podID="b01a33f3-aeec-4e44-a714-c81dada6136f" containerID="6b743696f88d411f48564f6608a3c5d613ce4cdbdef6d89438fb44a6616d1c30" exitCode=137
Sep 10 19:33:42.037108 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:42.036707    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7528-496639"
Sep 10 19:33:42.037108 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:42.036756    2546 scope.go:117] "RemoveContainer" containerID="6b743696f88d411f48564f6608a3c5d613ce4cdbdef6d89438fb44a6616d1c30"
Sep 10 19:33:42.045189 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:42.045168    2546 scope.go:117] "RemoveContainer" containerID="6b743696f88d411f48564f6608a3c5d613ce4cdbdef6d89438fb44a6616d1c30"
Sep 10 19:33:42.045510 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:33:42.045490    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b743696f88d411f48564f6608a3c5d613ce4cdbdef6d89438fb44a6616d1c30\": container with ID starting with 6b743696f88d411f48564f6608a3c5d613ce4cdbdef6d89438fb44a6616d1c30 not found: ID does not exist" containerID="6b743696f88d411f48564f6608a3c5d613ce4cdbdef6d89438fb44a6616d1c30"
Sep 10 19:33:42.045607 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:42.045516    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b743696f88d411f48564f6608a3c5d613ce4cdbdef6d89438fb44a6616d1c30"} err="failed to get container status \"6b743696f88d411f48564f6608a3c5d613ce4cdbdef6d89438fb44a6616d1c30\": rpc error: code = NotFound desc = could not find container \"6b743696f88d411f48564f6608a3c5d613ce4cdbdef6d89438fb44a6616d1c30\": container with ID starting with 6b743696f88d411f48564f6608a3c5d613ce4cdbdef6d89438fb44a6616d1c30 not found: ID does not exist"
Sep 10 19:33:43.944181 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:43.944119    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-cd7dca-openshift-default-654f7c496d-ngm26"
Sep 10 19:33:44.049757 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:44.049724    2546 generic.go:356] "Generic (PLEG): container finished" podID="4d817b15-2af4-4c24-a737-e2cfa443ed47" containerID="ba415640f1b7512b14ce52c0b55b76cf8b3d2f773c1bbd9e1da5ec81ccf7e231" exitCode=0
Sep 10 19:33:44.049938 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:44.049776    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-cd7dca-openshift-default-654f7c496d-ngm26"
Sep 10 19:33:44.049938 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:44.049801    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-cd7dca-openshift-default-654f7c496d-ngm26" event={"ID":"4d817b15-2af4-4c24-a737-e2cfa443ed47","Type":"ContainerDied","Data":"ba415640f1b7512b14ce52c0b55b76cf8b3d2f773c1bbd9e1da5ec81ccf7e231"}
Sep 10 19:33:44.049938 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:44.049845    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-cd7dca-openshift-default-654f7c496d-ngm26" event={"ID":"4d817b15-2af4-4c24-a737-e2cfa443ed47","Type":"ContainerDied","Data":"2a7dc869116a3e2252ccbf06ffa4b67582fc34d1fb9bceb672c374a5ef7586d3"}
Sep 10 19:33:44.049938 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:44.049866    2546 scope.go:117] "RemoveContainer" containerID="ba415640f1b7512b14ce52c0b55b76cf8b3d2f773c1bbd9e1da5ec81ccf7e231"
Sep 10 19:33:44.058634 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:44.058612    2546 scope.go:117] "RemoveContainer" containerID="05a850f6dee6b955483d7cab0ec3d480ca95eff6e909185d1e91e2e89a45cbd2"
Sep 10 19:33:44.067127 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:44.067108    2546 scope.go:117] "RemoveContainer" containerID="ba415640f1b7512b14ce52c0b55b76cf8b3d2f773c1bbd9e1da5ec81ccf7e231"
Sep 10 19:33:44.067436 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:33:44.067415    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba415640f1b7512b14ce52c0b55b76cf8b3d2f773c1bbd9e1da5ec81ccf7e231\": container with ID starting with ba415640f1b7512b14ce52c0b55b76cf8b3d2f773c1bbd9e1da5ec81ccf7e231 not found: ID does not exist" containerID="ba415640f1b7512b14ce52c0b55b76cf8b3d2f773c1bbd9e1da5ec81ccf7e231"
Sep 10 19:33:44.067530 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:44.067443    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba415640f1b7512b14ce52c0b55b76cf8b3d2f773c1bbd9e1da5ec81ccf7e231"} err="failed to get container status \"ba415640f1b7512b14ce52c0b55b76cf8b3d2f773c1bbd9e1da5ec81ccf7e231\": rpc error: code = NotFound desc = could not find container \"ba415640f1b7512b14ce52c0b55b76cf8b3d2f773c1bbd9e1da5ec81ccf7e231\": container with ID starting with ba415640f1b7512b14ce52c0b55b76cf8b3d2f773c1bbd9e1da5ec81ccf7e231 not found: ID does not exist"
Sep 10 19:33:44.067530 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:44.067460    2546 scope.go:117] "RemoveContainer" containerID="05a850f6dee6b955483d7cab0ec3d480ca95eff6e909185d1e91e2e89a45cbd2"
Sep 10 19:33:44.067687 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:33:44.067670    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05a850f6dee6b955483d7cab0ec3d480ca95eff6e909185d1e91e2e89a45cbd2\": container with ID starting with 05a850f6dee6b955483d7cab0ec3d480ca95eff6e909185d1e91e2e89a45cbd2 not found: ID does not exist" containerID="05a850f6dee6b955483d7cab0ec3d480ca95eff6e909185d1e91e2e89a45cbd2"
Sep 10 19:33:44.067745 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:44.067690    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05a850f6dee6b955483d7cab0ec3d480ca95eff6e909185d1e91e2e89a45cbd2"} err="failed to get container status \"05a850f6dee6b955483d7cab0ec3d480ca95eff6e909185d1e91e2e89a45cbd2\": rpc error: code = NotFound desc = could not find container \"05a850f6dee6b955483d7cab0ec3d480ca95eff6e909185d1e91e2e89a45cbd2\": container with ID starting with 05a850f6dee6b955483d7cab0ec3d480ca95eff6e909185d1e91e2e89a45cbd2 not found: ID does not exist"
Sep 10 19:33:44.093788 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:44.093760    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d817b15-2af4-4c24-a737-e2cfa443ed47-istiod-ca-cert\") pod \"4d817b15-2af4-4c24-a737-e2cfa443ed47\" (UID: \"4d817b15-2af4-4c24-a737-e2cfa443ed47\") "
Sep 10 19:33:44.093917 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:44.093822    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d817b15-2af4-4c24-a737-e2cfa443ed47-workload-socket\") pod \"4d817b15-2af4-4c24-a737-e2cfa443ed47\" (UID: \"4d817b15-2af4-4c24-a737-e2cfa443ed47\") "
Sep 10 19:33:44.094010 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:44.093991    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d817b15-2af4-4c24-a737-e2cfa443ed47-credential-socket\") pod \"4d817b15-2af4-4c24-a737-e2cfa443ed47\" (UID: \"4d817b15-2af4-4c24-a737-e2cfa443ed47\") "
Sep 10 19:33:44.094080 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:44.094059    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d817b15-2af4-4c24-a737-e2cfa443ed47-workload-certs\") pod \"4d817b15-2af4-4c24-a737-e2cfa443ed47\" (UID: \"4d817b15-2af4-4c24-a737-e2cfa443ed47\") "
Sep 10 19:33:44.094132 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:44.094074    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d817b15-2af4-4c24-a737-e2cfa443ed47-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4d817b15-2af4-4c24-a737-e2cfa443ed47" (UID: "4d817b15-2af4-4c24-a737-e2cfa443ed47"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 19:33:44.094132 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:44.094097    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d817b15-2af4-4c24-a737-e2cfa443ed47-istio-token\") pod \"4d817b15-2af4-4c24-a737-e2cfa443ed47\" (UID: \"4d817b15-2af4-4c24-a737-e2cfa443ed47\") "
Sep 10 19:33:44.094132 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:44.094118    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d817b15-2af4-4c24-a737-e2cfa443ed47-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4d817b15-2af4-4c24-a737-e2cfa443ed47" (UID: "4d817b15-2af4-4c24-a737-e2cfa443ed47"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:33:44.094273 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:44.094127    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d817b15-2af4-4c24-a737-e2cfa443ed47-istio-envoy\") pod \"4d817b15-2af4-4c24-a737-e2cfa443ed47\" (UID: \"4d817b15-2af4-4c24-a737-e2cfa443ed47\") "
Sep 10 19:33:44.094273 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:44.094180    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d817b15-2af4-4c24-a737-e2cfa443ed47-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4d817b15-2af4-4c24-a737-e2cfa443ed47" (UID: "4d817b15-2af4-4c24-a737-e2cfa443ed47"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:33:44.094273 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:44.094194    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d817b15-2af4-4c24-a737-e2cfa443ed47-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4d817b15-2af4-4c24-a737-e2cfa443ed47" (UID: "4d817b15-2af4-4c24-a737-e2cfa443ed47"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:33:44.094273 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:44.094223    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d817b15-2af4-4c24-a737-e2cfa443ed47-istio-data\") pod \"4d817b15-2af4-4c24-a737-e2cfa443ed47\" (UID: \"4d817b15-2af4-4c24-a737-e2cfa443ed47\") "
Sep 10 19:33:44.094273 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:44.094257    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r9tdp\" (UniqueName: \"kubernetes.io/projected/4d817b15-2af4-4c24-a737-e2cfa443ed47-kube-api-access-r9tdp\") pod \"4d817b15-2af4-4c24-a737-e2cfa443ed47\" (UID: \"4d817b15-2af4-4c24-a737-e2cfa443ed47\") "
Sep 10 19:33:44.094568 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:44.094314    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d817b15-2af4-4c24-a737-e2cfa443ed47-istio-podinfo\") pod \"4d817b15-2af4-4c24-a737-e2cfa443ed47\" (UID: \"4d817b15-2af4-4c24-a737-e2cfa443ed47\") "
Sep 10 19:33:44.094568 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:44.094434    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d817b15-2af4-4c24-a737-e2cfa443ed47-istio-data" (OuterVolumeSpecName: "istio-data") pod "4d817b15-2af4-4c24-a737-e2cfa443ed47" (UID: "4d817b15-2af4-4c24-a737-e2cfa443ed47"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:33:44.094686 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:44.094664    2546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d817b15-2af4-4c24-a737-e2cfa443ed47-workload-socket\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:33:44.094725 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:44.094693    2546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d817b15-2af4-4c24-a737-e2cfa443ed47-credential-socket\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:33:44.094725 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:44.094707    2546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d817b15-2af4-4c24-a737-e2cfa443ed47-workload-certs\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:33:44.094725 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:44.094720    2546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d817b15-2af4-4c24-a737-e2cfa443ed47-istio-data\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:33:44.094825 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:44.094731    2546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d817b15-2af4-4c24-a737-e2cfa443ed47-istiod-ca-cert\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:33:44.096283 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:44.096258    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d817b15-2af4-4c24-a737-e2cfa443ed47-kube-api-access-r9tdp" (OuterVolumeSpecName: "kube-api-access-r9tdp") pod "4d817b15-2af4-4c24-a737-e2cfa443ed47" (UID: "4d817b15-2af4-4c24-a737-e2cfa443ed47"). InnerVolumeSpecName "kube-api-access-r9tdp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 19:33:44.096438 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:44.096297    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d817b15-2af4-4c24-a737-e2cfa443ed47-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4d817b15-2af4-4c24-a737-e2cfa443ed47" (UID: "4d817b15-2af4-4c24-a737-e2cfa443ed47"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:33:44.096701 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:44.096679    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4d817b15-2af4-4c24-a737-e2cfa443ed47-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4d817b15-2af4-4c24-a737-e2cfa443ed47" (UID: "4d817b15-2af4-4c24-a737-e2cfa443ed47"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 19:33:44.096761 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:44.096733    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d817b15-2af4-4c24-a737-e2cfa443ed47-istio-token" (OuterVolumeSpecName: "istio-token") pod "4d817b15-2af4-4c24-a737-e2cfa443ed47" (UID: "4d817b15-2af4-4c24-a737-e2cfa443ed47"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 19:33:44.196285 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:44.196193    2546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d817b15-2af4-4c24-a737-e2cfa443ed47-istio-token\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:33:44.196285 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:44.196226    2546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d817b15-2af4-4c24-a737-e2cfa443ed47-istio-envoy\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:33:44.196285 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:44.196236    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9tdp\" (UniqueName: \"kubernetes.io/projected/4d817b15-2af4-4c24-a737-e2cfa443ed47-kube-api-access-r9tdp\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:33:44.196285 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:44.196247    2546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d817b15-2af4-4c24-a737-e2cfa443ed47-istio-podinfo\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:33:44.221708 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:44.221674    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-88n8f"]
Sep 10 19:33:44.372912 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:44.372876    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-cd7dca-openshift-default-654f7c496d-ngm26"]
Sep 10 19:33:44.377735 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:44.377710    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-cd7dca-openshift-default-654f7c496d-ngm26"]
Sep 10 19:33:45.282108 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:45.282066    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d817b15-2af4-4c24-a737-e2cfa443ed47" path="/var/lib/kubelet/pods/4d817b15-2af4-4c24-a737-e2cfa443ed47/volumes"
Sep 10 19:33:53.306657 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:53.306620    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-b64b1c/rate-test-model-b64b1c-kserve-69f895d475-5667b"]
Sep 10 19:33:53.307044 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:53.306985    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d817b15-2af4-4c24-a737-e2cfa443ed47" containerName="istio-proxy"
Sep 10 19:33:53.307044 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:53.306996    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d817b15-2af4-4c24-a737-e2cfa443ed47" containerName="istio-proxy"
Sep 10 19:33:53.307044 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:53.307005    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b01a33f3-aeec-4e44-a714-c81dada6136f" containerName="test-curl-7528-496639"
Sep 10 19:33:53.307044 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:53.307011    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="b01a33f3-aeec-4e44-a714-c81dada6136f" containerName="test-curl-7528-496639"
Sep 10 19:33:53.307205 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:53.307096    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="b01a33f3-aeec-4e44-a714-c81dada6136f" containerName="test-curl-7528-496639"
Sep 10 19:33:53.307205 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:53.307106    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d817b15-2af4-4c24-a737-e2cfa443ed47" containerName="istio-proxy"
Sep 10 19:33:53.307205 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:53.307111    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d817b15-2af4-4c24-a737-e2cfa443ed47" containerName="istio-proxy"
Sep 10 19:33:53.311519 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:53.311494    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-b64b1c/rate-test-model-b64b1c-kserve-69f895d475-5667b"
Sep 10 19:33:53.315553 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:53.315525    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-b64b1c\"/\"kube-root-ca.crt\""
Sep 10 19:33:53.315553 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:53.315538    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-b64b1c\"/\"rate-test-model-b64b1c-kserve-self-signed-certs\""
Sep 10 19:33:53.315727 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:53.315525    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-b64b1c\"/\"openshift-service-ca.crt\""
Sep 10 19:33:53.316110 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:53.316083    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-b64b1c\"/\"default-dockercfg-5k7v8\""
Sep 10 19:33:53.322118 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:53.322098    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-b64b1c/rate-test-model-b64b1c-kserve-69f895d475-5667b"]
Sep 10 19:33:53.376727 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:53.376693    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4b738a7-b6fc-4e0c-86de-d287cb2f2a61-tmp-dir\") pod \"rate-test-model-b64b1c-kserve-69f895d475-5667b\" (UID: \"e4b738a7-b6fc-4e0c-86de-d287cb2f2a61\") " pod="ai-tenant-e2e-shared-a-w3-b64b1c/rate-test-model-b64b1c-kserve-69f895d475-5667b"
Sep 10 19:33:53.376876 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:53.376751    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4b738a7-b6fc-4e0c-86de-d287cb2f2a61-model-cache\") pod \"rate-test-model-b64b1c-kserve-69f895d475-5667b\" (UID: \"e4b738a7-b6fc-4e0c-86de-d287cb2f2a61\") " pod="ai-tenant-e2e-shared-a-w3-b64b1c/rate-test-model-b64b1c-kserve-69f895d475-5667b"
Sep 10 19:33:53.376876 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:53.376822    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4b738a7-b6fc-4e0c-86de-d287cb2f2a61-tls-certs\") pod \"rate-test-model-b64b1c-kserve-69f895d475-5667b\" (UID: \"e4b738a7-b6fc-4e0c-86de-d287cb2f2a61\") " pod="ai-tenant-e2e-shared-a-w3-b64b1c/rate-test-model-b64b1c-kserve-69f895d475-5667b"
Sep 10 19:33:53.376876 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:53.376851    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pk42h\" (UniqueName: \"kubernetes.io/projected/e4b738a7-b6fc-4e0c-86de-d287cb2f2a61-kube-api-access-pk42h\") pod \"rate-test-model-b64b1c-kserve-69f895d475-5667b\" (UID: \"e4b738a7-b6fc-4e0c-86de-d287cb2f2a61\") " pod="ai-tenant-e2e-shared-a-w3-b64b1c/rate-test-model-b64b1c-kserve-69f895d475-5667b"
Sep 10 19:33:53.376876 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:53.376869    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4b738a7-b6fc-4e0c-86de-d287cb2f2a61-dshm\") pod \"rate-test-model-b64b1c-kserve-69f895d475-5667b\" (UID: \"e4b738a7-b6fc-4e0c-86de-d287cb2f2a61\") " pod="ai-tenant-e2e-shared-a-w3-b64b1c/rate-test-model-b64b1c-kserve-69f895d475-5667b"
Sep 10 19:33:53.377035 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:53.376927    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4b738a7-b6fc-4e0c-86de-d287cb2f2a61-home\") pod \"rate-test-model-b64b1c-kserve-69f895d475-5667b\" (UID: \"e4b738a7-b6fc-4e0c-86de-d287cb2f2a61\") " pod="ai-tenant-e2e-shared-a-w3-b64b1c/rate-test-model-b64b1c-kserve-69f895d475-5667b"
Sep 10 19:33:53.478492 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:53.478453    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4b738a7-b6fc-4e0c-86de-d287cb2f2a61-home\") pod \"rate-test-model-b64b1c-kserve-69f895d475-5667b\" (UID: \"e4b738a7-b6fc-4e0c-86de-d287cb2f2a61\") " pod="ai-tenant-e2e-shared-a-w3-b64b1c/rate-test-model-b64b1c-kserve-69f895d475-5667b"
Sep 10 19:33:53.478687 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:53.478542    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4b738a7-b6fc-4e0c-86de-d287cb2f2a61-tmp-dir\") pod \"rate-test-model-b64b1c-kserve-69f895d475-5667b\" (UID: \"e4b738a7-b6fc-4e0c-86de-d287cb2f2a61\") " pod="ai-tenant-e2e-shared-a-w3-b64b1c/rate-test-model-b64b1c-kserve-69f895d475-5667b"
Sep 10 19:33:53.478687 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:53.478624    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4b738a7-b6fc-4e0c-86de-d287cb2f2a61-model-cache\") pod \"rate-test-model-b64b1c-kserve-69f895d475-5667b\" (UID: \"e4b738a7-b6fc-4e0c-86de-d287cb2f2a61\") " pod="ai-tenant-e2e-shared-a-w3-b64b1c/rate-test-model-b64b1c-kserve-69f895d475-5667b"
Sep 10 19:33:53.478687 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:53.478671    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4b738a7-b6fc-4e0c-86de-d287cb2f2a61-tls-certs\") pod \"rate-test-model-b64b1c-kserve-69f895d475-5667b\" (UID: \"e4b738a7-b6fc-4e0c-86de-d287cb2f2a61\") " pod="ai-tenant-e2e-shared-a-w3-b64b1c/rate-test-model-b64b1c-kserve-69f895d475-5667b"
Sep 10 19:33:53.478859 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:53.478701    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pk42h\" (UniqueName: \"kubernetes.io/projected/e4b738a7-b6fc-4e0c-86de-d287cb2f2a61-kube-api-access-pk42h\") pod \"rate-test-model-b64b1c-kserve-69f895d475-5667b\" (UID: \"e4b738a7-b6fc-4e0c-86de-d287cb2f2a61\") " pod="ai-tenant-e2e-shared-a-w3-b64b1c/rate-test-model-b64b1c-kserve-69f895d475-5667b"
Sep 10 19:33:53.478859 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:53.478729    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4b738a7-b6fc-4e0c-86de-d287cb2f2a61-dshm\") pod \"rate-test-model-b64b1c-kserve-69f895d475-5667b\" (UID: \"e4b738a7-b6fc-4e0c-86de-d287cb2f2a61\") " pod="ai-tenant-e2e-shared-a-w3-b64b1c/rate-test-model-b64b1c-kserve-69f895d475-5667b"
Sep 10 19:33:53.478976 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:53.478899    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4b738a7-b6fc-4e0c-86de-d287cb2f2a61-home\") pod \"rate-test-model-b64b1c-kserve-69f895d475-5667b\" (UID: \"e4b738a7-b6fc-4e0c-86de-d287cb2f2a61\") " pod="ai-tenant-e2e-shared-a-w3-b64b1c/rate-test-model-b64b1c-kserve-69f895d475-5667b"
Sep 10 19:33:53.478976 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:53.478953    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4b738a7-b6fc-4e0c-86de-d287cb2f2a61-tmp-dir\") pod \"rate-test-model-b64b1c-kserve-69f895d475-5667b\" (UID: \"e4b738a7-b6fc-4e0c-86de-d287cb2f2a61\") " pod="ai-tenant-e2e-shared-a-w3-b64b1c/rate-test-model-b64b1c-kserve-69f895d475-5667b"
Sep 10 19:33:53.479381 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:53.479331    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4b738a7-b6fc-4e0c-86de-d287cb2f2a61-model-cache\") pod \"rate-test-model-b64b1c-kserve-69f895d475-5667b\" (UID: \"e4b738a7-b6fc-4e0c-86de-d287cb2f2a61\") " pod="ai-tenant-e2e-shared-a-w3-b64b1c/rate-test-model-b64b1c-kserve-69f895d475-5667b"
Sep 10 19:33:53.481740 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:53.481716    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4b738a7-b6fc-4e0c-86de-d287cb2f2a61-dshm\") pod \"rate-test-model-b64b1c-kserve-69f895d475-5667b\" (UID: \"e4b738a7-b6fc-4e0c-86de-d287cb2f2a61\") " pod="ai-tenant-e2e-shared-a-w3-b64b1c/rate-test-model-b64b1c-kserve-69f895d475-5667b"
Sep 10 19:33:53.482590 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:53.482566    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4b738a7-b6fc-4e0c-86de-d287cb2f2a61-tls-certs\") pod \"rate-test-model-b64b1c-kserve-69f895d475-5667b\" (UID: \"e4b738a7-b6fc-4e0c-86de-d287cb2f2a61\") " pod="ai-tenant-e2e-shared-a-w3-b64b1c/rate-test-model-b64b1c-kserve-69f895d475-5667b"
Sep 10 19:33:53.488597 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:53.488574    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pk42h\" (UniqueName: \"kubernetes.io/projected/e4b738a7-b6fc-4e0c-86de-d287cb2f2a61-kube-api-access-pk42h\") pod \"rate-test-model-b64b1c-kserve-69f895d475-5667b\" (UID: \"e4b738a7-b6fc-4e0c-86de-d287cb2f2a61\") " pod="ai-tenant-e2e-shared-a-w3-b64b1c/rate-test-model-b64b1c-kserve-69f895d475-5667b"
Sep 10 19:33:53.622315 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:53.622278    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-b64b1c/rate-test-model-b64b1c-kserve-69f895d475-5667b"
Sep 10 19:33:53.900944 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:53.900912    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-b64b1c/rate-test-model-b64b1c-kserve-69f895d475-5667b"]
Sep 10 19:33:53.903236 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:33:53.903196    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode4b738a7_b6fc_4e0c_86de_d287cb2f2a61.slice/crio-c087e78dc0a5e692e2f0aa57e96c33d19aa8ece32fc1e17dad05bf1e3e6df78d WatchSource:0}: Error finding container c087e78dc0a5e692e2f0aa57e96c33d19aa8ece32fc1e17dad05bf1e3e6df78d: Status 404 returned error can't find the container with id c087e78dc0a5e692e2f0aa57e96c33d19aa8ece32fc1e17dad05bf1e3e6df78d
Sep 10 19:33:54.106043 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:54.106008    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-b64b1c/rate-test-model-b64b1c-kserve-69f895d475-5667b" event={"ID":"e4b738a7-b6fc-4e0c-86de-d287cb2f2a61","Type":"ContainerStarted","Data":"aec855cbe4b29e1f13ecbb9f403825de2404a4150a151ae3c545dfb9db313a69"}
Sep 10 19:33:54.106246 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:54.106054    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-b64b1c/rate-test-model-b64b1c-kserve-69f895d475-5667b" event={"ID":"e4b738a7-b6fc-4e0c-86de-d287cb2f2a61","Type":"ContainerStarted","Data":"c087e78dc0a5e692e2f0aa57e96c33d19aa8ece32fc1e17dad05bf1e3e6df78d"}
Sep 10 19:33:54.130328 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:54.130278    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-b64b1c/rate-test-model-b64b1c-kserve-69f895d475-5667b" podStartSLOduration=1.130262821 podStartE2EDuration="1.130262821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:33:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:33:54.127559698 +0000 UTC m=+1397.356609724" watchObservedRunningTime="2026-09-10 19:33:54.130262821 +0000 UTC m=+1397.359312846"
Sep 10 19:33:54.836177 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:54.836143    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-66e9b4-54ffc6b84c-275sc"]
Sep 10 19:33:54.836607 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:54.836370    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-66e9b4-54ffc6b84c-275sc" podUID="0a20a9d7-2d6d-4dc7-b6d4-de7e20cbba45" containerName="maas-api" containerID="cri-o://bf6bde66375e685706fedcea51bb85ef2d1cd098e70288772f4c02c07ce9aa9e" gracePeriod=30
Sep 10 19:33:55.092739 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:55.092673    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-66e9b4-54ffc6b84c-275sc"
Sep 10 19:33:55.112008 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:55.111977    2546 generic.go:356] "Generic (PLEG): container finished" podID="0a20a9d7-2d6d-4dc7-b6d4-de7e20cbba45" containerID="bf6bde66375e685706fedcea51bb85ef2d1cd098e70288772f4c02c07ce9aa9e" exitCode=0
Sep 10 19:33:55.112162 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:55.112009    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-66e9b4-54ffc6b84c-275sc" event={"ID":"0a20a9d7-2d6d-4dc7-b6d4-de7e20cbba45","Type":"ContainerDied","Data":"bf6bde66375e685706fedcea51bb85ef2d1cd098e70288772f4c02c07ce9aa9e"}
Sep 10 19:33:55.112162 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:55.112041    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-66e9b4-54ffc6b84c-275sc"
Sep 10 19:33:55.112162 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:55.112078    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-66e9b4-54ffc6b84c-275sc" event={"ID":"0a20a9d7-2d6d-4dc7-b6d4-de7e20cbba45","Type":"ContainerDied","Data":"b6e464b362e1bd488a6dffa959b57e7e6ea7d9107cd61d22a10549ca81a2d96d"}
Sep 10 19:33:55.112162 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:55.112109    2546 scope.go:117] "RemoveContainer" containerID="bf6bde66375e685706fedcea51bb85ef2d1cd098e70288772f4c02c07ce9aa9e"
Sep 10 19:33:55.122726 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:55.122702    2546 scope.go:117] "RemoveContainer" containerID="bf6bde66375e685706fedcea51bb85ef2d1cd098e70288772f4c02c07ce9aa9e"
Sep 10 19:33:55.123027 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:33:55.123003    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf6bde66375e685706fedcea51bb85ef2d1cd098e70288772f4c02c07ce9aa9e\": container with ID starting with bf6bde66375e685706fedcea51bb85ef2d1cd098e70288772f4c02c07ce9aa9e not found: ID does not exist" containerID="bf6bde66375e685706fedcea51bb85ef2d1cd098e70288772f4c02c07ce9aa9e"
Sep 10 19:33:55.123111 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:55.123037    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf6bde66375e685706fedcea51bb85ef2d1cd098e70288772f4c02c07ce9aa9e"} err="failed to get container status \"bf6bde66375e685706fedcea51bb85ef2d1cd098e70288772f4c02c07ce9aa9e\": rpc error: code = NotFound desc = could not find container \"bf6bde66375e685706fedcea51bb85ef2d1cd098e70288772f4c02c07ce9aa9e\": container with ID starting with bf6bde66375e685706fedcea51bb85ef2d1cd098e70288772f4c02c07ce9aa9e not found: ID does not exist"
Sep 10 19:33:55.194004 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:55.193975    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a20a9d7-2d6d-4dc7-b6d4-de7e20cbba45-metrics-certs\") pod \"0a20a9d7-2d6d-4dc7-b6d4-de7e20cbba45\" (UID: \"0a20a9d7-2d6d-4dc7-b6d4-de7e20cbba45\") "
Sep 10 19:33:55.194205 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:55.194029    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a20a9d7-2d6d-4dc7-b6d4-de7e20cbba45-maas-api-tls\") pod \"0a20a9d7-2d6d-4dc7-b6d4-de7e20cbba45\" (UID: \"0a20a9d7-2d6d-4dc7-b6d4-de7e20cbba45\") "
Sep 10 19:33:55.194205 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:55.194073    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a20a9d7-2d6d-4dc7-b6d4-de7e20cbba45-openshift-service-ca\") pod \"0a20a9d7-2d6d-4dc7-b6d4-de7e20cbba45\" (UID: \"0a20a9d7-2d6d-4dc7-b6d4-de7e20cbba45\") "
Sep 10 19:33:55.194205 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:55.194107    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wfprr\" (UniqueName: \"kubernetes.io/projected/0a20a9d7-2d6d-4dc7-b6d4-de7e20cbba45-kube-api-access-wfprr\") pod \"0a20a9d7-2d6d-4dc7-b6d4-de7e20cbba45\" (UID: \"0a20a9d7-2d6d-4dc7-b6d4-de7e20cbba45\") "
Sep 10 19:33:55.194491 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:55.194453    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a20a9d7-2d6d-4dc7-b6d4-de7e20cbba45-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0a20a9d7-2d6d-4dc7-b6d4-de7e20cbba45" (UID: "0a20a9d7-2d6d-4dc7-b6d4-de7e20cbba45"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 19:33:55.196451 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:55.196406    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a20a9d7-2d6d-4dc7-b6d4-de7e20cbba45-kube-api-access-wfprr" (OuterVolumeSpecName: "kube-api-access-wfprr") pod "0a20a9d7-2d6d-4dc7-b6d4-de7e20cbba45" (UID: "0a20a9d7-2d6d-4dc7-b6d4-de7e20cbba45"). InnerVolumeSpecName "kube-api-access-wfprr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 19:33:55.196588 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:55.196468    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a20a9d7-2d6d-4dc7-b6d4-de7e20cbba45-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0a20a9d7-2d6d-4dc7-b6d4-de7e20cbba45" (UID: "0a20a9d7-2d6d-4dc7-b6d4-de7e20cbba45"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 19:33:55.196588 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:55.196507    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a20a9d7-2d6d-4dc7-b6d4-de7e20cbba45-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "0a20a9d7-2d6d-4dc7-b6d4-de7e20cbba45" (UID: "0a20a9d7-2d6d-4dc7-b6d4-de7e20cbba45"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 19:33:55.295221 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:55.295187    2546 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a20a9d7-2d6d-4dc7-b6d4-de7e20cbba45-metrics-certs\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:33:55.295221 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:55.295212    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a20a9d7-2d6d-4dc7-b6d4-de7e20cbba45-maas-api-tls\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:33:55.295221 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:55.295220    2546 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a20a9d7-2d6d-4dc7-b6d4-de7e20cbba45-openshift-service-ca\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:33:55.295483 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:55.295229    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfprr\" (UniqueName: \"kubernetes.io/projected/0a20a9d7-2d6d-4dc7-b6d4-de7e20cbba45-kube-api-access-wfprr\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:33:55.436375 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:55.436319    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-66e9b4-54ffc6b84c-275sc"]
Sep 10 19:33:55.440031 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:55.440001    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-66e9b4-54ffc6b84c-275sc"]
Sep 10 19:33:55.645683 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:55.645647    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-f15c09/sub-test-model-f15c09-kserve-68c8bbd867-52rx6" podUID="bc71d573-ac0c-4021-afdc-e418fd592da9" containerName="main" containerID="cri-o://2c8a674d173cce21e58620f0cb5c39cf9f4da60aee0258737df45bb7755faa03" gracePeriod=45
Sep 10 19:33:55.794988 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:55.794948    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-f15c09/sub-test-model-f15c09-kserve-68c8bbd867-52rx6" podUID="bc71d573-ac0c-4021-afdc-e418fd592da9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": dial tcp 10.133.0.63:8000: connect: connection refused"
Sep 10 19:33:55.900709 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:55.900682    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-f15c09/sub-test-model-f15c09-kserve-68c8bbd867-52rx6"
Sep 10 19:33:56.001979 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:56.001945    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc71d573-ac0c-4021-afdc-e418fd592da9-home\") pod \"bc71d573-ac0c-4021-afdc-e418fd592da9\" (UID: \"bc71d573-ac0c-4021-afdc-e418fd592da9\") "
Sep 10 19:33:56.002172 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:56.002056    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc71d573-ac0c-4021-afdc-e418fd592da9-tmp-dir\") pod \"bc71d573-ac0c-4021-afdc-e418fd592da9\" (UID: \"bc71d573-ac0c-4021-afdc-e418fd592da9\") "
Sep 10 19:33:56.002172 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:56.002079    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc71d573-ac0c-4021-afdc-e418fd592da9-dshm\") pod \"bc71d573-ac0c-4021-afdc-e418fd592da9\" (UID: \"bc71d573-ac0c-4021-afdc-e418fd592da9\") "
Sep 10 19:33:56.002172 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:56.002103    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dxhgk\" (UniqueName: \"kubernetes.io/projected/bc71d573-ac0c-4021-afdc-e418fd592da9-kube-api-access-dxhgk\") pod \"bc71d573-ac0c-4021-afdc-e418fd592da9\" (UID: \"bc71d573-ac0c-4021-afdc-e418fd592da9\") "
Sep 10 19:33:56.002172 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:56.002140    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc71d573-ac0c-4021-afdc-e418fd592da9-tls-certs\") pod \"bc71d573-ac0c-4021-afdc-e418fd592da9\" (UID: \"bc71d573-ac0c-4021-afdc-e418fd592da9\") "
Sep 10 19:33:56.002172 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:56.002163    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc71d573-ac0c-4021-afdc-e418fd592da9-model-cache\") pod \"bc71d573-ac0c-4021-afdc-e418fd592da9\" (UID: \"bc71d573-ac0c-4021-afdc-e418fd592da9\") "
Sep 10 19:33:56.002486 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:56.002246    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc71d573-ac0c-4021-afdc-e418fd592da9-home" (OuterVolumeSpecName: "home") pod "bc71d573-ac0c-4021-afdc-e418fd592da9" (UID: "bc71d573-ac0c-4021-afdc-e418fd592da9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:33:56.002486 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:56.002394    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc71d573-ac0c-4021-afdc-e418fd592da9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bc71d573-ac0c-4021-afdc-e418fd592da9" (UID: "bc71d573-ac0c-4021-afdc-e418fd592da9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:33:56.002596 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:56.002486    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc71d573-ac0c-4021-afdc-e418fd592da9-model-cache" (OuterVolumeSpecName: "model-cache") pod "bc71d573-ac0c-4021-afdc-e418fd592da9" (UID: "bc71d573-ac0c-4021-afdc-e418fd592da9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:33:56.002632 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:56.002610    2546 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc71d573-ac0c-4021-afdc-e418fd592da9-tmp-dir\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:33:56.002632 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:56.002627    2546 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc71d573-ac0c-4021-afdc-e418fd592da9-model-cache\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:33:56.002706 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:56.002635    2546 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc71d573-ac0c-4021-afdc-e418fd592da9-home\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:33:56.004300 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:56.004272    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc71d573-ac0c-4021-afdc-e418fd592da9-kube-api-access-dxhgk" (OuterVolumeSpecName: "kube-api-access-dxhgk") pod "bc71d573-ac0c-4021-afdc-e418fd592da9" (UID: "bc71d573-ac0c-4021-afdc-e418fd592da9"). InnerVolumeSpecName "kube-api-access-dxhgk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 19:33:56.004588 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:56.004571    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc71d573-ac0c-4021-afdc-e418fd592da9-dshm" (OuterVolumeSpecName: "dshm") pod "bc71d573-ac0c-4021-afdc-e418fd592da9" (UID: "bc71d573-ac0c-4021-afdc-e418fd592da9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:33:56.004726 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:56.004710    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc71d573-ac0c-4021-afdc-e418fd592da9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bc71d573-ac0c-4021-afdc-e418fd592da9" (UID: "bc71d573-ac0c-4021-afdc-e418fd592da9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 19:33:56.104042 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:56.103996    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxhgk\" (UniqueName: \"kubernetes.io/projected/bc71d573-ac0c-4021-afdc-e418fd592da9-kube-api-access-dxhgk\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:33:56.104042 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:56.104037    2546 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc71d573-ac0c-4021-afdc-e418fd592da9-tls-certs\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:33:56.104042 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:56.104046    2546 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc71d573-ac0c-4021-afdc-e418fd592da9-dshm\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:33:56.119248 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:56.119218    2546 generic.go:356] "Generic (PLEG): container finished" podID="bc71d573-ac0c-4021-afdc-e418fd592da9" containerID="2c8a674d173cce21e58620f0cb5c39cf9f4da60aee0258737df45bb7755faa03" exitCode=0
Sep 10 19:33:56.119391 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:56.119258    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-f15c09/sub-test-model-f15c09-kserve-68c8bbd867-52rx6"
Sep 10 19:33:56.119391 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:56.119272    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-f15c09/sub-test-model-f15c09-kserve-68c8bbd867-52rx6" event={"ID":"bc71d573-ac0c-4021-afdc-e418fd592da9","Type":"ContainerDied","Data":"2c8a674d173cce21e58620f0cb5c39cf9f4da60aee0258737df45bb7755faa03"}
Sep 10 19:33:56.119391 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:56.119312    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-f15c09/sub-test-model-f15c09-kserve-68c8bbd867-52rx6" event={"ID":"bc71d573-ac0c-4021-afdc-e418fd592da9","Type":"ContainerDied","Data":"3e6552c850122b9e907565b15dd59df8c239170e59b4753649ae9b11a814385f"}
Sep 10 19:33:56.119391 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:56.119330    2546 scope.go:117] "RemoveContainer" containerID="2c8a674d173cce21e58620f0cb5c39cf9f4da60aee0258737df45bb7755faa03"
Sep 10 19:33:56.127953 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:56.127938    2546 scope.go:117] "RemoveContainer" containerID="2c8a674d173cce21e58620f0cb5c39cf9f4da60aee0258737df45bb7755faa03"
Sep 10 19:33:56.128219 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:33:56.128199    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c8a674d173cce21e58620f0cb5c39cf9f4da60aee0258737df45bb7755faa03\": container with ID starting with 2c8a674d173cce21e58620f0cb5c39cf9f4da60aee0258737df45bb7755faa03 not found: ID does not exist" containerID="2c8a674d173cce21e58620f0cb5c39cf9f4da60aee0258737df45bb7755faa03"
Sep 10 19:33:56.128282 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:56.128227    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c8a674d173cce21e58620f0cb5c39cf9f4da60aee0258737df45bb7755faa03"} err="failed to get container status \"2c8a674d173cce21e58620f0cb5c39cf9f4da60aee0258737df45bb7755faa03\": rpc error: code = NotFound desc = could not find container \"2c8a674d173cce21e58620f0cb5c39cf9f4da60aee0258737df45bb7755faa03\": container with ID starting with 2c8a674d173cce21e58620f0cb5c39cf9f4da60aee0258737df45bb7755faa03 not found: ID does not exist"
Sep 10 19:33:56.144447 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:56.144423    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-f15c09/sub-test-model-f15c09-kserve-68c8bbd867-52rx6"]
Sep 10 19:33:56.150531 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:56.150511    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-f15c09/sub-test-model-f15c09-kserve-68c8bbd867-52rx6"]
Sep 10 19:33:57.281444 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:57.281410    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a20a9d7-2d6d-4dc7-b6d4-de7e20cbba45" path="/var/lib/kubelet/pods/0a20a9d7-2d6d-4dc7-b6d4-de7e20cbba45/volumes"
Sep 10 19:33:57.281848 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:57.281834    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc71d573-ac0c-4021-afdc-e418fd592da9" path="/var/lib/kubelet/pods/bc71d573-ac0c-4021-afdc-e418fd592da9/volumes"
Sep 10 19:33:58.836813 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:33:58.836771    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-88n8f"]
Sep 10 19:34:03.622653 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:34:03.622621    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-b64b1c/rate-test-model-b64b1c-kserve-69f895d475-5667b"
Sep 10 19:34:03.622653 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:34:03.622662    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-b64b1c/rate-test-model-b64b1c-kserve-69f895d475-5667b"
Sep 10 19:34:03.635365 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:34:03.635315    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-b64b1c/rate-test-model-b64b1c-kserve-69f895d475-5667b"
Sep 10 19:34:04.177600 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:34:04.177571    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-b64b1c/rate-test-model-b64b1c-kserve-69f895d475-5667b"
Sep 10 19:34:18.236130 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:34:18.236095    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-88n8f"]
Sep 10 19:34:21.219465 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:34:21.219435    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-cfa578-openshift-default-5d84f8dfd6-b8f6s"]
Sep 10 19:34:21.219892 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:34:21.219711    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-cfa578-openshift-default-5d84f8dfd6-b8f6s" podUID="2f9d35e3-2d14-4a94-9e02-30760eeb631d" containerName="istio-proxy" containerID="cri-o://007c8aa351f17dbc8897c33310d9f839a537cc6588b00479b885243f9dc822b9" gracePeriod=30
Sep 10 19:34:23.723204 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:34:23.723167    2546 patch_prober.go:28] interesting pod/e2e-shared-a-w2-cfa578-openshift-default-5d84f8dfd6-b8f6s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 10 19:34:23.723603 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:34:23.723232    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-cfa578-openshift-default-5d84f8dfd6-b8f6s" podUID="2f9d35e3-2d14-4a94-9e02-30760eeb631d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 10 19:34:26.480958 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:34:26.480896    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-cfa578-openshift-default-5d84f8dfd6-b8f6s"
Sep 10 19:34:26.506049 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:34:26.506022    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f9d35e3-2d14-4a94-9e02-30760eeb631d-istiod-ca-cert\") pod \"2f9d35e3-2d14-4a94-9e02-30760eeb631d\" (UID: \"2f9d35e3-2d14-4a94-9e02-30760eeb631d\") "
Sep 10 19:34:26.506180 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:34:26.506058    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f9d35e3-2d14-4a94-9e02-30760eeb631d-istio-podinfo\") pod \"2f9d35e3-2d14-4a94-9e02-30760eeb631d\" (UID: \"2f9d35e3-2d14-4a94-9e02-30760eeb631d\") "
Sep 10 19:34:26.506180 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:34:26.506077    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f9d35e3-2d14-4a94-9e02-30760eeb631d-workload-certs\") pod \"2f9d35e3-2d14-4a94-9e02-30760eeb631d\" (UID: \"2f9d35e3-2d14-4a94-9e02-30760eeb631d\") "
Sep 10 19:34:26.506180 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:34:26.506106    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f9d35e3-2d14-4a94-9e02-30760eeb631d-istio-envoy\") pod \"2f9d35e3-2d14-4a94-9e02-30760eeb631d\" (UID: \"2f9d35e3-2d14-4a94-9e02-30760eeb631d\") "
Sep 10 19:34:26.506180 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:34:26.506146    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f9d35e3-2d14-4a94-9e02-30760eeb631d-credential-socket\") pod \"2f9d35e3-2d14-4a94-9e02-30760eeb631d\" (UID: \"2f9d35e3-2d14-4a94-9e02-30760eeb631d\") "
Sep 10 19:34:26.506180 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:34:26.506169    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7qrkr\" (UniqueName: \"kubernetes.io/projected/2f9d35e3-2d14-4a94-9e02-30760eeb631d-kube-api-access-7qrkr\") pod \"2f9d35e3-2d14-4a94-9e02-30760eeb631d\" (UID: \"2f9d35e3-2d14-4a94-9e02-30760eeb631d\") "
Sep 10 19:34:26.506482 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:34:26.506183    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f9d35e3-2d14-4a94-9e02-30760eeb631d-istio-data\") pod \"2f9d35e3-2d14-4a94-9e02-30760eeb631d\" (UID: \"2f9d35e3-2d14-4a94-9e02-30760eeb631d\") "
Sep 10 19:34:26.506482 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:34:26.506220    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f9d35e3-2d14-4a94-9e02-30760eeb631d-istio-token\") pod \"2f9d35e3-2d14-4a94-9e02-30760eeb631d\" (UID: \"2f9d35e3-2d14-4a94-9e02-30760eeb631d\") "
Sep 10 19:34:26.506482 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:34:26.506389    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f9d35e3-2d14-4a94-9e02-30760eeb631d-workload-socket\") pod \"2f9d35e3-2d14-4a94-9e02-30760eeb631d\" (UID: \"2f9d35e3-2d14-4a94-9e02-30760eeb631d\") "
Sep 10 19:34:26.506482 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:34:26.506439    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f9d35e3-2d14-4a94-9e02-30760eeb631d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2f9d35e3-2d14-4a94-9e02-30760eeb631d" (UID: "2f9d35e3-2d14-4a94-9e02-30760eeb631d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 19:34:26.506698 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:34:26.506463    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f9d35e3-2d14-4a94-9e02-30760eeb631d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2f9d35e3-2d14-4a94-9e02-30760eeb631d" (UID: "2f9d35e3-2d14-4a94-9e02-30760eeb631d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:34:26.506698 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:34:26.506532    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f9d35e3-2d14-4a94-9e02-30760eeb631d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2f9d35e3-2d14-4a94-9e02-30760eeb631d" (UID: "2f9d35e3-2d14-4a94-9e02-30760eeb631d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:34:26.506698 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:34:26.506602    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f9d35e3-2d14-4a94-9e02-30760eeb631d-istio-data" (OuterVolumeSpecName: "istio-data") pod "2f9d35e3-2d14-4a94-9e02-30760eeb631d" (UID: "2f9d35e3-2d14-4a94-9e02-30760eeb631d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:34:26.506698 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:34:26.506676    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f9d35e3-2d14-4a94-9e02-30760eeb631d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2f9d35e3-2d14-4a94-9e02-30760eeb631d" (UID: "2f9d35e3-2d14-4a94-9e02-30760eeb631d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:34:26.506914 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:34:26.506769    2546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f9d35e3-2d14-4a94-9e02-30760eeb631d-workload-certs\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:34:26.506914 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:34:26.506789    2546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f9d35e3-2d14-4a94-9e02-30760eeb631d-credential-socket\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:34:26.506914 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:34:26.506804    2546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f9d35e3-2d14-4a94-9e02-30760eeb631d-istio-data\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:34:26.506914 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:34:26.506817    2546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f9d35e3-2d14-4a94-9e02-30760eeb631d-workload-socket\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:34:26.506914 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:34:26.506829    2546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f9d35e3-2d14-4a94-9e02-30760eeb631d-istiod-ca-cert\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:34:26.508533 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:34:26.508502    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f9d35e3-2d14-4a94-9e02-30760eeb631d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2f9d35e3-2d14-4a94-9e02-30760eeb631d" (UID: "2f9d35e3-2d14-4a94-9e02-30760eeb631d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:34:26.508616 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:34:26.508530    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f9d35e3-2d14-4a94-9e02-30760eeb631d-istio-token" (OuterVolumeSpecName: "istio-token") pod "2f9d35e3-2d14-4a94-9e02-30760eeb631d" (UID: "2f9d35e3-2d14-4a94-9e02-30760eeb631d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 19:34:26.508851 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:34:26.508821    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f9d35e3-2d14-4a94-9e02-30760eeb631d-kube-api-access-7qrkr" (OuterVolumeSpecName: "kube-api-access-7qrkr") pod "2f9d35e3-2d14-4a94-9e02-30760eeb631d" (UID: "2f9d35e3-2d14-4a94-9e02-30760eeb631d"). InnerVolumeSpecName "kube-api-access-7qrkr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 19:34:26.508905 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:34:26.508856    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2f9d35e3-2d14-4a94-9e02-30760eeb631d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2f9d35e3-2d14-4a94-9e02-30760eeb631d" (UID: "2f9d35e3-2d14-4a94-9e02-30760eeb631d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 19:34:26.607913 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:34:26.607882    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qrkr\" (UniqueName: \"kubernetes.io/projected/2f9d35e3-2d14-4a94-9e02-30760eeb631d-kube-api-access-7qrkr\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:34:26.607913 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:34:26.607911    2546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f9d35e3-2d14-4a94-9e02-30760eeb631d-istio-token\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:34:26.607913 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:34:26.607921    2546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f9d35e3-2d14-4a94-9e02-30760eeb631d-istio-podinfo\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:34:26.608160 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:34:26.607929    2546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f9d35e3-2d14-4a94-9e02-30760eeb631d-istio-envoy\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:34:27.289887 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:34:27.289854    2546 generic.go:356] "Generic (PLEG): container finished" podID="2f9d35e3-2d14-4a94-9e02-30760eeb631d" containerID="007c8aa351f17dbc8897c33310d9f839a537cc6588b00479b885243f9dc822b9" exitCode=0
Sep 10 19:34:27.290056 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:34:27.289897    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-cfa578-openshift-default-5d84f8dfd6-b8f6s" event={"ID":"2f9d35e3-2d14-4a94-9e02-30760eeb631d","Type":"ContainerDied","Data":"007c8aa351f17dbc8897c33310d9f839a537cc6588b00479b885243f9dc822b9"}
Sep 10 19:34:27.290056 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:34:27.289927    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-cfa578-openshift-default-5d84f8dfd6-b8f6s"
Sep 10 19:34:27.290056 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:34:27.289954    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-cfa578-openshift-default-5d84f8dfd6-b8f6s" event={"ID":"2f9d35e3-2d14-4a94-9e02-30760eeb631d","Type":"ContainerDied","Data":"27292c9728688b76b7fdbb2aa6f6c2ddd358b38581a00a1e58565f8466d33a9b"}
Sep 10 19:34:27.290056 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:34:27.289978    2546 scope.go:117] "RemoveContainer" containerID="007c8aa351f17dbc8897c33310d9f839a537cc6588b00479b885243f9dc822b9"
Sep 10 19:34:27.299206 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:34:27.299186    2546 scope.go:117] "RemoveContainer" containerID="34b7713956ca261c4297f998ad74a15f30dae7324ee4cd2365cefaa5bd149d7c"
Sep 10 19:34:27.307101 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:34:27.307076    2546 scope.go:117] "RemoveContainer" containerID="007c8aa351f17dbc8897c33310d9f839a537cc6588b00479b885243f9dc822b9"
Sep 10 19:34:27.307484 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:34:27.307429    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"007c8aa351f17dbc8897c33310d9f839a537cc6588b00479b885243f9dc822b9\": container with ID starting with 007c8aa351f17dbc8897c33310d9f839a537cc6588b00479b885243f9dc822b9 not found: ID does not exist" containerID="007c8aa351f17dbc8897c33310d9f839a537cc6588b00479b885243f9dc822b9"
Sep 10 19:34:27.307609 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:34:27.307474    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"007c8aa351f17dbc8897c33310d9f839a537cc6588b00479b885243f9dc822b9"} err="failed to get container status \"007c8aa351f17dbc8897c33310d9f839a537cc6588b00479b885243f9dc822b9\": rpc error: code = NotFound desc = could not find container \"007c8aa351f17dbc8897c33310d9f839a537cc6588b00479b885243f9dc822b9\": container with ID starting with 007c8aa351f17dbc8897c33310d9f839a537cc6588b00479b885243f9dc822b9 not found: ID does not exist"
Sep 10 19:34:27.307609 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:34:27.307499    2546 scope.go:117] "RemoveContainer" containerID="34b7713956ca261c4297f998ad74a15f30dae7324ee4cd2365cefaa5bd149d7c"
Sep 10 19:34:27.307935 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:34:27.307837    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34b7713956ca261c4297f998ad74a15f30dae7324ee4cd2365cefaa5bd149d7c\": container with ID starting with 34b7713956ca261c4297f998ad74a15f30dae7324ee4cd2365cefaa5bd149d7c not found: ID does not exist" containerID="34b7713956ca261c4297f998ad74a15f30dae7324ee4cd2365cefaa5bd149d7c"
Sep 10 19:34:27.307935 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:34:27.307891    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34b7713956ca261c4297f998ad74a15f30dae7324ee4cd2365cefaa5bd149d7c"} err="failed to get container status \"34b7713956ca261c4297f998ad74a15f30dae7324ee4cd2365cefaa5bd149d7c\": rpc error: code = NotFound desc = could not find container \"34b7713956ca261c4297f998ad74a15f30dae7324ee4cd2365cefaa5bd149d7c\": container with ID starting with 34b7713956ca261c4297f998ad74a15f30dae7324ee4cd2365cefaa5bd149d7c not found: ID does not exist"
Sep 10 19:34:27.309905 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:34:27.309884    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-cfa578-openshift-default-5d84f8dfd6-b8f6s"]
Sep 10 19:34:27.317132 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:34:27.317108    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-cfa578-openshift-default-5d84f8dfd6-b8f6s"]
Sep 10 19:34:29.281086 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:34:29.280986    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f9d35e3-2d14-4a94-9e02-30760eeb631d" path="/var/lib/kubelet/pods/2f9d35e3-2d14-4a94-9e02-30760eeb631d/volumes"
Sep 10 19:34:31.749060 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:34:31.749022    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-88n8f"]
Sep 10 19:34:40.308500 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:34:40.308467    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-94ca92-openshift-default-84d989b484-lgrhw"]
Sep 10 19:34:40.309001 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:34:40.308732    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-94ca92-openshift-default-84d989b484-lgrhw" podUID="d7a16dbc-c7e3-4368-bbca-4c80a7604b31" containerName="istio-proxy" containerID="cri-o://7a1cfba999ffbf7e46f993bc5ed080f4ac0989b0f544d215c21cc32ec27a20b8" gracePeriod=30
Sep 10 19:34:43.525433 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:34:43.525398    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-88n8f"]
Sep 10 19:34:45.567572 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:34:45.567539    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-94ca92-openshift-default-84d989b484-lgrhw"
Sep 10 19:34:45.684389 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:34:45.684319    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7a16dbc-c7e3-4368-bbca-4c80a7604b31-credential-socket\") pod \"d7a16dbc-c7e3-4368-bbca-4c80a7604b31\" (UID: \"d7a16dbc-c7e3-4368-bbca-4c80a7604b31\") "
Sep 10 19:34:45.684389 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:34:45.684398    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7a16dbc-c7e3-4368-bbca-4c80a7604b31-workload-socket\") pod \"d7a16dbc-c7e3-4368-bbca-4c80a7604b31\" (UID: \"d7a16dbc-c7e3-4368-bbca-4c80a7604b31\") "
Sep 10 19:34:45.684659 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:34:45.684418    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7a16dbc-c7e3-4368-bbca-4c80a7604b31-workload-certs\") pod \"d7a16dbc-c7e3-4368-bbca-4c80a7604b31\" (UID: \"d7a16dbc-c7e3-4368-bbca-4c80a7604b31\") "
Sep 10 19:34:45.684659 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:34:45.684435    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7a16dbc-c7e3-4368-bbca-4c80a7604b31-istiod-ca-cert\") pod \"d7a16dbc-c7e3-4368-bbca-4c80a7604b31\" (UID: \"d7a16dbc-c7e3-4368-bbca-4c80a7604b31\") "
Sep 10 19:34:45.684659 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:34:45.684451    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7a16dbc-c7e3-4368-bbca-4c80a7604b31-istio-token\") pod \"d7a16dbc-c7e3-4368-bbca-4c80a7604b31\" (UID: \"d7a16dbc-c7e3-4368-bbca-4c80a7604b31\") "
Sep 10 19:34:45.684659 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:34:45.684551    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7a16dbc-c7e3-4368-bbca-4c80a7604b31-istio-data\") pod \"d7a16dbc-c7e3-4368-bbca-4c80a7604b31\" (UID: \"d7a16dbc-c7e3-4368-bbca-4c80a7604b31\") "
Sep 10 19:34:45.684659 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:34:45.684577    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-klhc9\" (UniqueName: \"kubernetes.io/projected/d7a16dbc-c7e3-4368-bbca-4c80a7604b31-kube-api-access-klhc9\") pod \"d7a16dbc-c7e3-4368-bbca-4c80a7604b31\" (UID: \"d7a16dbc-c7e3-4368-bbca-4c80a7604b31\") "
Sep 10 19:34:45.684659 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:34:45.684611    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7a16dbc-c7e3-4368-bbca-4c80a7604b31-istio-podinfo\") pod \"d7a16dbc-c7e3-4368-bbca-4c80a7604b31\" (UID: \"d7a16dbc-c7e3-4368-bbca-4c80a7604b31\") "
Sep 10 19:34:45.684659 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:34:45.684614    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7a16dbc-c7e3-4368-bbca-4c80a7604b31-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d7a16dbc-c7e3-4368-bbca-4c80a7604b31" (UID: "d7a16dbc-c7e3-4368-bbca-4c80a7604b31"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:34:45.684659 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:34:45.684640    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7a16dbc-c7e3-4368-bbca-4c80a7604b31-istio-envoy\") pod \"d7a16dbc-c7e3-4368-bbca-4c80a7604b31\" (UID: \"d7a16dbc-c7e3-4368-bbca-4c80a7604b31\") "
Sep 10 19:34:45.685095 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:34:45.684712    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7a16dbc-c7e3-4368-bbca-4c80a7604b31-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d7a16dbc-c7e3-4368-bbca-4c80a7604b31" (UID: "d7a16dbc-c7e3-4368-bbca-4c80a7604b31"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:34:45.685095 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:34:45.684855    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7a16dbc-c7e3-4368-bbca-4c80a7604b31-istio-data" (OuterVolumeSpecName: "istio-data") pod "d7a16dbc-c7e3-4368-bbca-4c80a7604b31" (UID: "d7a16dbc-c7e3-4368-bbca-4c80a7604b31"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:34:45.685095 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:34:45.684864    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7a16dbc-c7e3-4368-bbca-4c80a7604b31-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d7a16dbc-c7e3-4368-bbca-4c80a7604b31" (UID: "d7a16dbc-c7e3-4368-bbca-4c80a7604b31"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 19:34:45.685095 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:34:45.684994    2546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7a16dbc-c7e3-4368-bbca-4c80a7604b31-istio-data\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:34:45.685095 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:34:45.685013    2546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7a16dbc-c7e3-4368-bbca-4c80a7604b31-credential-socket\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:34:45.685095 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:34:45.685027    2546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7a16dbc-c7e3-4368-bbca-4c80a7604b31-workload-socket\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:34:45.685095 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:34:45.685047    2546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7a16dbc-c7e3-4368-bbca-4c80a7604b31-istiod-ca-cert\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:34:45.685475 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:34:45.685133    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7a16dbc-c7e3-4368-bbca-4c80a7604b31-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d7a16dbc-c7e3-4368-bbca-4c80a7604b31" (UID: "d7a16dbc-c7e3-4368-bbca-4c80a7604b31"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:34:45.687030 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:34:45.687001    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7a16dbc-c7e3-4368-bbca-4c80a7604b31-istio-token" (OuterVolumeSpecName: "istio-token") pod "d7a16dbc-c7e3-4368-bbca-4c80a7604b31" (UID: "d7a16dbc-c7e3-4368-bbca-4c80a7604b31"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 19:34:45.687142 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:34:45.687031    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7a16dbc-c7e3-4368-bbca-4c80a7604b31-kube-api-access-klhc9" (OuterVolumeSpecName: "kube-api-access-klhc9") pod "d7a16dbc-c7e3-4368-bbca-4c80a7604b31" (UID: "d7a16dbc-c7e3-4368-bbca-4c80a7604b31"). InnerVolumeSpecName "kube-api-access-klhc9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 19:34:45.687253 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:34:45.687237    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d7a16dbc-c7e3-4368-bbca-4c80a7604b31-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d7a16dbc-c7e3-4368-bbca-4c80a7604b31" (UID: "d7a16dbc-c7e3-4368-bbca-4c80a7604b31"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 19:34:45.687310 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:34:45.687255    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7a16dbc-c7e3-4368-bbca-4c80a7604b31-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d7a16dbc-c7e3-4368-bbca-4c80a7604b31" (UID: "d7a16dbc-c7e3-4368-bbca-4c80a7604b31"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:34:45.785928 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:34:45.785876    2546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7a16dbc-c7e3-4368-bbca-4c80a7604b31-workload-certs\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:34:45.785928 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:34:45.785924    2546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7a16dbc-c7e3-4368-bbca-4c80a7604b31-istio-token\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:34:45.786168 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:34:45.785939    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klhc9\" (UniqueName: \"kubernetes.io/projected/d7a16dbc-c7e3-4368-bbca-4c80a7604b31-kube-api-access-klhc9\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:34:45.786168 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:34:45.785953    2546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7a16dbc-c7e3-4368-bbca-4c80a7604b31-istio-podinfo\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:34:45.786168 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:34:45.785964    2546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7a16dbc-c7e3-4368-bbca-4c80a7604b31-istio-envoy\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:34:46.387051 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:34:46.387018    2546 generic.go:356] "Generic (PLEG): container finished" podID="d7a16dbc-c7e3-4368-bbca-4c80a7604b31" containerID="7a1cfba999ffbf7e46f993bc5ed080f4ac0989b0f544d215c21cc32ec27a20b8" exitCode=0
Sep 10 19:34:46.387260 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:34:46.387071    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-94ca92-openshift-default-84d989b484-lgrhw"
Sep 10 19:34:46.387260 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:34:46.387103    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-94ca92-openshift-default-84d989b484-lgrhw" event={"ID":"d7a16dbc-c7e3-4368-bbca-4c80a7604b31","Type":"ContainerDied","Data":"7a1cfba999ffbf7e46f993bc5ed080f4ac0989b0f544d215c21cc32ec27a20b8"}
Sep 10 19:34:46.387260 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:34:46.387171    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-94ca92-openshift-default-84d989b484-lgrhw" event={"ID":"d7a16dbc-c7e3-4368-bbca-4c80a7604b31","Type":"ContainerDied","Data":"d6398d63170dc768c740fcbcdbc19730bfab37852b1b62b685db46e0a951fbae"}
Sep 10 19:34:46.387260 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:34:46.387201    2546 scope.go:117] "RemoveContainer" containerID="7a1cfba999ffbf7e46f993bc5ed080f4ac0989b0f544d215c21cc32ec27a20b8"
Sep 10 19:34:46.397429 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:34:46.397413    2546 scope.go:117] "RemoveContainer" containerID="1a1e8d1706687f8aa059de8d61a1b1fc0a6343f5b4addef05c15c44b06b48bb9"
Sep 10 19:34:46.405988 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:34:46.405967    2546 scope.go:117] "RemoveContainer" containerID="7a1cfba999ffbf7e46f993bc5ed080f4ac0989b0f544d215c21cc32ec27a20b8"
Sep 10 19:34:46.406315 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:34:46.406296    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a1cfba999ffbf7e46f993bc5ed080f4ac0989b0f544d215c21cc32ec27a20b8\": container with ID starting with 7a1cfba999ffbf7e46f993bc5ed080f4ac0989b0f544d215c21cc32ec27a20b8 not found: ID does not exist" containerID="7a1cfba999ffbf7e46f993bc5ed080f4ac0989b0f544d215c21cc32ec27a20b8"
Sep 10 19:34:46.406401 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:34:46.406323    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a1cfba999ffbf7e46f993bc5ed080f4ac0989b0f544d215c21cc32ec27a20b8"} err="failed to get container status \"7a1cfba999ffbf7e46f993bc5ed080f4ac0989b0f544d215c21cc32ec27a20b8\": rpc error: code = NotFound desc = could not find container \"7a1cfba999ffbf7e46f993bc5ed080f4ac0989b0f544d215c21cc32ec27a20b8\": container with ID starting with 7a1cfba999ffbf7e46f993bc5ed080f4ac0989b0f544d215c21cc32ec27a20b8 not found: ID does not exist"
Sep 10 19:34:46.406401 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:34:46.406363    2546 scope.go:117] "RemoveContainer" containerID="1a1e8d1706687f8aa059de8d61a1b1fc0a6343f5b4addef05c15c44b06b48bb9"
Sep 10 19:34:46.406642 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:34:46.406624    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a1e8d1706687f8aa059de8d61a1b1fc0a6343f5b4addef05c15c44b06b48bb9\": container with ID starting with 1a1e8d1706687f8aa059de8d61a1b1fc0a6343f5b4addef05c15c44b06b48bb9 not found: ID does not exist" containerID="1a1e8d1706687f8aa059de8d61a1b1fc0a6343f5b4addef05c15c44b06b48bb9"
Sep 10 19:34:46.406690 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:34:46.406650    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a1e8d1706687f8aa059de8d61a1b1fc0a6343f5b4addef05c15c44b06b48bb9"} err="failed to get container status \"1a1e8d1706687f8aa059de8d61a1b1fc0a6343f5b4addef05c15c44b06b48bb9\": rpc error: code = NotFound desc = could not find container \"1a1e8d1706687f8aa059de8d61a1b1fc0a6343f5b4addef05c15c44b06b48bb9\": container with ID starting with 1a1e8d1706687f8aa059de8d61a1b1fc0a6343f5b4addef05c15c44b06b48bb9 not found: ID does not exist"
Sep 10 19:34:46.412421 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:34:46.412398    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-94ca92-openshift-default-84d989b484-lgrhw"]
Sep 10 19:34:46.414639 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:34:46.414619    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-94ca92-openshift-default-84d989b484-lgrhw"]
Sep 10 19:34:47.280498 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:34:47.280464    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d7a16dbc-c7e3-4368-bbca-4c80a7604b31" path="/var/lib/kubelet/pods/d7a16dbc-c7e3-4368-bbca-4c80a7604b31/volumes"
Sep 10 19:34:52.999913 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:34:52.999874    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-88n8f"]
Sep 10 19:34:58.836392 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:34:58.836355    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-b64b1c/rate-test-model-b64b1c-kserve-69f895d475-5667b"]
Sep 10 19:35:00.574672 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:00.574628    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-c59b36-openshift-default-6bd9548cf8-vwr6f"]
Sep 10 19:35:00.575109 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:00.575093    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7a16dbc-c7e3-4368-bbca-4c80a7604b31" containerName="istio-proxy"
Sep 10 19:35:00.575158 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:00.575111    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7a16dbc-c7e3-4368-bbca-4c80a7604b31" containerName="istio-proxy"
Sep 10 19:35:00.575158 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:00.575119    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d817b15-2af4-4c24-a737-e2cfa443ed47" containerName="istio-proxy"
Sep 10 19:35:00.575158 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:00.575125    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d817b15-2af4-4c24-a737-e2cfa443ed47" containerName="istio-proxy"
Sep 10 19:35:00.575158 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:00.575134    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f9d35e3-2d14-4a94-9e02-30760eeb631d" containerName="istio-proxy"
Sep 10 19:35:00.575158 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:00.575140    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f9d35e3-2d14-4a94-9e02-30760eeb631d" containerName="istio-proxy"
Sep 10 19:35:00.575158 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:00.575158    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f9d35e3-2d14-4a94-9e02-30760eeb631d" containerName="istio-proxy"
Sep 10 19:35:00.575386 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:00.575163    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f9d35e3-2d14-4a94-9e02-30760eeb631d" containerName="istio-proxy"
Sep 10 19:35:00.575386 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:00.575170    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a20a9d7-2d6d-4dc7-b6d4-de7e20cbba45" containerName="maas-api"
Sep 10 19:35:00.575386 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:00.575176    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a20a9d7-2d6d-4dc7-b6d4-de7e20cbba45" containerName="maas-api"
Sep 10 19:35:00.575386 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:00.575186    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc71d573-ac0c-4021-afdc-e418fd592da9" containerName="main"
Sep 10 19:35:00.575386 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:00.575191    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc71d573-ac0c-4021-afdc-e418fd592da9" containerName="main"
Sep 10 19:35:00.575386 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:00.575198    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7a16dbc-c7e3-4368-bbca-4c80a7604b31" containerName="istio-proxy"
Sep 10 19:35:00.575386 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:00.575203    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7a16dbc-c7e3-4368-bbca-4c80a7604b31" containerName="istio-proxy"
Sep 10 19:35:00.575386 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:00.575251    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f9d35e3-2d14-4a94-9e02-30760eeb631d" containerName="istio-proxy"
Sep 10 19:35:00.575386 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:00.575257    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f9d35e3-2d14-4a94-9e02-30760eeb631d" containerName="istio-proxy"
Sep 10 19:35:00.575386 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:00.575264    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc71d573-ac0c-4021-afdc-e418fd592da9" containerName="main"
Sep 10 19:35:00.575386 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:00.575270    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7a16dbc-c7e3-4368-bbca-4c80a7604b31" containerName="istio-proxy"
Sep 10 19:35:00.575386 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:00.575275    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a20a9d7-2d6d-4dc7-b6d4-de7e20cbba45" containerName="maas-api"
Sep 10 19:35:00.575386 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:00.575282    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7a16dbc-c7e3-4368-bbca-4c80a7604b31" containerName="istio-proxy"
Sep 10 19:35:00.579870 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:00.579849    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c59b36-openshift-default-6bd9548cf8-vwr6f"
Sep 10 19:35:00.582531 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:00.582508    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-c59b36-openshift-default-dockercfg-tlxn2\""
Sep 10 19:35:00.592259 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:00.592237    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-c59b36-openshift-default-6bd9548cf8-vwr6f"]
Sep 10 19:35:00.728574 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:00.728530    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47463639-8fdd-4475-83a1-862ac426ef5f-workload-socket\") pod \"e2e-ipp-a-c59b36-openshift-default-6bd9548cf8-vwr6f\" (UID: \"47463639-8fdd-4475-83a1-862ac426ef5f\") " pod="openshift-ingress/e2e-ipp-a-c59b36-openshift-default-6bd9548cf8-vwr6f"
Sep 10 19:35:00.728756 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:00.728592    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47463639-8fdd-4475-83a1-862ac426ef5f-workload-certs\") pod \"e2e-ipp-a-c59b36-openshift-default-6bd9548cf8-vwr6f\" (UID: \"47463639-8fdd-4475-83a1-862ac426ef5f\") " pod="openshift-ingress/e2e-ipp-a-c59b36-openshift-default-6bd9548cf8-vwr6f"
Sep 10 19:35:00.728756 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:00.728647    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47463639-8fdd-4475-83a1-862ac426ef5f-istiod-ca-cert\") pod \"e2e-ipp-a-c59b36-openshift-default-6bd9548cf8-vwr6f\" (UID: \"47463639-8fdd-4475-83a1-862ac426ef5f\") " pod="openshift-ingress/e2e-ipp-a-c59b36-openshift-default-6bd9548cf8-vwr6f"
Sep 10 19:35:00.728756 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:00.728678    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47463639-8fdd-4475-83a1-862ac426ef5f-istio-data\") pod \"e2e-ipp-a-c59b36-openshift-default-6bd9548cf8-vwr6f\" (UID: \"47463639-8fdd-4475-83a1-862ac426ef5f\") " pod="openshift-ingress/e2e-ipp-a-c59b36-openshift-default-6bd9548cf8-vwr6f"
Sep 10 19:35:00.728756 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:00.728702    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsw6m\" (UniqueName: \"kubernetes.io/projected/47463639-8fdd-4475-83a1-862ac426ef5f-kube-api-access-zsw6m\") pod \"e2e-ipp-a-c59b36-openshift-default-6bd9548cf8-vwr6f\" (UID: \"47463639-8fdd-4475-83a1-862ac426ef5f\") " pod="openshift-ingress/e2e-ipp-a-c59b36-openshift-default-6bd9548cf8-vwr6f"
Sep 10 19:35:00.728756 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:00.728732    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47463639-8fdd-4475-83a1-862ac426ef5f-istio-envoy\") pod \"e2e-ipp-a-c59b36-openshift-default-6bd9548cf8-vwr6f\" (UID: \"47463639-8fdd-4475-83a1-862ac426ef5f\") " pod="openshift-ingress/e2e-ipp-a-c59b36-openshift-default-6bd9548cf8-vwr6f"
Sep 10 19:35:00.728933 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:00.728770    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47463639-8fdd-4475-83a1-862ac426ef5f-credential-socket\") pod \"e2e-ipp-a-c59b36-openshift-default-6bd9548cf8-vwr6f\" (UID: \"47463639-8fdd-4475-83a1-862ac426ef5f\") " pod="openshift-ingress/e2e-ipp-a-c59b36-openshift-default-6bd9548cf8-vwr6f"
Sep 10 19:35:00.728933 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:00.728796    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47463639-8fdd-4475-83a1-862ac426ef5f-istio-podinfo\") pod \"e2e-ipp-a-c59b36-openshift-default-6bd9548cf8-vwr6f\" (UID: \"47463639-8fdd-4475-83a1-862ac426ef5f\") " pod="openshift-ingress/e2e-ipp-a-c59b36-openshift-default-6bd9548cf8-vwr6f"
Sep 10 19:35:00.728933 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:00.728839    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47463639-8fdd-4475-83a1-862ac426ef5f-istio-token\") pod \"e2e-ipp-a-c59b36-openshift-default-6bd9548cf8-vwr6f\" (UID: \"47463639-8fdd-4475-83a1-862ac426ef5f\") " pod="openshift-ingress/e2e-ipp-a-c59b36-openshift-default-6bd9548cf8-vwr6f"
Sep 10 19:35:00.830445 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:00.830322    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47463639-8fdd-4475-83a1-862ac426ef5f-credential-socket\") pod \"e2e-ipp-a-c59b36-openshift-default-6bd9548cf8-vwr6f\" (UID: \"47463639-8fdd-4475-83a1-862ac426ef5f\") " pod="openshift-ingress/e2e-ipp-a-c59b36-openshift-default-6bd9548cf8-vwr6f"
Sep 10 19:35:00.830445 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:00.830390    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47463639-8fdd-4475-83a1-862ac426ef5f-istio-podinfo\") pod \"e2e-ipp-a-c59b36-openshift-default-6bd9548cf8-vwr6f\" (UID: \"47463639-8fdd-4475-83a1-862ac426ef5f\") " pod="openshift-ingress/e2e-ipp-a-c59b36-openshift-default-6bd9548cf8-vwr6f"
Sep 10 19:35:00.830445 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:00.830414    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47463639-8fdd-4475-83a1-862ac426ef5f-istio-token\") pod \"e2e-ipp-a-c59b36-openshift-default-6bd9548cf8-vwr6f\" (UID: \"47463639-8fdd-4475-83a1-862ac426ef5f\") " pod="openshift-ingress/e2e-ipp-a-c59b36-openshift-default-6bd9548cf8-vwr6f"
Sep 10 19:35:00.830758 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:00.830469    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47463639-8fdd-4475-83a1-862ac426ef5f-workload-socket\") pod \"e2e-ipp-a-c59b36-openshift-default-6bd9548cf8-vwr6f\" (UID: \"47463639-8fdd-4475-83a1-862ac426ef5f\") " pod="openshift-ingress/e2e-ipp-a-c59b36-openshift-default-6bd9548cf8-vwr6f"
Sep 10 19:35:00.830758 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:00.830498    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47463639-8fdd-4475-83a1-862ac426ef5f-workload-certs\") pod \"e2e-ipp-a-c59b36-openshift-default-6bd9548cf8-vwr6f\" (UID: \"47463639-8fdd-4475-83a1-862ac426ef5f\") " pod="openshift-ingress/e2e-ipp-a-c59b36-openshift-default-6bd9548cf8-vwr6f"
Sep 10 19:35:00.830758 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:00.830555    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47463639-8fdd-4475-83a1-862ac426ef5f-istiod-ca-cert\") pod \"e2e-ipp-a-c59b36-openshift-default-6bd9548cf8-vwr6f\" (UID: \"47463639-8fdd-4475-83a1-862ac426ef5f\") " pod="openshift-ingress/e2e-ipp-a-c59b36-openshift-default-6bd9548cf8-vwr6f"
Sep 10 19:35:00.830758 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:00.830589    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47463639-8fdd-4475-83a1-862ac426ef5f-istio-data\") pod \"e2e-ipp-a-c59b36-openshift-default-6bd9548cf8-vwr6f\" (UID: \"47463639-8fdd-4475-83a1-862ac426ef5f\") " pod="openshift-ingress/e2e-ipp-a-c59b36-openshift-default-6bd9548cf8-vwr6f"
Sep 10 19:35:00.830758 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:00.830611    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsw6m\" (UniqueName: \"kubernetes.io/projected/47463639-8fdd-4475-83a1-862ac426ef5f-kube-api-access-zsw6m\") pod \"e2e-ipp-a-c59b36-openshift-default-6bd9548cf8-vwr6f\" (UID: \"47463639-8fdd-4475-83a1-862ac426ef5f\") " pod="openshift-ingress/e2e-ipp-a-c59b36-openshift-default-6bd9548cf8-vwr6f"
Sep 10 19:35:00.830758 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:00.830641    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47463639-8fdd-4475-83a1-862ac426ef5f-istio-envoy\") pod \"e2e-ipp-a-c59b36-openshift-default-6bd9548cf8-vwr6f\" (UID: \"47463639-8fdd-4475-83a1-862ac426ef5f\") " pod="openshift-ingress/e2e-ipp-a-c59b36-openshift-default-6bd9548cf8-vwr6f"
Sep 10 19:35:00.831111 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:00.830814    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47463639-8fdd-4475-83a1-862ac426ef5f-credential-socket\") pod \"e2e-ipp-a-c59b36-openshift-default-6bd9548cf8-vwr6f\" (UID: \"47463639-8fdd-4475-83a1-862ac426ef5f\") " pod="openshift-ingress/e2e-ipp-a-c59b36-openshift-default-6bd9548cf8-vwr6f"
Sep 10 19:35:00.831111 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:00.830861    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47463639-8fdd-4475-83a1-862ac426ef5f-workload-socket\") pod \"e2e-ipp-a-c59b36-openshift-default-6bd9548cf8-vwr6f\" (UID: \"47463639-8fdd-4475-83a1-862ac426ef5f\") " pod="openshift-ingress/e2e-ipp-a-c59b36-openshift-default-6bd9548cf8-vwr6f"
Sep 10 19:35:00.831111 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:00.830925    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47463639-8fdd-4475-83a1-862ac426ef5f-workload-certs\") pod \"e2e-ipp-a-c59b36-openshift-default-6bd9548cf8-vwr6f\" (UID: \"47463639-8fdd-4475-83a1-862ac426ef5f\") " pod="openshift-ingress/e2e-ipp-a-c59b36-openshift-default-6bd9548cf8-vwr6f"
Sep 10 19:35:00.831111 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:00.831027    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47463639-8fdd-4475-83a1-862ac426ef5f-istio-data\") pod \"e2e-ipp-a-c59b36-openshift-default-6bd9548cf8-vwr6f\" (UID: \"47463639-8fdd-4475-83a1-862ac426ef5f\") " pod="openshift-ingress/e2e-ipp-a-c59b36-openshift-default-6bd9548cf8-vwr6f"
Sep 10 19:35:00.831444 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:00.831422    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47463639-8fdd-4475-83a1-862ac426ef5f-istiod-ca-cert\") pod \"e2e-ipp-a-c59b36-openshift-default-6bd9548cf8-vwr6f\" (UID: \"47463639-8fdd-4475-83a1-862ac426ef5f\") " pod="openshift-ingress/e2e-ipp-a-c59b36-openshift-default-6bd9548cf8-vwr6f"
Sep 10 19:35:00.832985 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:00.832953    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47463639-8fdd-4475-83a1-862ac426ef5f-istio-envoy\") pod \"e2e-ipp-a-c59b36-openshift-default-6bd9548cf8-vwr6f\" (UID: \"47463639-8fdd-4475-83a1-862ac426ef5f\") " pod="openshift-ingress/e2e-ipp-a-c59b36-openshift-default-6bd9548cf8-vwr6f"
Sep 10 19:35:00.833089 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:00.832989    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47463639-8fdd-4475-83a1-862ac426ef5f-istio-podinfo\") pod \"e2e-ipp-a-c59b36-openshift-default-6bd9548cf8-vwr6f\" (UID: \"47463639-8fdd-4475-83a1-862ac426ef5f\") " pod="openshift-ingress/e2e-ipp-a-c59b36-openshift-default-6bd9548cf8-vwr6f"
Sep 10 19:35:00.839654 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:00.839634    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47463639-8fdd-4475-83a1-862ac426ef5f-istio-token\") pod \"e2e-ipp-a-c59b36-openshift-default-6bd9548cf8-vwr6f\" (UID: \"47463639-8fdd-4475-83a1-862ac426ef5f\") " pod="openshift-ingress/e2e-ipp-a-c59b36-openshift-default-6bd9548cf8-vwr6f"
Sep 10 19:35:00.839981 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:00.839959    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsw6m\" (UniqueName: \"kubernetes.io/projected/47463639-8fdd-4475-83a1-862ac426ef5f-kube-api-access-zsw6m\") pod \"e2e-ipp-a-c59b36-openshift-default-6bd9548cf8-vwr6f\" (UID: \"47463639-8fdd-4475-83a1-862ac426ef5f\") " pod="openshift-ingress/e2e-ipp-a-c59b36-openshift-default-6bd9548cf8-vwr6f"
Sep 10 19:35:00.890907 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:00.890869    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c59b36-openshift-default-6bd9548cf8-vwr6f"
Sep 10 19:35:01.172679 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:01.172648    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-c59b36-openshift-default-6bd9548cf8-vwr6f"]
Sep 10 19:35:01.174767 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:35:01.174740    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47463639_8fdd_4475_83a1_862ac426ef5f.slice/crio-6a21e814f53909b534e0dc8b616a72e81e6014fab184a446839d1dd596aa8dd1 WatchSource:0}: Error finding container 6a21e814f53909b534e0dc8b616a72e81e6014fab184a446839d1dd596aa8dd1: Status 404 returned error can't find the container with id 6a21e814f53909b534e0dc8b616a72e81e6014fab184a446839d1dd596aa8dd1
Sep 10 19:35:01.177620 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:01.177583    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 19:35:01.177730 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:01.177652    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 19:35:01.177730 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:01.177681    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 19:35:01.462848 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:01.462757    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c59b36-openshift-default-6bd9548cf8-vwr6f" event={"ID":"47463639-8fdd-4475-83a1-862ac426ef5f","Type":"ContainerStarted","Data":"d6988a8f4b4854c9022ba8e260af0324c1514f29d0f4d2fd4fa91b6d409d2885"}
Sep 10 19:35:01.462848 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:01.462827    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c59b36-openshift-default-6bd9548cf8-vwr6f" event={"ID":"47463639-8fdd-4475-83a1-862ac426ef5f","Type":"ContainerStarted","Data":"6a21e814f53909b534e0dc8b616a72e81e6014fab184a446839d1dd596aa8dd1"}
Sep 10 19:35:01.891249 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:01.891215    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-c59b36-openshift-default-6bd9548cf8-vwr6f"
Sep 10 19:35:02.896368 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:02.896326    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-c59b36-openshift-default-6bd9548cf8-vwr6f"
Sep 10 19:35:02.922256 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:02.922204    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-c59b36-openshift-default-6bd9548cf8-vwr6f" podStartSLOduration=2.92218694 podStartE2EDuration="2.92218694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:35:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:35:01.509464548 +0000 UTC m=+1464.738514573" watchObservedRunningTime="2026-09-10 19:35:02.92218694 +0000 UTC m=+1466.151236964"
Sep 10 19:35:03.475069 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:03.475042    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-c59b36-openshift-default-6bd9548cf8-vwr6f"
Sep 10 19:35:03.475927 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:03.475909    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-c59b36-openshift-default-6bd9548cf8-vwr6f"
Sep 10 19:35:08.625562 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:08.625523    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-88n8f"]
Sep 10 19:35:13.856022 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:13.855977    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-b64b1c/rate-test-model-b64b1c-kserve-69f895d475-5667b" podUID="e4b738a7-b6fc-4e0c-86de-d287cb2f2a61" containerName="main" containerID="cri-o://aec855cbe4b29e1f13ecbb9f403825de2404a4150a151ae3c545dfb9db313a69" gracePeriod=45
Sep 10 19:35:14.140846 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:14.140816    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-b64b1c/rate-test-model-b64b1c-kserve-69f895d475-5667b"
Sep 10 19:35:14.142015 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:14.141996    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pk42h\" (UniqueName: \"kubernetes.io/projected/e4b738a7-b6fc-4e0c-86de-d287cb2f2a61-kube-api-access-pk42h\") pod \"e4b738a7-b6fc-4e0c-86de-d287cb2f2a61\" (UID: \"e4b738a7-b6fc-4e0c-86de-d287cb2f2a61\") "
Sep 10 19:35:14.142089 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:14.142041    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4b738a7-b6fc-4e0c-86de-d287cb2f2a61-home\") pod \"e4b738a7-b6fc-4e0c-86de-d287cb2f2a61\" (UID: \"e4b738a7-b6fc-4e0c-86de-d287cb2f2a61\") "
Sep 10 19:35:14.142089 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:14.142074    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4b738a7-b6fc-4e0c-86de-d287cb2f2a61-tmp-dir\") pod \"e4b738a7-b6fc-4e0c-86de-d287cb2f2a61\" (UID: \"e4b738a7-b6fc-4e0c-86de-d287cb2f2a61\") "
Sep 10 19:35:14.142202 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:14.142174    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4b738a7-b6fc-4e0c-86de-d287cb2f2a61-dshm\") pod \"e4b738a7-b6fc-4e0c-86de-d287cb2f2a61\" (UID: \"e4b738a7-b6fc-4e0c-86de-d287cb2f2a61\") "
Sep 10 19:35:14.142257 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:14.142201    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4b738a7-b6fc-4e0c-86de-d287cb2f2a61-model-cache\") pod \"e4b738a7-b6fc-4e0c-86de-d287cb2f2a61\" (UID: \"e4b738a7-b6fc-4e0c-86de-d287cb2f2a61\") "
Sep 10 19:35:14.142418 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:14.142316    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4b738a7-b6fc-4e0c-86de-d287cb2f2a61-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e4b738a7-b6fc-4e0c-86de-d287cb2f2a61" (UID: "e4b738a7-b6fc-4e0c-86de-d287cb2f2a61"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:35:14.142418 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:14.142362    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4b738a7-b6fc-4e0c-86de-d287cb2f2a61-tls-certs\") pod \"e4b738a7-b6fc-4e0c-86de-d287cb2f2a61\" (UID: \"e4b738a7-b6fc-4e0c-86de-d287cb2f2a61\") "
Sep 10 19:35:14.142418 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:14.142368    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4b738a7-b6fc-4e0c-86de-d287cb2f2a61-home" (OuterVolumeSpecName: "home") pod "e4b738a7-b6fc-4e0c-86de-d287cb2f2a61" (UID: "e4b738a7-b6fc-4e0c-86de-d287cb2f2a61"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:35:14.142591 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:14.142511    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4b738a7-b6fc-4e0c-86de-d287cb2f2a61-model-cache" (OuterVolumeSpecName: "model-cache") pod "e4b738a7-b6fc-4e0c-86de-d287cb2f2a61" (UID: "e4b738a7-b6fc-4e0c-86de-d287cb2f2a61"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:35:14.142724 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:14.142703    2546 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4b738a7-b6fc-4e0c-86de-d287cb2f2a61-home\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:35:14.142724 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:14.142721    2546 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4b738a7-b6fc-4e0c-86de-d287cb2f2a61-tmp-dir\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:35:14.142903 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:14.142735    2546 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4b738a7-b6fc-4e0c-86de-d287cb2f2a61-model-cache\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:35:14.144272 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:14.144249    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4b738a7-b6fc-4e0c-86de-d287cb2f2a61-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e4b738a7-b6fc-4e0c-86de-d287cb2f2a61" (UID: "e4b738a7-b6fc-4e0c-86de-d287cb2f2a61"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 19:35:14.144413 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:14.144392    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4b738a7-b6fc-4e0c-86de-d287cb2f2a61-dshm" (OuterVolumeSpecName: "dshm") pod "e4b738a7-b6fc-4e0c-86de-d287cb2f2a61" (UID: "e4b738a7-b6fc-4e0c-86de-d287cb2f2a61"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:35:14.144474 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:14.144395    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4b738a7-b6fc-4e0c-86de-d287cb2f2a61-kube-api-access-pk42h" (OuterVolumeSpecName: "kube-api-access-pk42h") pod "e4b738a7-b6fc-4e0c-86de-d287cb2f2a61" (UID: "e4b738a7-b6fc-4e0c-86de-d287cb2f2a61"). InnerVolumeSpecName "kube-api-access-pk42h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 19:35:14.243958 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:14.243920    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pk42h\" (UniqueName: \"kubernetes.io/projected/e4b738a7-b6fc-4e0c-86de-d287cb2f2a61-kube-api-access-pk42h\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:35:14.243958 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:14.243952    2546 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4b738a7-b6fc-4e0c-86de-d287cb2f2a61-dshm\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:35:14.243958 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:14.243961    2546 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4b738a7-b6fc-4e0c-86de-d287cb2f2a61-tls-certs\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:35:14.527809 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:14.527730    2546 generic.go:356] "Generic (PLEG): container finished" podID="e4b738a7-b6fc-4e0c-86de-d287cb2f2a61" containerID="aec855cbe4b29e1f13ecbb9f403825de2404a4150a151ae3c545dfb9db313a69" exitCode=0
Sep 10 19:35:14.527809 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:14.527774    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-b64b1c/rate-test-model-b64b1c-kserve-69f895d475-5667b"
Sep 10 19:35:14.528044 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:14.527817    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-b64b1c/rate-test-model-b64b1c-kserve-69f895d475-5667b" event={"ID":"e4b738a7-b6fc-4e0c-86de-d287cb2f2a61","Type":"ContainerDied","Data":"aec855cbe4b29e1f13ecbb9f403825de2404a4150a151ae3c545dfb9db313a69"}
Sep 10 19:35:14.528044 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:14.527867    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-b64b1c/rate-test-model-b64b1c-kserve-69f895d475-5667b" event={"ID":"e4b738a7-b6fc-4e0c-86de-d287cb2f2a61","Type":"ContainerDied","Data":"c087e78dc0a5e692e2f0aa57e96c33d19aa8ece32fc1e17dad05bf1e3e6df78d"}
Sep 10 19:35:14.528044 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:14.527890    2546 scope.go:117] "RemoveContainer" containerID="aec855cbe4b29e1f13ecbb9f403825de2404a4150a151ae3c545dfb9db313a69"
Sep 10 19:35:14.539361 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:14.539319    2546 scope.go:117] "RemoveContainer" containerID="aec855cbe4b29e1f13ecbb9f403825de2404a4150a151ae3c545dfb9db313a69"
Sep 10 19:35:14.539858 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:35:14.539837    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aec855cbe4b29e1f13ecbb9f403825de2404a4150a151ae3c545dfb9db313a69\": container with ID starting with aec855cbe4b29e1f13ecbb9f403825de2404a4150a151ae3c545dfb9db313a69 not found: ID does not exist" containerID="aec855cbe4b29e1f13ecbb9f403825de2404a4150a151ae3c545dfb9db313a69"
Sep 10 19:35:14.539937 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:14.539867    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aec855cbe4b29e1f13ecbb9f403825de2404a4150a151ae3c545dfb9db313a69"} err="failed to get container status \"aec855cbe4b29e1f13ecbb9f403825de2404a4150a151ae3c545dfb9db313a69\": rpc error: code = NotFound desc = could not find container \"aec855cbe4b29e1f13ecbb9f403825de2404a4150a151ae3c545dfb9db313a69\": container with ID starting with aec855cbe4b29e1f13ecbb9f403825de2404a4150a151ae3c545dfb9db313a69 not found: ID does not exist"
Sep 10 19:35:14.549128 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:14.549104    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-b64b1c/rate-test-model-b64b1c-kserve-69f895d475-5667b"]
Sep 10 19:35:14.553450 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:14.553428    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-b64b1c/rate-test-model-b64b1c-kserve-69f895d475-5667b"]
Sep 10 19:35:15.282662 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:15.282579    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4b738a7-b6fc-4e0c-86de-d287cb2f2a61" path="/var/lib/kubelet/pods/e4b738a7-b6fc-4e0c-86de-d287cb2f2a61/volumes"
Sep 10 19:35:18.476444 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:18.476403    2546 patch_prober.go:28] interesting pod/e2e-ipp-a-c59b36-openshift-default-6bd9548cf8-vwr6f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Sep 10 19:35:18.476885 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:18.476464    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-c59b36-openshift-default-6bd9548cf8-vwr6f" podUID="47463639-8fdd-4475-83a1-862ac426ef5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Sep 10 19:35:18.549923 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:18.549888    2546 generic.go:356] "Generic (PLEG): container finished" podID="47463639-8fdd-4475-83a1-862ac426ef5f" containerID="d6988a8f4b4854c9022ba8e260af0324c1514f29d0f4d2fd4fa91b6d409d2885" exitCode=0
Sep 10 19:35:18.550097 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:18.549960    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c59b36-openshift-default-6bd9548cf8-vwr6f" event={"ID":"47463639-8fdd-4475-83a1-862ac426ef5f","Type":"ContainerDied","Data":"d6988a8f4b4854c9022ba8e260af0324c1514f29d0f4d2fd4fa91b6d409d2885"}
Sep 10 19:35:18.550433 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:18.550418    2546 scope.go:117] "RemoveContainer" containerID="d6988a8f4b4854c9022ba8e260af0324c1514f29d0f4d2fd4fa91b6d409d2885"
Sep 10 19:35:18.551494 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:18.551465    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 19:35:18.551556 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:18.551539    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 19:35:18.551595 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:18.551570    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 19:35:19.557224 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:19.557174    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c59b36-openshift-default-6bd9548cf8-vwr6f" event={"ID":"47463639-8fdd-4475-83a1-862ac426ef5f","Type":"ContainerStarted","Data":"3a8e284bacb6d603f61104de0e3790c46365e8507eb4907b14ab9f4380e36233"}
Sep 10 19:35:19.891457 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:19.891410    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-c59b36-openshift-default-6bd9548cf8-vwr6f"
Sep 10 19:35:19.891738 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:19.891714    2546 patch_prober.go:28] interesting pod/e2e-ipp-a-c59b36-openshift-default-6bd9548cf8-vwr6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Sep 10 19:35:19.891813 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:19.891793    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c59b36-openshift-default-6bd9548cf8-vwr6f" podUID="47463639-8fdd-4475-83a1-862ac426ef5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Sep 10 19:35:20.891321 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:20.891281    2546 patch_prober.go:28] interesting pod/e2e-ipp-a-c59b36-openshift-default-6bd9548cf8-vwr6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Sep 10 19:35:20.891756 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:20.891353    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c59b36-openshift-default-6bd9548cf8-vwr6f" podUID="47463639-8fdd-4475-83a1-862ac426ef5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Sep 10 19:35:21.892040 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:21.892001    2546 patch_prober.go:28] interesting pod/e2e-ipp-a-c59b36-openshift-default-6bd9548cf8-vwr6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Sep 10 19:35:21.892491 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:21.892062    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c59b36-openshift-default-6bd9548cf8-vwr6f" podUID="47463639-8fdd-4475-83a1-862ac426ef5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Sep 10 19:35:22.891895 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:22.891857    2546 patch_prober.go:28] interesting pod/e2e-ipp-a-c59b36-openshift-default-6bd9548cf8-vwr6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Sep 10 19:35:22.892078 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:22.891919    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c59b36-openshift-default-6bd9548cf8-vwr6f" podUID="47463639-8fdd-4475-83a1-862ac426ef5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Sep 10 19:35:23.891620 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:23.891580    2546 patch_prober.go:28] interesting pod/e2e-ipp-a-c59b36-openshift-default-6bd9548cf8-vwr6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Sep 10 19:35:23.891822 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:23.891638    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c59b36-openshift-default-6bd9548cf8-vwr6f" podUID="47463639-8fdd-4475-83a1-862ac426ef5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Sep 10 19:35:24.895450 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:24.895415    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-c59b36-openshift-default-6bd9548cf8-vwr6f"
Sep 10 19:35:24.895916 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:24.895679    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-c59b36-openshift-default-6bd9548cf8-vwr6f"
Sep 10 19:35:24.896404 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:24.896386    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-c59b36-openshift-default-6bd9548cf8-vwr6f"
Sep 10 19:35:29.444862 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:29.444830    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-90ceb3-openshift-default-75b485b86d-259zj"]
Sep 10 19:35:29.445377 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:29.445361    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4b738a7-b6fc-4e0c-86de-d287cb2f2a61" containerName="main"
Sep 10 19:35:29.445432 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:29.445380    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4b738a7-b6fc-4e0c-86de-d287cb2f2a61" containerName="main"
Sep 10 19:35:29.446373 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:29.445992    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4b738a7-b6fc-4e0c-86de-d287cb2f2a61" containerName="main"
Sep 10 19:35:29.450289 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:29.450255    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-90ceb3-openshift-default-75b485b86d-259zj"
Sep 10 19:35:29.454254 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:29.454176    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-90ceb3-openshift-default-dockercfg-pgnxn\""
Sep 10 19:35:29.462705 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:29.462677    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-90ceb3-openshift-default-75b485b86d-259zj"]
Sep 10 19:35:29.492163 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:29.492126    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/868773b2-7d37-4196-9ab3-2256b295f31d-istiod-ca-cert\") pod \"e2e-ipp-b-90ceb3-openshift-default-75b485b86d-259zj\" (UID: \"868773b2-7d37-4196-9ab3-2256b295f31d\") " pod="openshift-ingress/e2e-ipp-b-90ceb3-openshift-default-75b485b86d-259zj"
Sep 10 19:35:29.492163 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:29.492164    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/868773b2-7d37-4196-9ab3-2256b295f31d-workload-socket\") pod \"e2e-ipp-b-90ceb3-openshift-default-75b485b86d-259zj\" (UID: \"868773b2-7d37-4196-9ab3-2256b295f31d\") " pod="openshift-ingress/e2e-ipp-b-90ceb3-openshift-default-75b485b86d-259zj"
Sep 10 19:35:29.492429 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:29.492182    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/868773b2-7d37-4196-9ab3-2256b295f31d-istio-token\") pod \"e2e-ipp-b-90ceb3-openshift-default-75b485b86d-259zj\" (UID: \"868773b2-7d37-4196-9ab3-2256b295f31d\") " pod="openshift-ingress/e2e-ipp-b-90ceb3-openshift-default-75b485b86d-259zj"
Sep 10 19:35:29.492429 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:29.492207    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/868773b2-7d37-4196-9ab3-2256b295f31d-workload-certs\") pod \"e2e-ipp-b-90ceb3-openshift-default-75b485b86d-259zj\" (UID: \"868773b2-7d37-4196-9ab3-2256b295f31d\") " pod="openshift-ingress/e2e-ipp-b-90ceb3-openshift-default-75b485b86d-259zj"
Sep 10 19:35:29.492429 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:29.492253    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/868773b2-7d37-4196-9ab3-2256b295f31d-istio-data\") pod \"e2e-ipp-b-90ceb3-openshift-default-75b485b86d-259zj\" (UID: \"868773b2-7d37-4196-9ab3-2256b295f31d\") " pod="openshift-ingress/e2e-ipp-b-90ceb3-openshift-default-75b485b86d-259zj"
Sep 10 19:35:29.492429 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:29.492274    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/868773b2-7d37-4196-9ab3-2256b295f31d-credential-socket\") pod \"e2e-ipp-b-90ceb3-openshift-default-75b485b86d-259zj\" (UID: \"868773b2-7d37-4196-9ab3-2256b295f31d\") " pod="openshift-ingress/e2e-ipp-b-90ceb3-openshift-default-75b485b86d-259zj"
Sep 10 19:35:29.492429 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:29.492290    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/868773b2-7d37-4196-9ab3-2256b295f31d-istio-podinfo\") pod \"e2e-ipp-b-90ceb3-openshift-default-75b485b86d-259zj\" (UID: \"868773b2-7d37-4196-9ab3-2256b295f31d\") " pod="openshift-ingress/e2e-ipp-b-90ceb3-openshift-default-75b485b86d-259zj"
Sep 10 19:35:29.492429 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:29.492317    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/868773b2-7d37-4196-9ab3-2256b295f31d-istio-envoy\") pod \"e2e-ipp-b-90ceb3-openshift-default-75b485b86d-259zj\" (UID: \"868773b2-7d37-4196-9ab3-2256b295f31d\") " pod="openshift-ingress/e2e-ipp-b-90ceb3-openshift-default-75b485b86d-259zj"
Sep 10 19:35:29.492429 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:29.492404    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgt72\" (UniqueName: \"kubernetes.io/projected/868773b2-7d37-4196-9ab3-2256b295f31d-kube-api-access-qgt72\") pod \"e2e-ipp-b-90ceb3-openshift-default-75b485b86d-259zj\" (UID: \"868773b2-7d37-4196-9ab3-2256b295f31d\") " pod="openshift-ingress/e2e-ipp-b-90ceb3-openshift-default-75b485b86d-259zj"
Sep 10 19:35:29.593666 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:29.593622    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/868773b2-7d37-4196-9ab3-2256b295f31d-istiod-ca-cert\") pod \"e2e-ipp-b-90ceb3-openshift-default-75b485b86d-259zj\" (UID: \"868773b2-7d37-4196-9ab3-2256b295f31d\") " pod="openshift-ingress/e2e-ipp-b-90ceb3-openshift-default-75b485b86d-259zj"
Sep 10 19:35:29.593666 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:29.593663    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/868773b2-7d37-4196-9ab3-2256b295f31d-workload-socket\") pod \"e2e-ipp-b-90ceb3-openshift-default-75b485b86d-259zj\" (UID: \"868773b2-7d37-4196-9ab3-2256b295f31d\") " pod="openshift-ingress/e2e-ipp-b-90ceb3-openshift-default-75b485b86d-259zj"
Sep 10 19:35:29.593918 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:29.593680    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/868773b2-7d37-4196-9ab3-2256b295f31d-istio-token\") pod \"e2e-ipp-b-90ceb3-openshift-default-75b485b86d-259zj\" (UID: \"868773b2-7d37-4196-9ab3-2256b295f31d\") " pod="openshift-ingress/e2e-ipp-b-90ceb3-openshift-default-75b485b86d-259zj"
Sep 10 19:35:29.593918 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:29.593696    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/868773b2-7d37-4196-9ab3-2256b295f31d-workload-certs\") pod \"e2e-ipp-b-90ceb3-openshift-default-75b485b86d-259zj\" (UID: \"868773b2-7d37-4196-9ab3-2256b295f31d\") " pod="openshift-ingress/e2e-ipp-b-90ceb3-openshift-default-75b485b86d-259zj"
Sep 10 19:35:29.593918 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:29.593740    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/868773b2-7d37-4196-9ab3-2256b295f31d-istio-data\") pod \"e2e-ipp-b-90ceb3-openshift-default-75b485b86d-259zj\" (UID: \"868773b2-7d37-4196-9ab3-2256b295f31d\") " pod="openshift-ingress/e2e-ipp-b-90ceb3-openshift-default-75b485b86d-259zj"
Sep 10 19:35:29.593918 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:29.593775    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/868773b2-7d37-4196-9ab3-2256b295f31d-credential-socket\") pod \"e2e-ipp-b-90ceb3-openshift-default-75b485b86d-259zj\" (UID: \"868773b2-7d37-4196-9ab3-2256b295f31d\") " pod="openshift-ingress/e2e-ipp-b-90ceb3-openshift-default-75b485b86d-259zj"
Sep 10 19:35:29.593918 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:29.593796    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/868773b2-7d37-4196-9ab3-2256b295f31d-istio-podinfo\") pod \"e2e-ipp-b-90ceb3-openshift-default-75b485b86d-259zj\" (UID: \"868773b2-7d37-4196-9ab3-2256b295f31d\") " pod="openshift-ingress/e2e-ipp-b-90ceb3-openshift-default-75b485b86d-259zj"
Sep 10 19:35:29.594180 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:29.594107    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/868773b2-7d37-4196-9ab3-2256b295f31d-istio-envoy\") pod \"e2e-ipp-b-90ceb3-openshift-default-75b485b86d-259zj\" (UID: \"868773b2-7d37-4196-9ab3-2256b295f31d\") " pod="openshift-ingress/e2e-ipp-b-90ceb3-openshift-default-75b485b86d-259zj"
Sep 10 19:35:29.594180 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:29.594126    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/868773b2-7d37-4196-9ab3-2256b295f31d-workload-socket\") pod \"e2e-ipp-b-90ceb3-openshift-default-75b485b86d-259zj\" (UID: \"868773b2-7d37-4196-9ab3-2256b295f31d\") " pod="openshift-ingress/e2e-ipp-b-90ceb3-openshift-default-75b485b86d-259zj"
Sep 10 19:35:29.594180 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:29.594140    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/868773b2-7d37-4196-9ab3-2256b295f31d-workload-certs\") pod \"e2e-ipp-b-90ceb3-openshift-default-75b485b86d-259zj\" (UID: \"868773b2-7d37-4196-9ab3-2256b295f31d\") " pod="openshift-ingress/e2e-ipp-b-90ceb3-openshift-default-75b485b86d-259zj"
Sep 10 19:35:29.594331 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:29.594182    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgt72\" (UniqueName: \"kubernetes.io/projected/868773b2-7d37-4196-9ab3-2256b295f31d-kube-api-access-qgt72\") pod \"e2e-ipp-b-90ceb3-openshift-default-75b485b86d-259zj\" (UID: \"868773b2-7d37-4196-9ab3-2256b295f31d\") " pod="openshift-ingress/e2e-ipp-b-90ceb3-openshift-default-75b485b86d-259zj"
Sep 10 19:35:29.594331 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:29.594288    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/868773b2-7d37-4196-9ab3-2256b295f31d-credential-socket\") pod \"e2e-ipp-b-90ceb3-openshift-default-75b485b86d-259zj\" (UID: \"868773b2-7d37-4196-9ab3-2256b295f31d\") " pod="openshift-ingress/e2e-ipp-b-90ceb3-openshift-default-75b485b86d-259zj"
Sep 10 19:35:29.594331 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:29.594313    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/868773b2-7d37-4196-9ab3-2256b295f31d-istio-data\") pod \"e2e-ipp-b-90ceb3-openshift-default-75b485b86d-259zj\" (UID: \"868773b2-7d37-4196-9ab3-2256b295f31d\") " pod="openshift-ingress/e2e-ipp-b-90ceb3-openshift-default-75b485b86d-259zj"
Sep 10 19:35:29.594513 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:29.594429    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/868773b2-7d37-4196-9ab3-2256b295f31d-istiod-ca-cert\") pod \"e2e-ipp-b-90ceb3-openshift-default-75b485b86d-259zj\" (UID: \"868773b2-7d37-4196-9ab3-2256b295f31d\") " pod="openshift-ingress/e2e-ipp-b-90ceb3-openshift-default-75b485b86d-259zj"
Sep 10 19:35:29.596410 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:29.596383    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/868773b2-7d37-4196-9ab3-2256b295f31d-istio-envoy\") pod \"e2e-ipp-b-90ceb3-openshift-default-75b485b86d-259zj\" (UID: \"868773b2-7d37-4196-9ab3-2256b295f31d\") " pod="openshift-ingress/e2e-ipp-b-90ceb3-openshift-default-75b485b86d-259zj"
Sep 10 19:35:29.596710 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:29.596687    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/868773b2-7d37-4196-9ab3-2256b295f31d-istio-podinfo\") pod \"e2e-ipp-b-90ceb3-openshift-default-75b485b86d-259zj\" (UID: \"868773b2-7d37-4196-9ab3-2256b295f31d\") " pod="openshift-ingress/e2e-ipp-b-90ceb3-openshift-default-75b485b86d-259zj"
Sep 10 19:35:29.602134 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:29.602104    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/868773b2-7d37-4196-9ab3-2256b295f31d-istio-token\") pod \"e2e-ipp-b-90ceb3-openshift-default-75b485b86d-259zj\" (UID: \"868773b2-7d37-4196-9ab3-2256b295f31d\") " pod="openshift-ingress/e2e-ipp-b-90ceb3-openshift-default-75b485b86d-259zj"
Sep 10 19:35:29.602414 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:29.602394    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgt72\" (UniqueName: \"kubernetes.io/projected/868773b2-7d37-4196-9ab3-2256b295f31d-kube-api-access-qgt72\") pod \"e2e-ipp-b-90ceb3-openshift-default-75b485b86d-259zj\" (UID: \"868773b2-7d37-4196-9ab3-2256b295f31d\") " pod="openshift-ingress/e2e-ipp-b-90ceb3-openshift-default-75b485b86d-259zj"
Sep 10 19:35:29.766076 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:29.765993    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-90ceb3-openshift-default-75b485b86d-259zj"
Sep 10 19:35:30.042927 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:30.042897    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-90ceb3-openshift-default-75b485b86d-259zj"]
Sep 10 19:35:30.045189 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:35:30.045157    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod868773b2_7d37_4196_9ab3_2256b295f31d.slice/crio-21a5d36fa5189715aa64ac47fc6c44073d41bd1e372b9bef26aada31cc53aeb2 WatchSource:0}: Error finding container 21a5d36fa5189715aa64ac47fc6c44073d41bd1e372b9bef26aada31cc53aeb2: Status 404 returned error can't find the container with id 21a5d36fa5189715aa64ac47fc6c44073d41bd1e372b9bef26aada31cc53aeb2
Sep 10 19:35:30.047440 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:30.047403    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 19:35:30.047553 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:30.047489    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 19:35:30.047553 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:30.047532    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 19:35:30.622071 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:30.622036    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-90ceb3-openshift-default-75b485b86d-259zj" event={"ID":"868773b2-7d37-4196-9ab3-2256b295f31d","Type":"ContainerStarted","Data":"f989e936f5235f846fcf19314b090ceb82f06480b59384c8729e26cad6a4d4d0"}
Sep 10 19:35:30.622547 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:30.622085    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-90ceb3-openshift-default-75b485b86d-259zj" event={"ID":"868773b2-7d37-4196-9ab3-2256b295f31d","Type":"ContainerStarted","Data":"21a5d36fa5189715aa64ac47fc6c44073d41bd1e372b9bef26aada31cc53aeb2"}
Sep 10 19:35:30.645431 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:30.645376    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-90ceb3-openshift-default-75b485b86d-259zj" podStartSLOduration=1.645360954 podStartE2EDuration="1.645360954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:35:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:35:30.643022273 +0000 UTC m=+1493.872072297" watchObservedRunningTime="2026-09-10 19:35:30.645360954 +0000 UTC m=+1493.874410975"
Sep 10 19:35:30.766388 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:30.766332    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-90ceb3-openshift-default-75b485b86d-259zj"
Sep 10 19:35:31.772435 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:31.772404    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-90ceb3-openshift-default-75b485b86d-259zj"
Sep 10 19:35:32.632160 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:32.632127    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-90ceb3-openshift-default-75b485b86d-259zj"
Sep 10 19:35:32.632992 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:32.632969    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-90ceb3-openshift-default-75b485b86d-259zj"
Sep 10 19:35:33.711937 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:33.711896    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-90ceb3-6b7cdf5f78-cdz2n"]
Sep 10 19:35:33.716893 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:33.716868    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-90ceb3-6b7cdf5f78-cdz2n"
Sep 10 19:35:33.719465 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:33.719441    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-90ceb3-serving-cert\""
Sep 10 19:35:33.728063 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:33.728038    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-90ceb3-6b7cdf5f78-cdz2n"]
Sep 10 19:35:33.835540 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:33.835503    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb7a0913-bb02-4165-8cf4-c8a451cf2f6a-maas-api-tls\") pod \"maas-api-e2e-ipp-b-90ceb3-6b7cdf5f78-cdz2n\" (UID: \"bb7a0913-bb02-4165-8cf4-c8a451cf2f6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-90ceb3-6b7cdf5f78-cdz2n"
Sep 10 19:35:33.835718 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:33.835573    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb7a0913-bb02-4165-8cf4-c8a451cf2f6a-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-90ceb3-6b7cdf5f78-cdz2n\" (UID: \"bb7a0913-bb02-4165-8cf4-c8a451cf2f6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-90ceb3-6b7cdf5f78-cdz2n"
Sep 10 19:35:33.835718 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:33.835659    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5slz2\" (UniqueName: \"kubernetes.io/projected/bb7a0913-bb02-4165-8cf4-c8a451cf2f6a-kube-api-access-5slz2\") pod \"maas-api-e2e-ipp-b-90ceb3-6b7cdf5f78-cdz2n\" (UID: \"bb7a0913-bb02-4165-8cf4-c8a451cf2f6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-90ceb3-6b7cdf5f78-cdz2n"
Sep 10 19:35:33.835718 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:33.835706    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb7a0913-bb02-4165-8cf4-c8a451cf2f6a-metrics-certs\") pod \"maas-api-e2e-ipp-b-90ceb3-6b7cdf5f78-cdz2n\" (UID: \"bb7a0913-bb02-4165-8cf4-c8a451cf2f6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-90ceb3-6b7cdf5f78-cdz2n"
Sep 10 19:35:33.936899 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:33.936859    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5slz2\" (UniqueName: \"kubernetes.io/projected/bb7a0913-bb02-4165-8cf4-c8a451cf2f6a-kube-api-access-5slz2\") pod \"maas-api-e2e-ipp-b-90ceb3-6b7cdf5f78-cdz2n\" (UID: \"bb7a0913-bb02-4165-8cf4-c8a451cf2f6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-90ceb3-6b7cdf5f78-cdz2n"
Sep 10 19:35:33.937104 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:33.936919    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb7a0913-bb02-4165-8cf4-c8a451cf2f6a-metrics-certs\") pod \"maas-api-e2e-ipp-b-90ceb3-6b7cdf5f78-cdz2n\" (UID: \"bb7a0913-bb02-4165-8cf4-c8a451cf2f6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-90ceb3-6b7cdf5f78-cdz2n"
Sep 10 19:35:33.937104 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:33.936948    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb7a0913-bb02-4165-8cf4-c8a451cf2f6a-maas-api-tls\") pod \"maas-api-e2e-ipp-b-90ceb3-6b7cdf5f78-cdz2n\" (UID: \"bb7a0913-bb02-4165-8cf4-c8a451cf2f6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-90ceb3-6b7cdf5f78-cdz2n"
Sep 10 19:35:33.937104 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:33.936986    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb7a0913-bb02-4165-8cf4-c8a451cf2f6a-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-90ceb3-6b7cdf5f78-cdz2n\" (UID: \"bb7a0913-bb02-4165-8cf4-c8a451cf2f6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-90ceb3-6b7cdf5f78-cdz2n"
Sep 10 19:35:33.937104 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:35:33.937094    2546 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-90ceb3-serving-cert: secret "maas-api-e2e-ipp-b-90ceb3-serving-cert" not found
Sep 10 19:35:33.937314 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:35:33.937170    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb7a0913-bb02-4165-8cf4-c8a451cf2f6a-maas-api-tls podName:bb7a0913-bb02-4165-8cf4-c8a451cf2f6a nodeName:}" failed. No retries permitted until 2026-09-10 19:35:34.437154869 +0000 UTC m=+1497.666204872 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bb7a0913-bb02-4165-8cf4-c8a451cf2f6a-maas-api-tls") pod "maas-api-e2e-ipp-b-90ceb3-6b7cdf5f78-cdz2n" (UID: "bb7a0913-bb02-4165-8cf4-c8a451cf2f6a") : secret "maas-api-e2e-ipp-b-90ceb3-serving-cert" not found
Sep 10 19:35:33.937767 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:33.937743    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb7a0913-bb02-4165-8cf4-c8a451cf2f6a-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-90ceb3-6b7cdf5f78-cdz2n\" (UID: \"bb7a0913-bb02-4165-8cf4-c8a451cf2f6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-90ceb3-6b7cdf5f78-cdz2n"
Sep 10 19:35:33.939608 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:33.939586    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb7a0913-bb02-4165-8cf4-c8a451cf2f6a-metrics-certs\") pod \"maas-api-e2e-ipp-b-90ceb3-6b7cdf5f78-cdz2n\" (UID: \"bb7a0913-bb02-4165-8cf4-c8a451cf2f6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-90ceb3-6b7cdf5f78-cdz2n"
Sep 10 19:35:33.945773 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:33.945748    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5slz2\" (UniqueName: \"kubernetes.io/projected/bb7a0913-bb02-4165-8cf4-c8a451cf2f6a-kube-api-access-5slz2\") pod \"maas-api-e2e-ipp-b-90ceb3-6b7cdf5f78-cdz2n\" (UID: \"bb7a0913-bb02-4165-8cf4-c8a451cf2f6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-90ceb3-6b7cdf5f78-cdz2n"
Sep 10 19:35:34.442795 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:34.442764    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb7a0913-bb02-4165-8cf4-c8a451cf2f6a-maas-api-tls\") pod \"maas-api-e2e-ipp-b-90ceb3-6b7cdf5f78-cdz2n\" (UID: \"bb7a0913-bb02-4165-8cf4-c8a451cf2f6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-90ceb3-6b7cdf5f78-cdz2n"
Sep 10 19:35:34.445258 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:34.445233    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb7a0913-bb02-4165-8cf4-c8a451cf2f6a-maas-api-tls\") pod \"maas-api-e2e-ipp-b-90ceb3-6b7cdf5f78-cdz2n\" (UID: \"bb7a0913-bb02-4165-8cf4-c8a451cf2f6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-90ceb3-6b7cdf5f78-cdz2n"
Sep 10 19:35:34.630263 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:34.630229    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-90ceb3-6b7cdf5f78-cdz2n"
Sep 10 19:35:34.906081 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:34.906056    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-90ceb3-6b7cdf5f78-cdz2n"]
Sep 10 19:35:34.907899 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:35:34.907873    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb7a0913_bb02_4165_8cf4_c8a451cf2f6a.slice/crio-0049fc2ff94ff1c4307dfd33a2ce189e68e56a07edada34f90431813083e11ca WatchSource:0}: Error finding container 0049fc2ff94ff1c4307dfd33a2ce189e68e56a07edada34f90431813083e11ca: Status 404 returned error can't find the container with id 0049fc2ff94ff1c4307dfd33a2ce189e68e56a07edada34f90431813083e11ca
Sep 10 19:35:34.913109 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:34.913089    2546 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 19:35:35.648549 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:35.648453    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-90ceb3-6b7cdf5f78-cdz2n" event={"ID":"bb7a0913-bb02-4165-8cf4-c8a451cf2f6a","Type":"ContainerStarted","Data":"ec927ed296ed019c98968dde64ae02f3508b6738f40ecbd0b900df2162d4b3ec"}
Sep 10 19:35:35.648549 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:35.648501    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-90ceb3-6b7cdf5f78-cdz2n" event={"ID":"bb7a0913-bb02-4165-8cf4-c8a451cf2f6a","Type":"ContainerStarted","Data":"0049fc2ff94ff1c4307dfd33a2ce189e68e56a07edada34f90431813083e11ca"}
Sep 10 19:35:35.648549 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:35.648528    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-90ceb3-6b7cdf5f78-cdz2n"
Sep 10 19:35:35.695894 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:35.695835    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-90ceb3-6b7cdf5f78-cdz2n" podStartSLOduration=2.244436967 podStartE2EDuration="2.695816738s" totalImagesPullingTime="451.379771ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:35:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:35:35.694777191 +0000 UTC m=+1498.923827221" watchObservedRunningTime="2026-09-10 19:35:35.695816738 +0000 UTC m=+1498.924866762"
Sep 10 19:35:40.520389 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:40.520335    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-88n8f"]
Sep 10 19:35:41.659013 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:35:41.658982    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-90ceb3-6b7cdf5f78-cdz2n"
Sep 10 19:36:03.673547 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:36:03.673504    2546 patch_prober.go:28] interesting pod/e2e-ipp-b-90ceb3-openshift-default-75b485b86d-259zj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 10 19:36:03.673986 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:36:03.673574    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-90ceb3-openshift-default-75b485b86d-259zj" podUID="868773b2-7d37-4196-9ab3-2256b295f31d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 10 19:36:04.799174 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:36:04.799135    2546 generic.go:356] "Generic (PLEG): container finished" podID="868773b2-7d37-4196-9ab3-2256b295f31d" containerID="f989e936f5235f846fcf19314b090ceb82f06480b59384c8729e26cad6a4d4d0" exitCode=0
Sep 10 19:36:04.799614 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:36:04.799207    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-90ceb3-openshift-default-75b485b86d-259zj" event={"ID":"868773b2-7d37-4196-9ab3-2256b295f31d","Type":"ContainerDied","Data":"f989e936f5235f846fcf19314b090ceb82f06480b59384c8729e26cad6a4d4d0"}
Sep 10 19:36:04.799667 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:36:04.799643    2546 scope.go:117] "RemoveContainer" containerID="f989e936f5235f846fcf19314b090ceb82f06480b59384c8729e26cad6a4d4d0"
Sep 10 19:36:04.800733 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:36:04.800701    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 19:36:04.800852 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:36:04.800788    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 19:36:04.800852 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:36:04.800838    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 19:36:05.806624 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:36:05.806588    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-90ceb3-openshift-default-75b485b86d-259zj" event={"ID":"868773b2-7d37-4196-9ab3-2256b295f31d","Type":"ContainerStarted","Data":"a6315b670a1592e8be4098093882226fe62ae5cab1768203b1f099d57792d589"}
Sep 10 19:36:06.766785 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:36:06.766747    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-90ceb3-openshift-default-75b485b86d-259zj"
Sep 10 19:36:06.767210 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:36:06.767113    2546 patch_prober.go:28] interesting pod/e2e-ipp-b-90ceb3-openshift-default-75b485b86d-259zj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Sep 10 19:36:06.767403 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:36:06.767379    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-90ceb3-openshift-default-75b485b86d-259zj" podUID="868773b2-7d37-4196-9ab3-2256b295f31d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Sep 10 19:36:07.767014 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:36:07.766966    2546 patch_prober.go:28] interesting pod/e2e-ipp-b-90ceb3-openshift-default-75b485b86d-259zj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Sep 10 19:36:07.767549 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:36:07.767035    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-90ceb3-openshift-default-75b485b86d-259zj" podUID="868773b2-7d37-4196-9ab3-2256b295f31d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Sep 10 19:36:08.767500 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:36:08.767457    2546 patch_prober.go:28] interesting pod/e2e-ipp-b-90ceb3-openshift-default-75b485b86d-259zj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Sep 10 19:36:08.768104 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:36:08.768069    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-90ceb3-openshift-default-75b485b86d-259zj" podUID="868773b2-7d37-4196-9ab3-2256b295f31d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Sep 10 19:36:09.535588 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:36:09.535543    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-c59b36/ipp-route-c59b36-kserve-84f5ffff95-jwm46"]
Sep 10 19:36:09.543469 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:36:09.543443    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-c59b36/ipp-route-c59b36-kserve-84f5ffff95-jwm46"
Sep 10 19:36:09.546691 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:36:09.546658    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-c59b36\"/\"ipp-route-c59b36-kserve-self-signed-certs\""
Sep 10 19:36:09.547162 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:36:09.547137    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-c59b36\"/\"kube-root-ca.crt\""
Sep 10 19:36:09.547381 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:36:09.547143    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-c59b36\"/\"default-dockercfg-wq74x\""
Sep 10 19:36:09.547507 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:36:09.547178    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-c59b36\"/\"openshift-service-ca.crt\""
Sep 10 19:36:09.548642 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:36:09.548618    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-c59b36/ipp-route-c59b36-kserve-84f5ffff95-jwm46"]
Sep 10 19:36:09.590957 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:36:09.590909    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70b83199-6656-4f76-b242-e8508d5dbd20-tmp-dir\") pod \"ipp-route-c59b36-kserve-84f5ffff95-jwm46\" (UID: \"70b83199-6656-4f76-b242-e8508d5dbd20\") " pod="ai-tenant-e2e-ipp-a-c59b36/ipp-route-c59b36-kserve-84f5ffff95-jwm46"
Sep 10 19:36:09.590957 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:36:09.590952    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70b83199-6656-4f76-b242-e8508d5dbd20-model-cache\") pod \"ipp-route-c59b36-kserve-84f5ffff95-jwm46\" (UID: \"70b83199-6656-4f76-b242-e8508d5dbd20\") " pod="ai-tenant-e2e-ipp-a-c59b36/ipp-route-c59b36-kserve-84f5ffff95-jwm46"
Sep 10 19:36:09.591217 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:36:09.590995    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70b83199-6656-4f76-b242-e8508d5dbd20-dshm\") pod \"ipp-route-c59b36-kserve-84f5ffff95-jwm46\" (UID: \"70b83199-6656-4f76-b242-e8508d5dbd20\") " pod="ai-tenant-e2e-ipp-a-c59b36/ipp-route-c59b36-kserve-84f5ffff95-jwm46"
Sep 10 19:36:09.591217 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:36:09.591023    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70b83199-6656-4f76-b242-e8508d5dbd20-tls-certs\") pod \"ipp-route-c59b36-kserve-84f5ffff95-jwm46\" (UID: \"70b83199-6656-4f76-b242-e8508d5dbd20\") " pod="ai-tenant-e2e-ipp-a-c59b36/ipp-route-c59b36-kserve-84f5ffff95-jwm46"
Sep 10 19:36:09.591217 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:36:09.591090    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pf96\" (UniqueName: \"kubernetes.io/projected/70b83199-6656-4f76-b242-e8508d5dbd20-kube-api-access-2pf96\") pod \"ipp-route-c59b36-kserve-84f5ffff95-jwm46\" (UID: \"70b83199-6656-4f76-b242-e8508d5dbd20\") " pod="ai-tenant-e2e-ipp-a-c59b36/ipp-route-c59b36-kserve-84f5ffff95-jwm46"
Sep 10 19:36:09.591217 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:36:09.591133    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70b83199-6656-4f76-b242-e8508d5dbd20-home\") pod \"ipp-route-c59b36-kserve-84f5ffff95-jwm46\" (UID: \"70b83199-6656-4f76-b242-e8508d5dbd20\") " pod="ai-tenant-e2e-ipp-a-c59b36/ipp-route-c59b36-kserve-84f5ffff95-jwm46"
Sep 10 19:36:09.692076 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:36:09.692038    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70b83199-6656-4f76-b242-e8508d5dbd20-dshm\") pod \"ipp-route-c59b36-kserve-84f5ffff95-jwm46\" (UID: \"70b83199-6656-4f76-b242-e8508d5dbd20\") " pod="ai-tenant-e2e-ipp-a-c59b36/ipp-route-c59b36-kserve-84f5ffff95-jwm46"
Sep 10 19:36:09.692076 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:36:09.692086    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70b83199-6656-4f76-b242-e8508d5dbd20-tls-certs\") pod \"ipp-route-c59b36-kserve-84f5ffff95-jwm46\" (UID: \"70b83199-6656-4f76-b242-e8508d5dbd20\") " pod="ai-tenant-e2e-ipp-a-c59b36/ipp-route-c59b36-kserve-84f5ffff95-jwm46"
Sep 10 19:36:09.692319 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:36:09.692133    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pf96\" (UniqueName: \"kubernetes.io/projected/70b83199-6656-4f76-b242-e8508d5dbd20-kube-api-access-2pf96\") pod \"ipp-route-c59b36-kserve-84f5ffff95-jwm46\" (UID: \"70b83199-6656-4f76-b242-e8508d5dbd20\") " pod="ai-tenant-e2e-ipp-a-c59b36/ipp-route-c59b36-kserve-84f5ffff95-jwm46"
Sep 10 19:36:09.692319 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:36:09.692155    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70b83199-6656-4f76-b242-e8508d5dbd20-home\") pod \"ipp-route-c59b36-kserve-84f5ffff95-jwm46\" (UID: \"70b83199-6656-4f76-b242-e8508d5dbd20\") " pod="ai-tenant-e2e-ipp-a-c59b36/ipp-route-c59b36-kserve-84f5ffff95-jwm46"
Sep 10 19:36:09.692319 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:36:09.692191    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70b83199-6656-4f76-b242-e8508d5dbd20-tmp-dir\") pod \"ipp-route-c59b36-kserve-84f5ffff95-jwm46\" (UID: \"70b83199-6656-4f76-b242-e8508d5dbd20\") " pod="ai-tenant-e2e-ipp-a-c59b36/ipp-route-c59b36-kserve-84f5ffff95-jwm46"
Sep 10 19:36:09.692319 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:36:09.692217    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70b83199-6656-4f76-b242-e8508d5dbd20-model-cache\") pod \"ipp-route-c59b36-kserve-84f5ffff95-jwm46\" (UID: \"70b83199-6656-4f76-b242-e8508d5dbd20\") " pod="ai-tenant-e2e-ipp-a-c59b36/ipp-route-c59b36-kserve-84f5ffff95-jwm46"
Sep 10 19:36:09.692602 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:36:09.692578    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70b83199-6656-4f76-b242-e8508d5dbd20-home\") pod \"ipp-route-c59b36-kserve-84f5ffff95-jwm46\" (UID: \"70b83199-6656-4f76-b242-e8508d5dbd20\") " pod="ai-tenant-e2e-ipp-a-c59b36/ipp-route-c59b36-kserve-84f5ffff95-jwm46"
Sep 10 19:36:09.692767 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:36:09.692620    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70b83199-6656-4f76-b242-e8508d5dbd20-model-cache\") pod \"ipp-route-c59b36-kserve-84f5ffff95-jwm46\" (UID: \"70b83199-6656-4f76-b242-e8508d5dbd20\") " pod="ai-tenant-e2e-ipp-a-c59b36/ipp-route-c59b36-kserve-84f5ffff95-jwm46"
Sep 10 19:36:09.692767 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:36:09.692640    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70b83199-6656-4f76-b242-e8508d5dbd20-tmp-dir\") pod \"ipp-route-c59b36-kserve-84f5ffff95-jwm46\" (UID: \"70b83199-6656-4f76-b242-e8508d5dbd20\") " pod="ai-tenant-e2e-ipp-a-c59b36/ipp-route-c59b36-kserve-84f5ffff95-jwm46"
Sep 10 19:36:09.694572 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:36:09.694550    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70b83199-6656-4f76-b242-e8508d5dbd20-dshm\") pod \"ipp-route-c59b36-kserve-84f5ffff95-jwm46\" (UID: \"70b83199-6656-4f76-b242-e8508d5dbd20\") " pod="ai-tenant-e2e-ipp-a-c59b36/ipp-route-c59b36-kserve-84f5ffff95-jwm46"
Sep 10 19:36:09.694817 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:36:09.694800    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70b83199-6656-4f76-b242-e8508d5dbd20-tls-certs\") pod \"ipp-route-c59b36-kserve-84f5ffff95-jwm46\" (UID: \"70b83199-6656-4f76-b242-e8508d5dbd20\") " pod="ai-tenant-e2e-ipp-a-c59b36/ipp-route-c59b36-kserve-84f5ffff95-jwm46"
Sep 10 19:36:09.702028 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:36:09.701982    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pf96\" (UniqueName: \"kubernetes.io/projected/70b83199-6656-4f76-b242-e8508d5dbd20-kube-api-access-2pf96\") pod \"ipp-route-c59b36-kserve-84f5ffff95-jwm46\" (UID: \"70b83199-6656-4f76-b242-e8508d5dbd20\") " pod="ai-tenant-e2e-ipp-a-c59b36/ipp-route-c59b36-kserve-84f5ffff95-jwm46"
Sep 10 19:36:09.766521 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:36:09.766479    2546 patch_prober.go:28] interesting pod/e2e-ipp-b-90ceb3-openshift-default-75b485b86d-259zj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Sep 10 19:36:09.766822 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:36:09.766793    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-90ceb3-openshift-default-75b485b86d-259zj" podUID="868773b2-7d37-4196-9ab3-2256b295f31d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Sep 10 19:36:09.855518 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:36:09.855418    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-c59b36/ipp-route-c59b36-kserve-84f5ffff95-jwm46"
Sep 10 19:36:10.145180 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:36:10.145148    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-c59b36/ipp-route-c59b36-kserve-84f5ffff95-jwm46"]
Sep 10 19:36:10.147504 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:36:10.147467    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod70b83199_6656_4f76_b242_e8508d5dbd20.slice/crio-ce87710c04c12a368dfdb997563a623339910bb4ed22f6bd87fc640cc419c5fa WatchSource:0}: Error finding container ce87710c04c12a368dfdb997563a623339910bb4ed22f6bd87fc640cc419c5fa: Status 404 returned error can't find the container with id ce87710c04c12a368dfdb997563a623339910bb4ed22f6bd87fc640cc419c5fa
Sep 10 19:36:10.767198 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:36:10.767155    2546 patch_prober.go:28] interesting pod/e2e-ipp-b-90ceb3-openshift-default-75b485b86d-259zj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Sep 10 19:36:10.767418 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:36:10.767210    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-90ceb3-openshift-default-75b485b86d-259zj" podUID="868773b2-7d37-4196-9ab3-2256b295f31d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Sep 10 19:36:10.832676 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:36:10.832639    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-c59b36/ipp-route-c59b36-kserve-84f5ffff95-jwm46" event={"ID":"70b83199-6656-4f76-b242-e8508d5dbd20","Type":"ContainerStarted","Data":"3998b38bf7ff675c6f99eeeb98af179084f6040374a53af09467f4f85ae26c77"}
Sep 10 19:36:10.832676 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:36:10.832681    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-c59b36/ipp-route-c59b36-kserve-84f5ffff95-jwm46" event={"ID":"70b83199-6656-4f76-b242-e8508d5dbd20","Type":"ContainerStarted","Data":"ce87710c04c12a368dfdb997563a623339910bb4ed22f6bd87fc640cc419c5fa"}
Sep 10 19:36:10.851030 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:36:10.850975    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-c59b36/ipp-route-c59b36-kserve-84f5ffff95-jwm46" podStartSLOduration=1.850957925 podStartE2EDuration="1.850957925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:36:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:36:10.85031685 +0000 UTC m=+1534.079366900" watchObservedRunningTime="2026-09-10 19:36:10.850957925 +0000 UTC m=+1534.080007949"
Sep 10 19:36:11.770327 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:36:11.770293    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-90ceb3-openshift-default-75b485b86d-259zj"
Sep 10 19:36:11.770814 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:36:11.770779    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-90ceb3-openshift-default-75b485b86d-259zj"
Sep 10 19:36:11.771602 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:36:11.771578    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-90ceb3-openshift-default-75b485b86d-259zj"
Sep 10 19:36:12.723742 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:36:12.723705    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-88n8f"]
Sep 10 19:36:19.856540 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:36:19.856483    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-c59b36/ipp-route-c59b36-kserve-84f5ffff95-jwm46"
Sep 10 19:36:19.856959 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:36:19.856560    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-c59b36/ipp-route-c59b36-kserve-84f5ffff95-jwm46"
Sep 10 19:36:19.869369 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:36:19.869325    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-c59b36/ipp-route-c59b36-kserve-84f5ffff95-jwm46"
Sep 10 19:36:19.891649 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:36:19.891613    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-c59b36/ipp-route-c59b36-kserve-84f5ffff95-jwm46"
Sep 10 19:36:26.921221 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:36:26.921190    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-88n8f"]
Sep 10 19:36:37.437407 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:36:37.437378    2546 scope.go:117] "RemoveContainer" containerID="e58ab58ee4f59578b1fec31a6b4ef55bdb5aa59f6040079c8dd5fe70b1451759"
Sep 10 19:36:46.422299 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:36:46.422262    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-88n8f"]
Sep 10 19:37:12.826072 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:37:12.826032    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-88n8f"]
Sep 10 19:37:32.118417 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:37:32.118312    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-88n8f"]
Sep 10 19:37:40.060355 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:37:40.060312    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-c59b36/ipp-route-c59b36-kserve-84f5ffff95-jwm46"]
Sep 10 19:37:45.463697 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:37:45.463661    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-88n8f"]
Sep 10 19:37:55.080365 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:37:55.080308    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-c59b36/ipp-route-c59b36-kserve-84f5ffff95-jwm46" podUID="70b83199-6656-4f76-b242-e8508d5dbd20" containerName="main" containerID="cri-o://3998b38bf7ff675c6f99eeeb98af179084f6040374a53af09467f4f85ae26c77" gracePeriod=45
Sep 10 19:37:55.328615 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:37:55.328588    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-c59b36/ipp-route-c59b36-kserve-84f5ffff95-jwm46"
Sep 10 19:37:55.369804 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:37:55.369764    2546 generic.go:356] "Generic (PLEG): container finished" podID="70b83199-6656-4f76-b242-e8508d5dbd20" containerID="3998b38bf7ff675c6f99eeeb98af179084f6040374a53af09467f4f85ae26c77" exitCode=0
Sep 10 19:37:55.369977 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:37:55.369817    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-c59b36/ipp-route-c59b36-kserve-84f5ffff95-jwm46"
Sep 10 19:37:55.369977 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:37:55.369835    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-c59b36/ipp-route-c59b36-kserve-84f5ffff95-jwm46" event={"ID":"70b83199-6656-4f76-b242-e8508d5dbd20","Type":"ContainerDied","Data":"3998b38bf7ff675c6f99eeeb98af179084f6040374a53af09467f4f85ae26c77"}
Sep 10 19:37:55.369977 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:37:55.369875    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-c59b36/ipp-route-c59b36-kserve-84f5ffff95-jwm46" event={"ID":"70b83199-6656-4f76-b242-e8508d5dbd20","Type":"ContainerDied","Data":"ce87710c04c12a368dfdb997563a623339910bb4ed22f6bd87fc640cc419c5fa"}
Sep 10 19:37:55.369977 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:37:55.369898    2546 scope.go:117] "RemoveContainer" containerID="3998b38bf7ff675c6f99eeeb98af179084f6040374a53af09467f4f85ae26c77"
Sep 10 19:37:55.379425 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:37:55.379406    2546 scope.go:117] "RemoveContainer" containerID="3998b38bf7ff675c6f99eeeb98af179084f6040374a53af09467f4f85ae26c77"
Sep 10 19:37:55.379742 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:37:55.379710    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3998b38bf7ff675c6f99eeeb98af179084f6040374a53af09467f4f85ae26c77\": container with ID starting with 3998b38bf7ff675c6f99eeeb98af179084f6040374a53af09467f4f85ae26c77 not found: ID does not exist" containerID="3998b38bf7ff675c6f99eeeb98af179084f6040374a53af09467f4f85ae26c77"
Sep 10 19:37:55.379799 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:37:55.379752    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3998b38bf7ff675c6f99eeeb98af179084f6040374a53af09467f4f85ae26c77"} err="failed to get container status \"3998b38bf7ff675c6f99eeeb98af179084f6040374a53af09467f4f85ae26c77\": rpc error: code = NotFound desc = could not find container \"3998b38bf7ff675c6f99eeeb98af179084f6040374a53af09467f4f85ae26c77\": container with ID starting with 3998b38bf7ff675c6f99eeeb98af179084f6040374a53af09467f4f85ae26c77 not found: ID does not exist"
Sep 10 19:37:55.506721 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:37:55.506676    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70b83199-6656-4f76-b242-e8508d5dbd20-home\") pod \"70b83199-6656-4f76-b242-e8508d5dbd20\" (UID: \"70b83199-6656-4f76-b242-e8508d5dbd20\") "
Sep 10 19:37:55.506924 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:37:55.506733    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70b83199-6656-4f76-b242-e8508d5dbd20-tls-certs\") pod \"70b83199-6656-4f76-b242-e8508d5dbd20\" (UID: \"70b83199-6656-4f76-b242-e8508d5dbd20\") "
Sep 10 19:37:55.506924 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:37:55.506775    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70b83199-6656-4f76-b242-e8508d5dbd20-dshm\") pod \"70b83199-6656-4f76-b242-e8508d5dbd20\" (UID: \"70b83199-6656-4f76-b242-e8508d5dbd20\") "
Sep 10 19:37:55.506924 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:37:55.506798    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70b83199-6656-4f76-b242-e8508d5dbd20-model-cache\") pod \"70b83199-6656-4f76-b242-e8508d5dbd20\" (UID: \"70b83199-6656-4f76-b242-e8508d5dbd20\") "
Sep 10 19:37:55.507119 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:37:55.506968    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70b83199-6656-4f76-b242-e8508d5dbd20-home" (OuterVolumeSpecName: "home") pod "70b83199-6656-4f76-b242-e8508d5dbd20" (UID: "70b83199-6656-4f76-b242-e8508d5dbd20"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:37:55.507119 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:37:55.507040    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70b83199-6656-4f76-b242-e8508d5dbd20-tmp-dir\") pod \"70b83199-6656-4f76-b242-e8508d5dbd20\" (UID: \"70b83199-6656-4f76-b242-e8508d5dbd20\") "
Sep 10 19:37:55.507119 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:37:55.507088    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70b83199-6656-4f76-b242-e8508d5dbd20-model-cache" (OuterVolumeSpecName: "model-cache") pod "70b83199-6656-4f76-b242-e8508d5dbd20" (UID: "70b83199-6656-4f76-b242-e8508d5dbd20"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:37:55.507241 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:37:55.507155    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2pf96\" (UniqueName: \"kubernetes.io/projected/70b83199-6656-4f76-b242-e8508d5dbd20-kube-api-access-2pf96\") pod \"70b83199-6656-4f76-b242-e8508d5dbd20\" (UID: \"70b83199-6656-4f76-b242-e8508d5dbd20\") "
Sep 10 19:37:55.507280 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:37:55.507258    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70b83199-6656-4f76-b242-e8508d5dbd20-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "70b83199-6656-4f76-b242-e8508d5dbd20" (UID: "70b83199-6656-4f76-b242-e8508d5dbd20"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:37:55.507483 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:37:55.507466    2546 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70b83199-6656-4f76-b242-e8508d5dbd20-tmp-dir\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:37:55.507582 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:37:55.507485    2546 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70b83199-6656-4f76-b242-e8508d5dbd20-home\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:37:55.507582 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:37:55.507494    2546 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70b83199-6656-4f76-b242-e8508d5dbd20-model-cache\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:37:55.509120 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:37:55.509096    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70b83199-6656-4f76-b242-e8508d5dbd20-dshm" (OuterVolumeSpecName: "dshm") pod "70b83199-6656-4f76-b242-e8508d5dbd20" (UID: "70b83199-6656-4f76-b242-e8508d5dbd20"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:37:55.509120 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:37:55.509107    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70b83199-6656-4f76-b242-e8508d5dbd20-kube-api-access-2pf96" (OuterVolumeSpecName: "kube-api-access-2pf96") pod "70b83199-6656-4f76-b242-e8508d5dbd20" (UID: "70b83199-6656-4f76-b242-e8508d5dbd20"). InnerVolumeSpecName "kube-api-access-2pf96". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 19:37:55.509272 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:37:55.509140    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70b83199-6656-4f76-b242-e8508d5dbd20-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "70b83199-6656-4f76-b242-e8508d5dbd20" (UID: "70b83199-6656-4f76-b242-e8508d5dbd20"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 19:37:55.608367 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:37:55.608293    2546 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70b83199-6656-4f76-b242-e8508d5dbd20-tls-certs\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:37:55.608367 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:37:55.608329    2546 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70b83199-6656-4f76-b242-e8508d5dbd20-dshm\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:37:55.608367 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:37:55.608369    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pf96\" (UniqueName: \"kubernetes.io/projected/70b83199-6656-4f76-b242-e8508d5dbd20-kube-api-access-2pf96\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:37:55.693818 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:37:55.693781    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-c59b36/ipp-route-c59b36-kserve-84f5ffff95-jwm46"]
Sep 10 19:37:55.695819 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:37:55.695792    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-c59b36/ipp-route-c59b36-kserve-84f5ffff95-jwm46"]
Sep 10 19:37:57.281122 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:37:57.281083    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70b83199-6656-4f76-b242-e8508d5dbd20" path="/var/lib/kubelet/pods/70b83199-6656-4f76-b242-e8508d5dbd20/volumes"
Sep 10 19:38:23.438422 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:23.438387    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-c59b36-openshift-default-6bd9548cf8-vwr6f"]
Sep 10 19:38:23.438951 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:23.438618    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-c59b36-openshift-default-6bd9548cf8-vwr6f" podUID="47463639-8fdd-4475-83a1-862ac426ef5f" containerName="istio-proxy" containerID="cri-o://3a8e284bacb6d603f61104de0e3790c46365e8507eb4907b14ab9f4380e36233" gracePeriod=30
Sep 10 19:38:24.898469 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:24.898419    2546 patch_prober.go:28] interesting pod/e2e-ipp-a-c59b36-openshift-default-6bd9548cf8-vwr6f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 10 19:38:24.898859 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:24.898511    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-c59b36-openshift-default-6bd9548cf8-vwr6f" podUID="47463639-8fdd-4475-83a1-862ac426ef5f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 10 19:38:28.695131 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:28.695103    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c59b36-openshift-default-6bd9548cf8-vwr6f"
Sep 10 19:38:28.801828 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:28.801735    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47463639-8fdd-4475-83a1-862ac426ef5f-workload-socket\") pod \"47463639-8fdd-4475-83a1-862ac426ef5f\" (UID: \"47463639-8fdd-4475-83a1-862ac426ef5f\") "
Sep 10 19:38:28.801828 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:28.801797    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47463639-8fdd-4475-83a1-862ac426ef5f-workload-certs\") pod \"47463639-8fdd-4475-83a1-862ac426ef5f\" (UID: \"47463639-8fdd-4475-83a1-862ac426ef5f\") "
Sep 10 19:38:28.801828 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:28.801826    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47463639-8fdd-4475-83a1-862ac426ef5f-istio-token\") pod \"47463639-8fdd-4475-83a1-862ac426ef5f\" (UID: \"47463639-8fdd-4475-83a1-862ac426ef5f\") "
Sep 10 19:38:28.802121 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:28.801854    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zsw6m\" (UniqueName: \"kubernetes.io/projected/47463639-8fdd-4475-83a1-862ac426ef5f-kube-api-access-zsw6m\") pod \"47463639-8fdd-4475-83a1-862ac426ef5f\" (UID: \"47463639-8fdd-4475-83a1-862ac426ef5f\") "
Sep 10 19:38:28.802121 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:28.801937    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47463639-8fdd-4475-83a1-862ac426ef5f-credential-socket\") pod \"47463639-8fdd-4475-83a1-862ac426ef5f\" (UID: \"47463639-8fdd-4475-83a1-862ac426ef5f\") "
Sep 10 19:38:28.802121 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:28.801966    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47463639-8fdd-4475-83a1-862ac426ef5f-istio-data\") pod \"47463639-8fdd-4475-83a1-862ac426ef5f\" (UID: \"47463639-8fdd-4475-83a1-862ac426ef5f\") "
Sep 10 19:38:28.802121 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:28.802009    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47463639-8fdd-4475-83a1-862ac426ef5f-istio-podinfo\") pod \"47463639-8fdd-4475-83a1-862ac426ef5f\" (UID: \"47463639-8fdd-4475-83a1-862ac426ef5f\") "
Sep 10 19:38:28.802121 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:28.802060    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47463639-8fdd-4475-83a1-862ac426ef5f-istiod-ca-cert\") pod \"47463639-8fdd-4475-83a1-862ac426ef5f\" (UID: \"47463639-8fdd-4475-83a1-862ac426ef5f\") "
Sep 10 19:38:28.802121 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:28.802088    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47463639-8fdd-4475-83a1-862ac426ef5f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "47463639-8fdd-4475-83a1-862ac426ef5f" (UID: "47463639-8fdd-4475-83a1-862ac426ef5f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:38:28.802121 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:28.802103    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47463639-8fdd-4475-83a1-862ac426ef5f-istio-envoy\") pod \"47463639-8fdd-4475-83a1-862ac426ef5f\" (UID: \"47463639-8fdd-4475-83a1-862ac426ef5f\") "
Sep 10 19:38:28.802545 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:28.802188    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47463639-8fdd-4475-83a1-862ac426ef5f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "47463639-8fdd-4475-83a1-862ac426ef5f" (UID: "47463639-8fdd-4475-83a1-862ac426ef5f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:38:28.802545 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:28.802401    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47463639-8fdd-4475-83a1-862ac426ef5f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "47463639-8fdd-4475-83a1-862ac426ef5f" (UID: "47463639-8fdd-4475-83a1-862ac426ef5f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:38:28.802653 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:28.802544    2546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47463639-8fdd-4475-83a1-862ac426ef5f-workload-socket\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:38:28.802653 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:28.802566    2546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47463639-8fdd-4475-83a1-862ac426ef5f-workload-certs\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:38:28.802653 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:28.802581    2546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47463639-8fdd-4475-83a1-862ac426ef5f-credential-socket\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:38:28.802653 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:28.802613    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47463639-8fdd-4475-83a1-862ac426ef5f-istio-data" (OuterVolumeSpecName: "istio-data") pod "47463639-8fdd-4475-83a1-862ac426ef5f" (UID: "47463639-8fdd-4475-83a1-862ac426ef5f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:38:28.802856 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:28.802708    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47463639-8fdd-4475-83a1-862ac426ef5f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "47463639-8fdd-4475-83a1-862ac426ef5f" (UID: "47463639-8fdd-4475-83a1-862ac426ef5f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 19:38:28.804276 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:28.804239    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47463639-8fdd-4475-83a1-862ac426ef5f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "47463639-8fdd-4475-83a1-862ac426ef5f" (UID: "47463639-8fdd-4475-83a1-862ac426ef5f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:38:28.804414 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:28.804364    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47463639-8fdd-4475-83a1-862ac426ef5f-kube-api-access-zsw6m" (OuterVolumeSpecName: "kube-api-access-zsw6m") pod "47463639-8fdd-4475-83a1-862ac426ef5f" (UID: "47463639-8fdd-4475-83a1-862ac426ef5f"). InnerVolumeSpecName "kube-api-access-zsw6m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 19:38:28.804577 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:28.804561    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47463639-8fdd-4475-83a1-862ac426ef5f-istio-token" (OuterVolumeSpecName: "istio-token") pod "47463639-8fdd-4475-83a1-862ac426ef5f" (UID: "47463639-8fdd-4475-83a1-862ac426ef5f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 19:38:28.804644 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:28.804575    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/47463639-8fdd-4475-83a1-862ac426ef5f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "47463639-8fdd-4475-83a1-862ac426ef5f" (UID: "47463639-8fdd-4475-83a1-862ac426ef5f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 19:38:28.904041 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:28.904003    2546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47463639-8fdd-4475-83a1-862ac426ef5f-istio-data\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:38:28.904041 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:28.904035    2546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47463639-8fdd-4475-83a1-862ac426ef5f-istio-podinfo\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:38:28.904041 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:28.904045    2546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47463639-8fdd-4475-83a1-862ac426ef5f-istiod-ca-cert\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:38:28.904041 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:28.904054    2546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47463639-8fdd-4475-83a1-862ac426ef5f-istio-envoy\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:38:28.904319 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:28.904062    2546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47463639-8fdd-4475-83a1-862ac426ef5f-istio-token\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:38:28.904319 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:28.904072    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsw6m\" (UniqueName: \"kubernetes.io/projected/47463639-8fdd-4475-83a1-862ac426ef5f-kube-api-access-zsw6m\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:38:29.536442 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:29.536409    2546 generic.go:356] "Generic (PLEG): container finished" podID="47463639-8fdd-4475-83a1-862ac426ef5f" containerID="3a8e284bacb6d603f61104de0e3790c46365e8507eb4907b14ab9f4380e36233" exitCode=0
Sep 10 19:38:29.536642 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:29.536464    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c59b36-openshift-default-6bd9548cf8-vwr6f"
Sep 10 19:38:29.536642 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:29.536488    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c59b36-openshift-default-6bd9548cf8-vwr6f" event={"ID":"47463639-8fdd-4475-83a1-862ac426ef5f","Type":"ContainerDied","Data":"3a8e284bacb6d603f61104de0e3790c46365e8507eb4907b14ab9f4380e36233"}
Sep 10 19:38:29.536642 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:29.536544    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c59b36-openshift-default-6bd9548cf8-vwr6f" event={"ID":"47463639-8fdd-4475-83a1-862ac426ef5f","Type":"ContainerDied","Data":"6a21e814f53909b534e0dc8b616a72e81e6014fab184a446839d1dd596aa8dd1"}
Sep 10 19:38:29.536642 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:29.536559    2546 scope.go:117] "RemoveContainer" containerID="3a8e284bacb6d603f61104de0e3790c46365e8507eb4907b14ab9f4380e36233"
Sep 10 19:38:29.545140 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:29.545122    2546 scope.go:117] "RemoveContainer" containerID="d6988a8f4b4854c9022ba8e260af0324c1514f29d0f4d2fd4fa91b6d409d2885"
Sep 10 19:38:29.552988 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:29.552972    2546 scope.go:117] "RemoveContainer" containerID="3a8e284bacb6d603f61104de0e3790c46365e8507eb4907b14ab9f4380e36233"
Sep 10 19:38:29.553243 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:38:29.553224    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a8e284bacb6d603f61104de0e3790c46365e8507eb4907b14ab9f4380e36233\": container with ID starting with 3a8e284bacb6d603f61104de0e3790c46365e8507eb4907b14ab9f4380e36233 not found: ID does not exist" containerID="3a8e284bacb6d603f61104de0e3790c46365e8507eb4907b14ab9f4380e36233"
Sep 10 19:38:29.553290 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:29.553256    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a8e284bacb6d603f61104de0e3790c46365e8507eb4907b14ab9f4380e36233"} err="failed to get container status \"3a8e284bacb6d603f61104de0e3790c46365e8507eb4907b14ab9f4380e36233\": rpc error: code = NotFound desc = could not find container \"3a8e284bacb6d603f61104de0e3790c46365e8507eb4907b14ab9f4380e36233\": container with ID starting with 3a8e284bacb6d603f61104de0e3790c46365e8507eb4907b14ab9f4380e36233 not found: ID does not exist"
Sep 10 19:38:29.553290 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:29.553273    2546 scope.go:117] "RemoveContainer" containerID="d6988a8f4b4854c9022ba8e260af0324c1514f29d0f4d2fd4fa91b6d409d2885"
Sep 10 19:38:29.553552 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:38:29.553533    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6988a8f4b4854c9022ba8e260af0324c1514f29d0f4d2fd4fa91b6d409d2885\": container with ID starting with d6988a8f4b4854c9022ba8e260af0324c1514f29d0f4d2fd4fa91b6d409d2885 not found: ID does not exist" containerID="d6988a8f4b4854c9022ba8e260af0324c1514f29d0f4d2fd4fa91b6d409d2885"
Sep 10 19:38:29.553599 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:29.553558    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6988a8f4b4854c9022ba8e260af0324c1514f29d0f4d2fd4fa91b6d409d2885"} err="failed to get container status \"d6988a8f4b4854c9022ba8e260af0324c1514f29d0f4d2fd4fa91b6d409d2885\": rpc error: code = NotFound desc = could not find container \"d6988a8f4b4854c9022ba8e260af0324c1514f29d0f4d2fd4fa91b6d409d2885\": container with ID starting with d6988a8f4b4854c9022ba8e260af0324c1514f29d0f4d2fd4fa91b6d409d2885 not found: ID does not exist"
Sep 10 19:38:29.560282 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:29.560259    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-c59b36-openshift-default-6bd9548cf8-vwr6f"]
Sep 10 19:38:29.567440 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:29.567419    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-c59b36-openshift-default-6bd9548cf8-vwr6f"]
Sep 10 19:38:31.280758 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:31.280723    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47463639-8fdd-4475-83a1-862ac426ef5f" path="/var/lib/kubelet/pods/47463639-8fdd-4475-83a1-862ac426ef5f/volumes"
Sep 10 19:38:34.261105 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:34.261070    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-90ceb3-6b7cdf5f78-cdz2n"]
Sep 10 19:38:34.261516 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:34.261306    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-90ceb3-6b7cdf5f78-cdz2n" podUID="bb7a0913-bb02-4165-8cf4-c8a451cf2f6a" containerName="maas-api" containerID="cri-o://ec927ed296ed019c98968dde64ae02f3508b6738f40ecbd0b900df2162d4b3ec" gracePeriod=30
Sep 10 19:38:34.511670 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:34.511603    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-90ceb3-6b7cdf5f78-cdz2n"
Sep 10 19:38:34.564807 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:34.564766    2546 generic.go:356] "Generic (PLEG): container finished" podID="bb7a0913-bb02-4165-8cf4-c8a451cf2f6a" containerID="ec927ed296ed019c98968dde64ae02f3508b6738f40ecbd0b900df2162d4b3ec" exitCode=0
Sep 10 19:38:34.564807 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:34.564809    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-90ceb3-6b7cdf5f78-cdz2n"
Sep 10 19:38:34.565058 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:34.564852    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-90ceb3-6b7cdf5f78-cdz2n" event={"ID":"bb7a0913-bb02-4165-8cf4-c8a451cf2f6a","Type":"ContainerDied","Data":"ec927ed296ed019c98968dde64ae02f3508b6738f40ecbd0b900df2162d4b3ec"}
Sep 10 19:38:34.565058 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:34.564908    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-90ceb3-6b7cdf5f78-cdz2n" event={"ID":"bb7a0913-bb02-4165-8cf4-c8a451cf2f6a","Type":"ContainerDied","Data":"0049fc2ff94ff1c4307dfd33a2ce189e68e56a07edada34f90431813083e11ca"}
Sep 10 19:38:34.565058 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:34.564941    2546 scope.go:117] "RemoveContainer" containerID="ec927ed296ed019c98968dde64ae02f3508b6738f40ecbd0b900df2162d4b3ec"
Sep 10 19:38:34.573648 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:34.573629    2546 scope.go:117] "RemoveContainer" containerID="ec927ed296ed019c98968dde64ae02f3508b6738f40ecbd0b900df2162d4b3ec"
Sep 10 19:38:34.573921 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:38:34.573902    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec927ed296ed019c98968dde64ae02f3508b6738f40ecbd0b900df2162d4b3ec\": container with ID starting with ec927ed296ed019c98968dde64ae02f3508b6738f40ecbd0b900df2162d4b3ec not found: ID does not exist" containerID="ec927ed296ed019c98968dde64ae02f3508b6738f40ecbd0b900df2162d4b3ec"
Sep 10 19:38:34.573983 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:34.573929    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec927ed296ed019c98968dde64ae02f3508b6738f40ecbd0b900df2162d4b3ec"} err="failed to get container status \"ec927ed296ed019c98968dde64ae02f3508b6738f40ecbd0b900df2162d4b3ec\": rpc error: code = NotFound desc = could not find container \"ec927ed296ed019c98968dde64ae02f3508b6738f40ecbd0b900df2162d4b3ec\": container with ID starting with ec927ed296ed019c98968dde64ae02f3508b6738f40ecbd0b900df2162d4b3ec not found: ID does not exist"
Sep 10 19:38:34.662711 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:34.662670    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb7a0913-bb02-4165-8cf4-c8a451cf2f6a-openshift-service-ca\") pod \"bb7a0913-bb02-4165-8cf4-c8a451cf2f6a\" (UID: \"bb7a0913-bb02-4165-8cf4-c8a451cf2f6a\") "
Sep 10 19:38:34.662928 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:34.662812    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb7a0913-bb02-4165-8cf4-c8a451cf2f6a-maas-api-tls\") pod \"bb7a0913-bb02-4165-8cf4-c8a451cf2f6a\" (UID: \"bb7a0913-bb02-4165-8cf4-c8a451cf2f6a\") "
Sep 10 19:38:34.662928 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:34.662872    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb7a0913-bb02-4165-8cf4-c8a451cf2f6a-metrics-certs\") pod \"bb7a0913-bb02-4165-8cf4-c8a451cf2f6a\" (UID: \"bb7a0913-bb02-4165-8cf4-c8a451cf2f6a\") "
Sep 10 19:38:34.663048 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:34.662976    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5slz2\" (UniqueName: \"kubernetes.io/projected/bb7a0913-bb02-4165-8cf4-c8a451cf2f6a-kube-api-access-5slz2\") pod \"bb7a0913-bb02-4165-8cf4-c8a451cf2f6a\" (UID: \"bb7a0913-bb02-4165-8cf4-c8a451cf2f6a\") "
Sep 10 19:38:34.663712 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:34.663654    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb7a0913-bb02-4165-8cf4-c8a451cf2f6a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "bb7a0913-bb02-4165-8cf4-c8a451cf2f6a" (UID: "bb7a0913-bb02-4165-8cf4-c8a451cf2f6a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 19:38:34.666282 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:34.666242    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb7a0913-bb02-4165-8cf4-c8a451cf2f6a-kube-api-access-5slz2" (OuterVolumeSpecName: "kube-api-access-5slz2") pod "bb7a0913-bb02-4165-8cf4-c8a451cf2f6a" (UID: "bb7a0913-bb02-4165-8cf4-c8a451cf2f6a"). InnerVolumeSpecName "kube-api-access-5slz2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 19:38:34.666683 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:34.666651    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb7a0913-bb02-4165-8cf4-c8a451cf2f6a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "bb7a0913-bb02-4165-8cf4-c8a451cf2f6a" (UID: "bb7a0913-bb02-4165-8cf4-c8a451cf2f6a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 19:38:34.666683 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:34.666675    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb7a0913-bb02-4165-8cf4-c8a451cf2f6a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "bb7a0913-bb02-4165-8cf4-c8a451cf2f6a" (UID: "bb7a0913-bb02-4165-8cf4-c8a451cf2f6a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 19:38:34.764835 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:34.764753    2546 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb7a0913-bb02-4165-8cf4-c8a451cf2f6a-metrics-certs\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:38:34.764835 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:34.764783    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5slz2\" (UniqueName: \"kubernetes.io/projected/bb7a0913-bb02-4165-8cf4-c8a451cf2f6a-kube-api-access-5slz2\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:38:34.764835 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:34.764793    2546 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb7a0913-bb02-4165-8cf4-c8a451cf2f6a-openshift-service-ca\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:38:34.764835 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:34.764801    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb7a0913-bb02-4165-8cf4-c8a451cf2f6a-maas-api-tls\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:38:34.887256 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:34.887224    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-90ceb3-6b7cdf5f78-cdz2n"]
Sep 10 19:38:34.891233 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:34.891207    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-90ceb3-6b7cdf5f78-cdz2n"]
Sep 10 19:38:35.282051 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:35.282016    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb7a0913-bb02-4165-8cf4-c8a451cf2f6a" path="/var/lib/kubelet/pods/bb7a0913-bb02-4165-8cf4-c8a451cf2f6a/volumes"
Sep 10 19:38:37.824797 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:37.822166    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-88n8f"]
Sep 10 19:38:41.178559 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:41.178526    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-90ceb3-openshift-default-75b485b86d-259zj"]
Sep 10 19:38:41.179068 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:41.178743    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-90ceb3-openshift-default-75b485b86d-259zj" podUID="868773b2-7d37-4196-9ab3-2256b295f31d" containerName="istio-proxy" containerID="cri-o://a6315b670a1592e8be4098093882226fe62ae5cab1768203b1f099d57792d589" gracePeriod=30
Sep 10 19:38:41.773320 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:41.773284    2546 patch_prober.go:28] interesting pod/e2e-ipp-b-90ceb3-openshift-default-75b485b86d-259zj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 10 19:38:41.773511 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:41.773383    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-90ceb3-openshift-default-75b485b86d-259zj" podUID="868773b2-7d37-4196-9ab3-2256b295f31d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 10 19:38:46.431931 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:46.431901    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-90ceb3-openshift-default-75b485b86d-259zj"
Sep 10 19:38:46.478578 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:46.478482    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/868773b2-7d37-4196-9ab3-2256b295f31d-workload-certs\") pod \"868773b2-7d37-4196-9ab3-2256b295f31d\" (UID: \"868773b2-7d37-4196-9ab3-2256b295f31d\") "
Sep 10 19:38:46.478578 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:46.478520    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/868773b2-7d37-4196-9ab3-2256b295f31d-credential-socket\") pod \"868773b2-7d37-4196-9ab3-2256b295f31d\" (UID: \"868773b2-7d37-4196-9ab3-2256b295f31d\") "
Sep 10 19:38:46.478578 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:46.478543    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qgt72\" (UniqueName: \"kubernetes.io/projected/868773b2-7d37-4196-9ab3-2256b295f31d-kube-api-access-qgt72\") pod \"868773b2-7d37-4196-9ab3-2256b295f31d\" (UID: \"868773b2-7d37-4196-9ab3-2256b295f31d\") "
Sep 10 19:38:46.478578 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:46.478576    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/868773b2-7d37-4196-9ab3-2256b295f31d-istio-envoy\") pod \"868773b2-7d37-4196-9ab3-2256b295f31d\" (UID: \"868773b2-7d37-4196-9ab3-2256b295f31d\") "
Sep 10 19:38:46.478936 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:46.478597    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/868773b2-7d37-4196-9ab3-2256b295f31d-istio-data\") pod \"868773b2-7d37-4196-9ab3-2256b295f31d\" (UID: \"868773b2-7d37-4196-9ab3-2256b295f31d\") "
Sep 10 19:38:46.478936 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:46.478627    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/868773b2-7d37-4196-9ab3-2256b295f31d-istio-podinfo\") pod \"868773b2-7d37-4196-9ab3-2256b295f31d\" (UID: \"868773b2-7d37-4196-9ab3-2256b295f31d\") "
Sep 10 19:38:46.478936 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:46.478737    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/868773b2-7d37-4196-9ab3-2256b295f31d-istio-token\") pod \"868773b2-7d37-4196-9ab3-2256b295f31d\" (UID: \"868773b2-7d37-4196-9ab3-2256b295f31d\") "
Sep 10 19:38:46.478936 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:46.478810    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/868773b2-7d37-4196-9ab3-2256b295f31d-workload-socket\") pod \"868773b2-7d37-4196-9ab3-2256b295f31d\" (UID: \"868773b2-7d37-4196-9ab3-2256b295f31d\") "
Sep 10 19:38:46.478936 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:46.478792    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/868773b2-7d37-4196-9ab3-2256b295f31d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "868773b2-7d37-4196-9ab3-2256b295f31d" (UID: "868773b2-7d37-4196-9ab3-2256b295f31d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:38:46.478936 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:46.478856    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/868773b2-7d37-4196-9ab3-2256b295f31d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "868773b2-7d37-4196-9ab3-2256b295f31d" (UID: "868773b2-7d37-4196-9ab3-2256b295f31d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:38:46.478936 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:46.478875    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/868773b2-7d37-4196-9ab3-2256b295f31d-istiod-ca-cert\") pod \"868773b2-7d37-4196-9ab3-2256b295f31d\" (UID: \"868773b2-7d37-4196-9ab3-2256b295f31d\") "
Sep 10 19:38:46.479314 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:46.478957    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/868773b2-7d37-4196-9ab3-2256b295f31d-istio-data" (OuterVolumeSpecName: "istio-data") pod "868773b2-7d37-4196-9ab3-2256b295f31d" (UID: "868773b2-7d37-4196-9ab3-2256b295f31d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:38:46.479314 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:46.479165    2546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/868773b2-7d37-4196-9ab3-2256b295f31d-workload-certs\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:38:46.479314 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:46.479188    2546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/868773b2-7d37-4196-9ab3-2256b295f31d-credential-socket\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:38:46.479314 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:46.479205    2546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/868773b2-7d37-4196-9ab3-2256b295f31d-istio-data\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:38:46.479314 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:46.479279    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/868773b2-7d37-4196-9ab3-2256b295f31d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "868773b2-7d37-4196-9ab3-2256b295f31d" (UID: "868773b2-7d37-4196-9ab3-2256b295f31d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:38:46.479314 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:46.479308    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/868773b2-7d37-4196-9ab3-2256b295f31d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "868773b2-7d37-4196-9ab3-2256b295f31d" (UID: "868773b2-7d37-4196-9ab3-2256b295f31d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 19:38:46.481036 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:46.481010    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/868773b2-7d37-4196-9ab3-2256b295f31d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "868773b2-7d37-4196-9ab3-2256b295f31d" (UID: "868773b2-7d37-4196-9ab3-2256b295f31d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 19:38:46.481158 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:46.481008    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/868773b2-7d37-4196-9ab3-2256b295f31d-istio-token" (OuterVolumeSpecName: "istio-token") pod "868773b2-7d37-4196-9ab3-2256b295f31d" (UID: "868773b2-7d37-4196-9ab3-2256b295f31d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 19:38:46.481235 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:46.481213    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/868773b2-7d37-4196-9ab3-2256b295f31d-kube-api-access-qgt72" (OuterVolumeSpecName: "kube-api-access-qgt72") pod "868773b2-7d37-4196-9ab3-2256b295f31d" (UID: "868773b2-7d37-4196-9ab3-2256b295f31d"). InnerVolumeSpecName "kube-api-access-qgt72". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 19:38:46.481371 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:46.481335    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/868773b2-7d37-4196-9ab3-2256b295f31d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "868773b2-7d37-4196-9ab3-2256b295f31d" (UID: "868773b2-7d37-4196-9ab3-2256b295f31d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:38:46.580505 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:46.580473    2546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/868773b2-7d37-4196-9ab3-2256b295f31d-workload-socket\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:38:46.580505 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:46.580500    2546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/868773b2-7d37-4196-9ab3-2256b295f31d-istiod-ca-cert\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:38:46.580505 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:46.580509    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgt72\" (UniqueName: \"kubernetes.io/projected/868773b2-7d37-4196-9ab3-2256b295f31d-kube-api-access-qgt72\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:38:46.580505 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:46.580518    2546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/868773b2-7d37-4196-9ab3-2256b295f31d-istio-envoy\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:38:46.580772 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:46.580526    2546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/868773b2-7d37-4196-9ab3-2256b295f31d-istio-podinfo\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:38:46.580772 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:46.580533    2546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/868773b2-7d37-4196-9ab3-2256b295f31d-istio-token\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:38:46.623822 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:46.623787    2546 generic.go:356] "Generic (PLEG): container finished" podID="868773b2-7d37-4196-9ab3-2256b295f31d" containerID="a6315b670a1592e8be4098093882226fe62ae5cab1768203b1f099d57792d589" exitCode=0
Sep 10 19:38:46.623991 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:46.623839    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-90ceb3-openshift-default-75b485b86d-259zj"
Sep 10 19:38:46.623991 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:46.623872    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-90ceb3-openshift-default-75b485b86d-259zj" event={"ID":"868773b2-7d37-4196-9ab3-2256b295f31d","Type":"ContainerDied","Data":"a6315b670a1592e8be4098093882226fe62ae5cab1768203b1f099d57792d589"}
Sep 10 19:38:46.623991 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:46.623935    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-90ceb3-openshift-default-75b485b86d-259zj" event={"ID":"868773b2-7d37-4196-9ab3-2256b295f31d","Type":"ContainerDied","Data":"21a5d36fa5189715aa64ac47fc6c44073d41bd1e372b9bef26aada31cc53aeb2"}
Sep 10 19:38:46.623991 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:46.623958    2546 scope.go:117] "RemoveContainer" containerID="a6315b670a1592e8be4098093882226fe62ae5cab1768203b1f099d57792d589"
Sep 10 19:38:46.633968 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:46.633947    2546 scope.go:117] "RemoveContainer" containerID="f989e936f5235f846fcf19314b090ceb82f06480b59384c8729e26cad6a4d4d0"
Sep 10 19:38:46.642584 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:46.642562    2546 scope.go:117] "RemoveContainer" containerID="a6315b670a1592e8be4098093882226fe62ae5cab1768203b1f099d57792d589"
Sep 10 19:38:46.642889 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:38:46.642869    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6315b670a1592e8be4098093882226fe62ae5cab1768203b1f099d57792d589\": container with ID starting with a6315b670a1592e8be4098093882226fe62ae5cab1768203b1f099d57792d589 not found: ID does not exist" containerID="a6315b670a1592e8be4098093882226fe62ae5cab1768203b1f099d57792d589"
Sep 10 19:38:46.642953 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:46.642900    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6315b670a1592e8be4098093882226fe62ae5cab1768203b1f099d57792d589"} err="failed to get container status \"a6315b670a1592e8be4098093882226fe62ae5cab1768203b1f099d57792d589\": rpc error: code = NotFound desc = could not find container \"a6315b670a1592e8be4098093882226fe62ae5cab1768203b1f099d57792d589\": container with ID starting with a6315b670a1592e8be4098093882226fe62ae5cab1768203b1f099d57792d589 not found: ID does not exist"
Sep 10 19:38:46.642953 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:46.642918    2546 scope.go:117] "RemoveContainer" containerID="f989e936f5235f846fcf19314b090ceb82f06480b59384c8729e26cad6a4d4d0"
Sep 10 19:38:46.643193 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:38:46.643171    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f989e936f5235f846fcf19314b090ceb82f06480b59384c8729e26cad6a4d4d0\": container with ID starting with f989e936f5235f846fcf19314b090ceb82f06480b59384c8729e26cad6a4d4d0 not found: ID does not exist" containerID="f989e936f5235f846fcf19314b090ceb82f06480b59384c8729e26cad6a4d4d0"
Sep 10 19:38:46.643239 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:46.643203    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f989e936f5235f846fcf19314b090ceb82f06480b59384c8729e26cad6a4d4d0"} err="failed to get container status \"f989e936f5235f846fcf19314b090ceb82f06480b59384c8729e26cad6a4d4d0\": rpc error: code = NotFound desc = could not find container \"f989e936f5235f846fcf19314b090ceb82f06480b59384c8729e26cad6a4d4d0\": container with ID starting with f989e936f5235f846fcf19314b090ceb82f06480b59384c8729e26cad6a4d4d0 not found: ID does not exist"
Sep 10 19:38:46.649676 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:46.649652    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-90ceb3-openshift-default-75b485b86d-259zj"]
Sep 10 19:38:46.654593 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:46.654568    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-90ceb3-openshift-default-75b485b86d-259zj"]
Sep 10 19:38:47.280874 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:47.280828    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="868773b2-7d37-4196-9ab3-2256b295f31d" path="/var/lib/kubelet/pods/868773b2-7d37-4196-9ab3-2256b295f31d/volumes"
Sep 10 19:38:52.250703 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:52.250663    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b64b1c-549df87f58-fz2jg"]
Sep 10 19:38:52.251166 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:52.250980    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b64b1c-549df87f58-fz2jg" podUID="e9f3bf25-92a3-411e-8b4c-10681ba8611b" containerName="maas-api" containerID="cri-o://4a92492bd8809bc5884f2fa27f68c7507e19156e5cd5bb9e835eb39ce9a26231" gracePeriod=30
Sep 10 19:38:52.504678 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:52.504616    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b64b1c-549df87f58-fz2jg"
Sep 10 19:38:52.533594 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:52.533560    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9f3bf25-92a3-411e-8b4c-10681ba8611b-maas-api-tls\") pod \"e9f3bf25-92a3-411e-8b4c-10681ba8611b\" (UID: \"e9f3bf25-92a3-411e-8b4c-10681ba8611b\") "
Sep 10 19:38:52.533779 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:52.533629    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9f3bf25-92a3-411e-8b4c-10681ba8611b-openshift-service-ca\") pod \"e9f3bf25-92a3-411e-8b4c-10681ba8611b\" (UID: \"e9f3bf25-92a3-411e-8b4c-10681ba8611b\") "
Sep 10 19:38:52.533779 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:52.533649    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9f3bf25-92a3-411e-8b4c-10681ba8611b-metrics-certs\") pod \"e9f3bf25-92a3-411e-8b4c-10681ba8611b\" (UID: \"e9f3bf25-92a3-411e-8b4c-10681ba8611b\") "
Sep 10 19:38:52.533779 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:52.533681    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bw8qx\" (UniqueName: \"kubernetes.io/projected/e9f3bf25-92a3-411e-8b4c-10681ba8611b-kube-api-access-bw8qx\") pod \"e9f3bf25-92a3-411e-8b4c-10681ba8611b\" (UID: \"e9f3bf25-92a3-411e-8b4c-10681ba8611b\") "
Sep 10 19:38:52.534060 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:52.534036    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9f3bf25-92a3-411e-8b4c-10681ba8611b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e9f3bf25-92a3-411e-8b4c-10681ba8611b" (UID: "e9f3bf25-92a3-411e-8b4c-10681ba8611b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 19:38:52.535856 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:52.535830    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9f3bf25-92a3-411e-8b4c-10681ba8611b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e9f3bf25-92a3-411e-8b4c-10681ba8611b" (UID: "e9f3bf25-92a3-411e-8b4c-10681ba8611b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 19:38:52.535856 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:52.535837    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9f3bf25-92a3-411e-8b4c-10681ba8611b-kube-api-access-bw8qx" (OuterVolumeSpecName: "kube-api-access-bw8qx") pod "e9f3bf25-92a3-411e-8b4c-10681ba8611b" (UID: "e9f3bf25-92a3-411e-8b4c-10681ba8611b"). InnerVolumeSpecName "kube-api-access-bw8qx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 19:38:52.536013 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:52.535882    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9f3bf25-92a3-411e-8b4c-10681ba8611b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e9f3bf25-92a3-411e-8b4c-10681ba8611b" (UID: "e9f3bf25-92a3-411e-8b4c-10681ba8611b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 19:38:52.634562 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:52.634523    2546 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9f3bf25-92a3-411e-8b4c-10681ba8611b-openshift-service-ca\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:38:52.634562 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:52.634555    2546 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e9f3bf25-92a3-411e-8b4c-10681ba8611b-metrics-certs\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:38:52.634562 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:52.634564    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bw8qx\" (UniqueName: \"kubernetes.io/projected/e9f3bf25-92a3-411e-8b4c-10681ba8611b-kube-api-access-bw8qx\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:38:52.634562 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:52.634573    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9f3bf25-92a3-411e-8b4c-10681ba8611b-maas-api-tls\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:38:52.655401 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:52.655285    2546 generic.go:356] "Generic (PLEG): container finished" podID="e9f3bf25-92a3-411e-8b4c-10681ba8611b" containerID="4a92492bd8809bc5884f2fa27f68c7507e19156e5cd5bb9e835eb39ce9a26231" exitCode=0
Sep 10 19:38:52.655709 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:52.655460    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b64b1c-549df87f58-fz2jg" event={"ID":"e9f3bf25-92a3-411e-8b4c-10681ba8611b","Type":"ContainerDied","Data":"4a92492bd8809bc5884f2fa27f68c7507e19156e5cd5bb9e835eb39ce9a26231"}
Sep 10 19:38:52.655709 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:52.655515    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b64b1c-549df87f58-fz2jg" event={"ID":"e9f3bf25-92a3-411e-8b4c-10681ba8611b","Type":"ContainerDied","Data":"8ad2f4a99c7b15d44a12b550ca9b5d05e301359c1a7632cf4178a75a428b4848"}
Sep 10 19:38:52.655709 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:52.655546    2546 scope.go:117] "RemoveContainer" containerID="4a92492bd8809bc5884f2fa27f68c7507e19156e5cd5bb9e835eb39ce9a26231"
Sep 10 19:38:52.655709 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:52.655684    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b64b1c-549df87f58-fz2jg"
Sep 10 19:38:52.666695 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:52.666670    2546 scope.go:117] "RemoveContainer" containerID="4a92492bd8809bc5884f2fa27f68c7507e19156e5cd5bb9e835eb39ce9a26231"
Sep 10 19:38:52.666991 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:38:52.666959    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a92492bd8809bc5884f2fa27f68c7507e19156e5cd5bb9e835eb39ce9a26231\": container with ID starting with 4a92492bd8809bc5884f2fa27f68c7507e19156e5cd5bb9e835eb39ce9a26231 not found: ID does not exist" containerID="4a92492bd8809bc5884f2fa27f68c7507e19156e5cd5bb9e835eb39ce9a26231"
Sep 10 19:38:52.667049 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:52.667003    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a92492bd8809bc5884f2fa27f68c7507e19156e5cd5bb9e835eb39ce9a26231"} err="failed to get container status \"4a92492bd8809bc5884f2fa27f68c7507e19156e5cd5bb9e835eb39ce9a26231\": rpc error: code = NotFound desc = could not find container \"4a92492bd8809bc5884f2fa27f68c7507e19156e5cd5bb9e835eb39ce9a26231\": container with ID starting with 4a92492bd8809bc5884f2fa27f68c7507e19156e5cd5bb9e835eb39ce9a26231 not found: ID does not exist"
Sep 10 19:38:52.680287 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:52.680256    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b64b1c-549df87f58-fz2jg"]
Sep 10 19:38:52.687525 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:52.687500    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b64b1c-549df87f58-fz2jg"]
Sep 10 19:38:53.280668 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:53.280626    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9f3bf25-92a3-411e-8b4c-10681ba8611b" path="/var/lib/kubelet/pods/e9f3bf25-92a3-411e-8b4c-10681ba8611b/volumes"
Sep 10 19:38:55.325821 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:38:55.325786    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-88n8f"]
Sep 10 19:39:10.699145 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:39:10.699057    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f15c09-29817810-7f8p2"]
Sep 10 19:39:10.702027 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:39:10.701992    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f15c09-29817810-7f8p2"]
Sep 10 19:39:11.281859 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:39:11.281825    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf87606e-cc4f-4fe3-a023-0cfa667edee6" path="/var/lib/kubelet/pods/bf87606e-cc4f-4fe3-a023-0cfa667edee6/volumes"
Sep 10 19:39:13.513066 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:39:13.513029    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-88n8f"]
Sep 10 19:39:17.699588 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:39:17.699552    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-f15c09-openshift-default-7b7497d674-g7xs6"]
Sep 10 19:39:17.700077 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:39:17.699765    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-f15c09-openshift-default-7b7497d674-g7xs6" podUID="e2f23b51-72c3-45c6-82cf-da36af76ec82" containerName="istio-proxy" containerID="cri-o://4fcec4413d8e8cac002177160fc5d6bf867fc242d6f5610c70c593220e2f311f" gracePeriod=30
Sep 10 19:39:21.511174 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:39:21.511136    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-88n8f"]
Sep 10 19:39:22.945928 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:39:22.945904    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-f15c09-openshift-default-7b7497d674-g7xs6"
Sep 10 19:39:23.110751 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:39:23.110723    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nctdh\" (UniqueName: \"kubernetes.io/projected/e2f23b51-72c3-45c6-82cf-da36af76ec82-kube-api-access-nctdh\") pod \"e2f23b51-72c3-45c6-82cf-da36af76ec82\" (UID: \"e2f23b51-72c3-45c6-82cf-da36af76ec82\") "
Sep 10 19:39:23.110942 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:39:23.110768    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2f23b51-72c3-45c6-82cf-da36af76ec82-istio-data\") pod \"e2f23b51-72c3-45c6-82cf-da36af76ec82\" (UID: \"e2f23b51-72c3-45c6-82cf-da36af76ec82\") "
Sep 10 19:39:23.110942 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:39:23.110787    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2f23b51-72c3-45c6-82cf-da36af76ec82-istio-token\") pod \"e2f23b51-72c3-45c6-82cf-da36af76ec82\" (UID: \"e2f23b51-72c3-45c6-82cf-da36af76ec82\") "
Sep 10 19:39:23.110942 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:39:23.110810    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2f23b51-72c3-45c6-82cf-da36af76ec82-istiod-ca-cert\") pod \"e2f23b51-72c3-45c6-82cf-da36af76ec82\" (UID: \"e2f23b51-72c3-45c6-82cf-da36af76ec82\") "
Sep 10 19:39:23.110942 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:39:23.110839    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2f23b51-72c3-45c6-82cf-da36af76ec82-istio-envoy\") pod \"e2f23b51-72c3-45c6-82cf-da36af76ec82\" (UID: \"e2f23b51-72c3-45c6-82cf-da36af76ec82\") "
Sep 10 19:39:23.110942 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:39:23.110888    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2f23b51-72c3-45c6-82cf-da36af76ec82-credential-socket\") pod \"e2f23b51-72c3-45c6-82cf-da36af76ec82\" (UID: \"e2f23b51-72c3-45c6-82cf-da36af76ec82\") "
Sep 10 19:39:23.110942 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:39:23.110931    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2f23b51-72c3-45c6-82cf-da36af76ec82-workload-socket\") pod \"e2f23b51-72c3-45c6-82cf-da36af76ec82\" (UID: \"e2f23b51-72c3-45c6-82cf-da36af76ec82\") "
Sep 10 19:39:23.111269 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:39:23.110957    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2f23b51-72c3-45c6-82cf-da36af76ec82-workload-certs\") pod \"e2f23b51-72c3-45c6-82cf-da36af76ec82\" (UID: \"e2f23b51-72c3-45c6-82cf-da36af76ec82\") "
Sep 10 19:39:23.111269 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:39:23.110988    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2f23b51-72c3-45c6-82cf-da36af76ec82-istio-podinfo\") pod \"e2f23b51-72c3-45c6-82cf-da36af76ec82\" (UID: \"e2f23b51-72c3-45c6-82cf-da36af76ec82\") "
Sep 10 19:39:23.111269 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:39:23.111035    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2f23b51-72c3-45c6-82cf-da36af76ec82-istio-data" (OuterVolumeSpecName: "istio-data") pod "e2f23b51-72c3-45c6-82cf-da36af76ec82" (UID: "e2f23b51-72c3-45c6-82cf-da36af76ec82"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:39:23.111506 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:39:23.111258    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2f23b51-72c3-45c6-82cf-da36af76ec82-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e2f23b51-72c3-45c6-82cf-da36af76ec82" (UID: "e2f23b51-72c3-45c6-82cf-da36af76ec82"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:39:23.111506 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:39:23.111270    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2f23b51-72c3-45c6-82cf-da36af76ec82-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e2f23b51-72c3-45c6-82cf-da36af76ec82" (UID: "e2f23b51-72c3-45c6-82cf-da36af76ec82"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:39:23.111506 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:39:23.111323    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2f23b51-72c3-45c6-82cf-da36af76ec82-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e2f23b51-72c3-45c6-82cf-da36af76ec82" (UID: "e2f23b51-72c3-45c6-82cf-da36af76ec82"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 19:39:23.111506 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:39:23.111364    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2f23b51-72c3-45c6-82cf-da36af76ec82-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e2f23b51-72c3-45c6-82cf-da36af76ec82" (UID: "e2f23b51-72c3-45c6-82cf-da36af76ec82"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:39:23.111506 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:39:23.111405    2546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2f23b51-72c3-45c6-82cf-da36af76ec82-istio-data\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:39:23.113076 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:39:23.113042    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2f23b51-72c3-45c6-82cf-da36af76ec82-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e2f23b51-72c3-45c6-82cf-da36af76ec82" (UID: "e2f23b51-72c3-45c6-82cf-da36af76ec82"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:39:23.113247 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:39:23.113228    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2f23b51-72c3-45c6-82cf-da36af76ec82-kube-api-access-nctdh" (OuterVolumeSpecName: "kube-api-access-nctdh") pod "e2f23b51-72c3-45c6-82cf-da36af76ec82" (UID: "e2f23b51-72c3-45c6-82cf-da36af76ec82"). InnerVolumeSpecName "kube-api-access-nctdh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 19:39:23.113590 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:39:23.113564    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e2f23b51-72c3-45c6-82cf-da36af76ec82-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e2f23b51-72c3-45c6-82cf-da36af76ec82" (UID: "e2f23b51-72c3-45c6-82cf-da36af76ec82"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 19:39:23.113681 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:39:23.113571    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2f23b51-72c3-45c6-82cf-da36af76ec82-istio-token" (OuterVolumeSpecName: "istio-token") pod "e2f23b51-72c3-45c6-82cf-da36af76ec82" (UID: "e2f23b51-72c3-45c6-82cf-da36af76ec82"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 19:39:23.211893 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:39:23.211855    2546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2f23b51-72c3-45c6-82cf-da36af76ec82-istio-token\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:39:23.211893 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:39:23.211886    2546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2f23b51-72c3-45c6-82cf-da36af76ec82-istiod-ca-cert\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:39:23.211893 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:39:23.211894    2546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2f23b51-72c3-45c6-82cf-da36af76ec82-istio-envoy\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:39:23.211893 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:39:23.211902    2546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2f23b51-72c3-45c6-82cf-da36af76ec82-credential-socket\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:39:23.212159 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:39:23.211912    2546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2f23b51-72c3-45c6-82cf-da36af76ec82-workload-socket\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:39:23.212159 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:39:23.211920    2546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2f23b51-72c3-45c6-82cf-da36af76ec82-workload-certs\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:39:23.212159 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:39:23.211930    2546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2f23b51-72c3-45c6-82cf-da36af76ec82-istio-podinfo\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:39:23.212159 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:39:23.211938    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nctdh\" (UniqueName: \"kubernetes.io/projected/e2f23b51-72c3-45c6-82cf-da36af76ec82-kube-api-access-nctdh\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:39:23.807725 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:39:23.807691    2546 generic.go:356] "Generic (PLEG): container finished" podID="e2f23b51-72c3-45c6-82cf-da36af76ec82" containerID="4fcec4413d8e8cac002177160fc5d6bf867fc242d6f5610c70c593220e2f311f" exitCode=0
Sep 10 19:39:23.807903 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:39:23.807735    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-f15c09-openshift-default-7b7497d674-g7xs6"
Sep 10 19:39:23.807903 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:39:23.807780    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-f15c09-openshift-default-7b7497d674-g7xs6" event={"ID":"e2f23b51-72c3-45c6-82cf-da36af76ec82","Type":"ContainerDied","Data":"4fcec4413d8e8cac002177160fc5d6bf867fc242d6f5610c70c593220e2f311f"}
Sep 10 19:39:23.807903 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:39:23.807837    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-f15c09-openshift-default-7b7497d674-g7xs6" event={"ID":"e2f23b51-72c3-45c6-82cf-da36af76ec82","Type":"ContainerDied","Data":"3095a190d4f9ca1d9ee4ba4bac3199fce5b8e39d8f68cba88a8d1c4749ac424f"}
Sep 10 19:39:23.807903 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:39:23.807860    2546 scope.go:117] "RemoveContainer" containerID="4fcec4413d8e8cac002177160fc5d6bf867fc242d6f5610c70c593220e2f311f"
Sep 10 19:39:23.816458 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:39:23.816424    2546 scope.go:117] "RemoveContainer" containerID="da3562f518237999c0b345e73729ebb1f0b8302e1e166f90b4d4383e1a29e5b3"
Sep 10 19:39:23.824623 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:39:23.824529    2546 scope.go:117] "RemoveContainer" containerID="4fcec4413d8e8cac002177160fc5d6bf867fc242d6f5610c70c593220e2f311f"
Sep 10 19:39:23.829763 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:39:23.824967    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fcec4413d8e8cac002177160fc5d6bf867fc242d6f5610c70c593220e2f311f\": container with ID starting with 4fcec4413d8e8cac002177160fc5d6bf867fc242d6f5610c70c593220e2f311f not found: ID does not exist" containerID="4fcec4413d8e8cac002177160fc5d6bf867fc242d6f5610c70c593220e2f311f"
Sep 10 19:39:23.829763 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:39:23.825007    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fcec4413d8e8cac002177160fc5d6bf867fc242d6f5610c70c593220e2f311f"} err="failed to get container status \"4fcec4413d8e8cac002177160fc5d6bf867fc242d6f5610c70c593220e2f311f\": rpc error: code = NotFound desc = could not find container \"4fcec4413d8e8cac002177160fc5d6bf867fc242d6f5610c70c593220e2f311f\": container with ID starting with 4fcec4413d8e8cac002177160fc5d6bf867fc242d6f5610c70c593220e2f311f not found: ID does not exist"
Sep 10 19:39:23.829763 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:39:23.825033    2546 scope.go:117] "RemoveContainer" containerID="da3562f518237999c0b345e73729ebb1f0b8302e1e166f90b4d4383e1a29e5b3"
Sep 10 19:39:23.829763 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:39:23.828799    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-f15c09-openshift-default-7b7497d674-g7xs6"]
Sep 10 19:39:23.832044 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:39:23.832010    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da3562f518237999c0b345e73729ebb1f0b8302e1e166f90b4d4383e1a29e5b3\": container with ID starting with da3562f518237999c0b345e73729ebb1f0b8302e1e166f90b4d4383e1a29e5b3 not found: ID does not exist" containerID="da3562f518237999c0b345e73729ebb1f0b8302e1e166f90b4d4383e1a29e5b3"
Sep 10 19:39:23.832171 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:39:23.832040    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-f15c09-openshift-default-7b7497d674-g7xs6"]
Sep 10 19:39:23.832171 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:39:23.832054    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da3562f518237999c0b345e73729ebb1f0b8302e1e166f90b4d4383e1a29e5b3"} err="failed to get container status \"da3562f518237999c0b345e73729ebb1f0b8302e1e166f90b4d4383e1a29e5b3\": rpc error: code = NotFound desc = could not find container \"da3562f518237999c0b345e73729ebb1f0b8302e1e166f90b4d4383e1a29e5b3\": container with ID starting with da3562f518237999c0b345e73729ebb1f0b8302e1e166f90b4d4383e1a29e5b3 not found: ID does not exist"
Sep 10 19:39:25.280873 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:39:25.280840    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2f23b51-72c3-45c6-82cf-da36af76ec82" path="/var/lib/kubelet/pods/e2f23b51-72c3-45c6-82cf-da36af76ec82/volumes"
Sep 10 19:39:37.485779 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:39:37.485746    2546 scope.go:117] "RemoveContainer" containerID="18e98e09473fe27910dc0922629ef152a9146e6d743c60a08434698da03e2283"
Sep 10 19:40:44.721252 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:40:44.721209    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-88n8f"]
Sep 10 19:41:10.419865 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:41:10.419821    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-88n8f"]
Sep 10 19:41:19.629737 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:41:19.629705    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-88n8f"]
Sep 10 19:41:26.123776 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:41:26.123735    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-88n8f"]
Sep 10 19:41:32.431184 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:41:32.431141    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-88n8f"]
Sep 10 19:41:48.728845 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:41:48.728807    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-88n8f"]
Sep 10 19:41:57.820294 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:41:57.820255    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-88n8f"]
Sep 10 19:42:03.643713 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:42:03.643622    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-wpm7h"]
Sep 10 19:42:03.644384 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:42:03.643863    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-wpm7h" podUID="69c8f195-819a-4f81-916f-21461bef0ad6" containerName="manager" containerID="cri-o://e123fae4c4770984c0cb9c4b6ee1fb74ce929da86d26df883d2d98b00650bef4" gracePeriod=10
Sep 10 19:42:04.544804 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:42:04.544774    2546 generic.go:356] "Generic (PLEG): container finished" podID="69c8f195-819a-4f81-916f-21461bef0ad6" containerID="e123fae4c4770984c0cb9c4b6ee1fb74ce929da86d26df883d2d98b00650bef4" exitCode=0
Sep 10 19:42:04.544963 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:42:04.544846    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-wpm7h" event={"ID":"69c8f195-819a-4f81-916f-21461bef0ad6","Type":"ContainerDied","Data":"e123fae4c4770984c0cb9c4b6ee1fb74ce929da86d26df883d2d98b00650bef4"}
Sep 10 19:42:04.590253 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:42:04.590229    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-wpm7h"
Sep 10 19:42:04.652270 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:42:04.652165    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ktckh\" (UniqueName: \"kubernetes.io/projected/69c8f195-819a-4f81-916f-21461bef0ad6-kube-api-access-ktckh\") pod \"69c8f195-819a-4f81-916f-21461bef0ad6\" (UID: \"69c8f195-819a-4f81-916f-21461bef0ad6\") "
Sep 10 19:42:04.652270 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:42:04.652209    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/69c8f195-819a-4f81-916f-21461bef0ad6-extensions-socket-volume\") pod \"69c8f195-819a-4f81-916f-21461bef0ad6\" (UID: \"69c8f195-819a-4f81-916f-21461bef0ad6\") "
Sep 10 19:42:04.652892 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:42:04.652585    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69c8f195-819a-4f81-916f-21461bef0ad6-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "69c8f195-819a-4f81-916f-21461bef0ad6" (UID: "69c8f195-819a-4f81-916f-21461bef0ad6"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:42:04.654273 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:42:04.654238    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69c8f195-819a-4f81-916f-21461bef0ad6-kube-api-access-ktckh" (OuterVolumeSpecName: "kube-api-access-ktckh") pod "69c8f195-819a-4f81-916f-21461bef0ad6" (UID: "69c8f195-819a-4f81-916f-21461bef0ad6"). InnerVolumeSpecName "kube-api-access-ktckh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 19:42:04.753781 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:42:04.753738    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktckh\" (UniqueName: \"kubernetes.io/projected/69c8f195-819a-4f81-916f-21461bef0ad6-kube-api-access-ktckh\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:42:04.753781 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:42:04.753773    2546 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/69c8f195-819a-4f81-916f-21461bef0ad6-extensions-socket-volume\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:42:05.550939 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:42:05.550855    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-wpm7h"
Sep 10 19:42:05.550939 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:42:05.550880    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-wpm7h" event={"ID":"69c8f195-819a-4f81-916f-21461bef0ad6","Type":"ContainerDied","Data":"1d883d1864d892c7c6450e04679a0a4721b7924e8e53aa277d7118839fe1740a"}
Sep 10 19:42:05.551149 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:42:05.550944    2546 scope.go:117] "RemoveContainer" containerID="e123fae4c4770984c0cb9c4b6ee1fb74ce929da86d26df883d2d98b00650bef4"
Sep 10 19:42:05.569599 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:42:05.569571    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-wpm7h"]
Sep 10 19:42:05.576742 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:42:05.576712    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-wpm7h"]
Sep 10 19:42:07.281229 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:42:07.281192    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69c8f195-819a-4f81-916f-21461bef0ad6" path="/var/lib/kubelet/pods/69c8f195-819a-4f81-916f-21461bef0ad6/volumes"
Sep 10 19:43:04.342179 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:43:04.342140    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dkgb4"]
Sep 10 19:43:04.342629 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:43:04.342498    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69c8f195-819a-4f81-916f-21461bef0ad6" containerName="manager"
Sep 10 19:43:04.342629 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:43:04.342511    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="69c8f195-819a-4f81-916f-21461bef0ad6" containerName="manager"
Sep 10 19:43:04.342629 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:43:04.342518    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47463639-8fdd-4475-83a1-862ac426ef5f" containerName="istio-proxy"
Sep 10 19:43:04.342629 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:43:04.342524    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="47463639-8fdd-4475-83a1-862ac426ef5f" containerName="istio-proxy"
Sep 10 19:43:04.342629 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:43:04.342533    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2f23b51-72c3-45c6-82cf-da36af76ec82" containerName="istio-proxy"
Sep 10 19:43:04.342629 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:43:04.342539    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2f23b51-72c3-45c6-82cf-da36af76ec82" containerName="istio-proxy"
Sep 10 19:43:04.342629 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:43:04.342547    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="868773b2-7d37-4196-9ab3-2256b295f31d" containerName="istio-proxy"
Sep 10 19:43:04.342629 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:43:04.342553    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="868773b2-7d37-4196-9ab3-2256b295f31d" containerName="istio-proxy"
Sep 10 19:43:04.342629 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:43:04.342559    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="868773b2-7d37-4196-9ab3-2256b295f31d" containerName="istio-proxy"
Sep 10 19:43:04.342629 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:43:04.342564    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="868773b2-7d37-4196-9ab3-2256b295f31d" containerName="istio-proxy"
Sep 10 19:43:04.342629 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:43:04.342572    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb7a0913-bb02-4165-8cf4-c8a451cf2f6a" containerName="maas-api"
Sep 10 19:43:04.342629 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:43:04.342577    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb7a0913-bb02-4165-8cf4-c8a451cf2f6a" containerName="maas-api"
Sep 10 19:43:04.342629 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:43:04.342585    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9f3bf25-92a3-411e-8b4c-10681ba8611b" containerName="maas-api"
Sep 10 19:43:04.342629 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:43:04.342591    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9f3bf25-92a3-411e-8b4c-10681ba8611b" containerName="maas-api"
Sep 10 19:43:04.342629 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:43:04.342597    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70b83199-6656-4f76-b242-e8508d5dbd20" containerName="main"
Sep 10 19:43:04.342629 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:43:04.342602    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="70b83199-6656-4f76-b242-e8508d5dbd20" containerName="main"
Sep 10 19:43:04.343229 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:43:04.342663    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="868773b2-7d37-4196-9ab3-2256b295f31d" containerName="istio-proxy"
Sep 10 19:43:04.343229 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:43:04.342672    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb7a0913-bb02-4165-8cf4-c8a451cf2f6a" containerName="maas-api"
Sep 10 19:43:04.343229 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:43:04.342678    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="69c8f195-819a-4f81-916f-21461bef0ad6" containerName="manager"
Sep 10 19:43:04.343229 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:43:04.342685    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="47463639-8fdd-4475-83a1-862ac426ef5f" containerName="istio-proxy"
Sep 10 19:43:04.343229 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:43:04.342690    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2f23b51-72c3-45c6-82cf-da36af76ec82" containerName="istio-proxy"
Sep 10 19:43:04.343229 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:43:04.342695    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2f23b51-72c3-45c6-82cf-da36af76ec82" containerName="istio-proxy"
Sep 10 19:43:04.343229 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:43:04.342700    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9f3bf25-92a3-411e-8b4c-10681ba8611b" containerName="maas-api"
Sep 10 19:43:04.343229 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:43:04.342705    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="47463639-8fdd-4475-83a1-862ac426ef5f" containerName="istio-proxy"
Sep 10 19:43:04.343229 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:43:04.342710    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="70b83199-6656-4f76-b242-e8508d5dbd20" containerName="main"
Sep 10 19:43:04.345752 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:43:04.345730    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dkgb4"
Sep 10 19:43:04.349547 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:43:04.349527    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-89nlp\""
Sep 10 19:43:04.359313 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:43:04.359281    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dkgb4"]
Sep 10 19:43:04.479152 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:43:04.479106    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fpbv\" (UniqueName: \"kubernetes.io/projected/b3cfc842-39ae-4984-9d76-98a5215e996c-kube-api-access-6fpbv\") pod \"kuadrant-operator-controller-manager-5769f99d9f-dkgb4\" (UID: \"b3cfc842-39ae-4984-9d76-98a5215e996c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dkgb4"
Sep 10 19:43:04.479365 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:43:04.479181    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b3cfc842-39ae-4984-9d76-98a5215e996c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-dkgb4\" (UID: \"b3cfc842-39ae-4984-9d76-98a5215e996c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dkgb4"
Sep 10 19:43:04.580431 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:43:04.580386    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fpbv\" (UniqueName: \"kubernetes.io/projected/b3cfc842-39ae-4984-9d76-98a5215e996c-kube-api-access-6fpbv\") pod \"kuadrant-operator-controller-manager-5769f99d9f-dkgb4\" (UID: \"b3cfc842-39ae-4984-9d76-98a5215e996c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dkgb4"
Sep 10 19:43:04.580610 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:43:04.580461    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b3cfc842-39ae-4984-9d76-98a5215e996c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-dkgb4\" (UID: \"b3cfc842-39ae-4984-9d76-98a5215e996c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dkgb4"
Sep 10 19:43:04.580817 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:43:04.580796    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b3cfc842-39ae-4984-9d76-98a5215e996c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-dkgb4\" (UID: \"b3cfc842-39ae-4984-9d76-98a5215e996c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dkgb4"
Sep 10 19:43:04.589917 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:43:04.589882    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fpbv\" (UniqueName: \"kubernetes.io/projected/b3cfc842-39ae-4984-9d76-98a5215e996c-kube-api-access-6fpbv\") pod \"kuadrant-operator-controller-manager-5769f99d9f-dkgb4\" (UID: \"b3cfc842-39ae-4984-9d76-98a5215e996c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dkgb4"
Sep 10 19:43:04.655701 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:43:04.655664    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dkgb4"
Sep 10 19:43:04.939572 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:43:04.939546    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dkgb4"]
Sep 10 19:43:04.942497 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:43:04.942465    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3cfc842_39ae_4984_9d76_98a5215e996c.slice/crio-0e8e731eba2822df9fca576fa83758c01516b6265491c88c970841ba3a487786 WatchSource:0}: Error finding container 0e8e731eba2822df9fca576fa83758c01516b6265491c88c970841ba3a487786: Status 404 returned error can't find the container with id 0e8e731eba2822df9fca576fa83758c01516b6265491c88c970841ba3a487786
Sep 10 19:43:05.821482 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:43:05.821445    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dkgb4" event={"ID":"b3cfc842-39ae-4984-9d76-98a5215e996c","Type":"ContainerStarted","Data":"e01ee7d624928abc488d831c8d62f28a9f1971038fd247daf65704074e7765eb"}
Sep 10 19:43:05.821880 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:43:05.821506    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dkgb4" event={"ID":"b3cfc842-39ae-4984-9d76-98a5215e996c","Type":"ContainerStarted","Data":"0e8e731eba2822df9fca576fa83758c01516b6265491c88c970841ba3a487786"}
Sep 10 19:43:05.821880 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:43:05.821562    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dkgb4"
Sep 10 19:43:05.848742 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:43:05.848691    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dkgb4" podStartSLOduration=1.848675156 podStartE2EDuration="1.848675156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:43:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:43:05.847431686 +0000 UTC m=+1949.076481714" watchObservedRunningTime="2026-09-10 19:43:05.848675156 +0000 UTC m=+1949.077725180"
Sep 10 19:43:16.828488 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:43:16.828457    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dkgb4"
Sep 10 19:43:26.518754 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:43:26.518717    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-88n8f"]
Sep 10 19:43:34.621126 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:43:34.621032    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-88n8f"]
Sep 10 19:43:43.223355 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:43:43.223304    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-88n8f"]
Sep 10 19:43:52.448384 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:43:52.448323    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-d6143394-kserve-6c45955c95-mrzl7"]
Sep 10 19:43:52.448985 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:43:52.448842    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47463639-8fdd-4475-83a1-862ac426ef5f" containerName="istio-proxy"
Sep 10 19:43:52.448985 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:43:52.448862    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="47463639-8fdd-4475-83a1-862ac426ef5f" containerName="istio-proxy"
Sep 10 19:43:52.448985 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:43:52.448875    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2f23b51-72c3-45c6-82cf-da36af76ec82" containerName="istio-proxy"
Sep 10 19:43:52.448985 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:43:52.448884    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2f23b51-72c3-45c6-82cf-da36af76ec82" containerName="istio-proxy"
Sep 10 19:43:52.449206 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:43:52.448993    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="868773b2-7d37-4196-9ab3-2256b295f31d" containerName="istio-proxy"
Sep 10 19:43:52.452134 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:43:52.452113    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-d6143394-kserve-6c45955c95-mrzl7"
Sep 10 19:43:52.455265 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:43:52.455240    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-d6143394-kserve-self-signed-certs\""
Sep 10 19:43:52.461811 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:43:52.461787    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-d6143394-kserve-6c45955c95-mrzl7"]
Sep 10 19:43:52.506283 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:43:52.506247    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3eaf2e23-fb35-43c0-be5a-511981fbad06-tmp-dir\") pod \"e2e-same-id-a-d6143394-kserve-6c45955c95-mrzl7\" (UID: \"3eaf2e23-fb35-43c0-be5a-511981fbad06\") " pod="llm/e2e-same-id-a-d6143394-kserve-6c45955c95-mrzl7"
Sep 10 19:43:52.506283 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:43:52.506288    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjqbd\" (UniqueName: \"kubernetes.io/projected/3eaf2e23-fb35-43c0-be5a-511981fbad06-kube-api-access-xjqbd\") pod \"e2e-same-id-a-d6143394-kserve-6c45955c95-mrzl7\" (UID: \"3eaf2e23-fb35-43c0-be5a-511981fbad06\") " pod="llm/e2e-same-id-a-d6143394-kserve-6c45955c95-mrzl7"
Sep 10 19:43:52.506555 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:43:52.506324    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3eaf2e23-fb35-43c0-be5a-511981fbad06-tls-certs\") pod \"e2e-same-id-a-d6143394-kserve-6c45955c95-mrzl7\" (UID: \"3eaf2e23-fb35-43c0-be5a-511981fbad06\") " pod="llm/e2e-same-id-a-d6143394-kserve-6c45955c95-mrzl7"
Sep 10 19:43:52.506555 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:43:52.506364    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3eaf2e23-fb35-43c0-be5a-511981fbad06-dshm\") pod \"e2e-same-id-a-d6143394-kserve-6c45955c95-mrzl7\" (UID: \"3eaf2e23-fb35-43c0-be5a-511981fbad06\") " pod="llm/e2e-same-id-a-d6143394-kserve-6c45955c95-mrzl7"
Sep 10 19:43:52.506555 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:43:52.506381    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3eaf2e23-fb35-43c0-be5a-511981fbad06-home\") pod \"e2e-same-id-a-d6143394-kserve-6c45955c95-mrzl7\" (UID: \"3eaf2e23-fb35-43c0-be5a-511981fbad06\") " pod="llm/e2e-same-id-a-d6143394-kserve-6c45955c95-mrzl7"
Sep 10 19:43:52.506555 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:43:52.506424    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3eaf2e23-fb35-43c0-be5a-511981fbad06-model-cache\") pod \"e2e-same-id-a-d6143394-kserve-6c45955c95-mrzl7\" (UID: \"3eaf2e23-fb35-43c0-be5a-511981fbad06\") " pod="llm/e2e-same-id-a-d6143394-kserve-6c45955c95-mrzl7"
Sep 10 19:43:52.607123 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:43:52.607086    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3eaf2e23-fb35-43c0-be5a-511981fbad06-tmp-dir\") pod \"e2e-same-id-a-d6143394-kserve-6c45955c95-mrzl7\" (UID: \"3eaf2e23-fb35-43c0-be5a-511981fbad06\") " pod="llm/e2e-same-id-a-d6143394-kserve-6c45955c95-mrzl7"
Sep 10 19:43:52.607317 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:43:52.607134    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjqbd\" (UniqueName: \"kubernetes.io/projected/3eaf2e23-fb35-43c0-be5a-511981fbad06-kube-api-access-xjqbd\") pod \"e2e-same-id-a-d6143394-kserve-6c45955c95-mrzl7\" (UID: \"3eaf2e23-fb35-43c0-be5a-511981fbad06\") " pod="llm/e2e-same-id-a-d6143394-kserve-6c45955c95-mrzl7"
Sep 10 19:43:52.607317 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:43:52.607196    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3eaf2e23-fb35-43c0-be5a-511981fbad06-tls-certs\") pod \"e2e-same-id-a-d6143394-kserve-6c45955c95-mrzl7\" (UID: \"3eaf2e23-fb35-43c0-be5a-511981fbad06\") " pod="llm/e2e-same-id-a-d6143394-kserve-6c45955c95-mrzl7"
Sep 10 19:43:52.607317 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:43:52.607219    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3eaf2e23-fb35-43c0-be5a-511981fbad06-dshm\") pod \"e2e-same-id-a-d6143394-kserve-6c45955c95-mrzl7\" (UID: \"3eaf2e23-fb35-43c0-be5a-511981fbad06\") " pod="llm/e2e-same-id-a-d6143394-kserve-6c45955c95-mrzl7"
Sep 10 19:43:52.607317 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:43:52.607245    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3eaf2e23-fb35-43c0-be5a-511981fbad06-home\") pod \"e2e-same-id-a-d6143394-kserve-6c45955c95-mrzl7\" (UID: \"3eaf2e23-fb35-43c0-be5a-511981fbad06\") " pod="llm/e2e-same-id-a-d6143394-kserve-6c45955c95-mrzl7"
Sep 10 19:43:52.607577 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:43:52.607323    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3eaf2e23-fb35-43c0-be5a-511981fbad06-model-cache\") pod \"e2e-same-id-a-d6143394-kserve-6c45955c95-mrzl7\" (UID: \"3eaf2e23-fb35-43c0-be5a-511981fbad06\") " pod="llm/e2e-same-id-a-d6143394-kserve-6c45955c95-mrzl7"
Sep 10 19:43:52.607577 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:43:52.607546    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3eaf2e23-fb35-43c0-be5a-511981fbad06-tmp-dir\") pod \"e2e-same-id-a-d6143394-kserve-6c45955c95-mrzl7\" (UID: \"3eaf2e23-fb35-43c0-be5a-511981fbad06\") " pod="llm/e2e-same-id-a-d6143394-kserve-6c45955c95-mrzl7"
Sep 10 19:43:52.607687 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:43:52.607665    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3eaf2e23-fb35-43c0-be5a-511981fbad06-home\") pod \"e2e-same-id-a-d6143394-kserve-6c45955c95-mrzl7\" (UID: \"3eaf2e23-fb35-43c0-be5a-511981fbad06\") " pod="llm/e2e-same-id-a-d6143394-kserve-6c45955c95-mrzl7"
Sep 10 19:43:52.607726 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:43:52.607705    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3eaf2e23-fb35-43c0-be5a-511981fbad06-model-cache\") pod \"e2e-same-id-a-d6143394-kserve-6c45955c95-mrzl7\" (UID: \"3eaf2e23-fb35-43c0-be5a-511981fbad06\") " pod="llm/e2e-same-id-a-d6143394-kserve-6c45955c95-mrzl7"
Sep 10 19:43:52.609579 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:43:52.609554    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3eaf2e23-fb35-43c0-be5a-511981fbad06-dshm\") pod \"e2e-same-id-a-d6143394-kserve-6c45955c95-mrzl7\" (UID: \"3eaf2e23-fb35-43c0-be5a-511981fbad06\") " pod="llm/e2e-same-id-a-d6143394-kserve-6c45955c95-mrzl7"
Sep 10 19:43:52.609921 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:43:52.609901    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3eaf2e23-fb35-43c0-be5a-511981fbad06-tls-certs\") pod \"e2e-same-id-a-d6143394-kserve-6c45955c95-mrzl7\" (UID: \"3eaf2e23-fb35-43c0-be5a-511981fbad06\") " pod="llm/e2e-same-id-a-d6143394-kserve-6c45955c95-mrzl7"
Sep 10 19:43:52.615624 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:43:52.615600    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjqbd\" (UniqueName: \"kubernetes.io/projected/3eaf2e23-fb35-43c0-be5a-511981fbad06-kube-api-access-xjqbd\") pod \"e2e-same-id-a-d6143394-kserve-6c45955c95-mrzl7\" (UID: \"3eaf2e23-fb35-43c0-be5a-511981fbad06\") " pod="llm/e2e-same-id-a-d6143394-kserve-6c45955c95-mrzl7"
Sep 10 19:43:52.761847 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:43:52.761751    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-d6143394-kserve-6c45955c95-mrzl7"
Sep 10 19:43:53.036144 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:43:53.036114    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-d6143394-kserve-6c45955c95-mrzl7"]
Sep 10 19:43:53.038536 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:43:53.038506    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3eaf2e23_fb35_43c0_be5a_511981fbad06.slice/crio-dfb722e69fc0132139d8b3ed65e3af59767234ce81dec518fd2640f44e079ea5 WatchSource:0}: Error finding container dfb722e69fc0132139d8b3ed65e3af59767234ce81dec518fd2640f44e079ea5: Status 404 returned error can't find the container with id dfb722e69fc0132139d8b3ed65e3af59767234ce81dec518fd2640f44e079ea5
Sep 10 19:43:54.047415 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:43:54.047375    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-d6143394-kserve-6c45955c95-mrzl7" event={"ID":"3eaf2e23-fb35-43c0-be5a-511981fbad06","Type":"ContainerStarted","Data":"f5dc7a1c5ca581fd52a134057b3c9881dd0d75c215507daf7c7de6ac513703f1"}
Sep 10 19:43:54.047922 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:43:54.047429    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-d6143394-kserve-6c45955c95-mrzl7" event={"ID":"3eaf2e23-fb35-43c0-be5a-511981fbad06","Type":"ContainerStarted","Data":"dfb722e69fc0132139d8b3ed65e3af59767234ce81dec518fd2640f44e079ea5"}
Sep 10 19:43:54.067063 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:43:54.067004    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-d6143394-kserve-6c45955c95-mrzl7" podStartSLOduration=2.066985519 podStartE2EDuration="2.066985519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:43:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:43:54.065454294 +0000 UTC m=+1997.294504322" watchObservedRunningTime="2026-09-10 19:43:54.066985519 +0000 UTC m=+1997.296035543"
Sep 10 19:43:55.027943 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:43:55.027906    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-88n8f"]
Sep 10 19:44:02.762502 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:44:02.762450    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-d6143394-kserve-6c45955c95-mrzl7"
Sep 10 19:44:02.762924 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:44:02.762532    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-d6143394-kserve-6c45955c95-mrzl7"
Sep 10 19:44:02.775383 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:44:02.775353    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-d6143394-kserve-6c45955c95-mrzl7"
Sep 10 19:44:03.099221 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:44:03.099136    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-d6143394-kserve-6c45955c95-mrzl7"
Sep 10 19:44:05.424423 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:44:05.424386    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-88n8f"]
Sep 10 19:44:16.773689 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:44:16.773654    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-d6143394-kserve-6c45955c95-mrzl7"]
Sep 10 19:44:19.041944 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:44:19.041907    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-88n8f"]
Sep 10 19:44:28.019517 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:44:28.019479    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-88n8f"]
Sep 10 19:44:31.793088 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:44:31.793020    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-d6143394-kserve-6c45955c95-mrzl7" podUID="3eaf2e23-fb35-43c0-be5a-511981fbad06" containerName="main" containerID="cri-o://f5dc7a1c5ca581fd52a134057b3c9881dd0d75c215507daf7c7de6ac513703f1" gracePeriod=45
Sep 10 19:44:32.042376 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:44:32.042331    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-d6143394-kserve-6c45955c95-mrzl7"
Sep 10 19:44:32.169792 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:44:32.169745    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3eaf2e23-fb35-43c0-be5a-511981fbad06-home\") pod \"3eaf2e23-fb35-43c0-be5a-511981fbad06\" (UID: \"3eaf2e23-fb35-43c0-be5a-511981fbad06\") "
Sep 10 19:44:32.169792 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:44:32.169792    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3eaf2e23-fb35-43c0-be5a-511981fbad06-tls-certs\") pod \"3eaf2e23-fb35-43c0-be5a-511981fbad06\" (UID: \"3eaf2e23-fb35-43c0-be5a-511981fbad06\") "
Sep 10 19:44:32.170017 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:44:32.169822    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3eaf2e23-fb35-43c0-be5a-511981fbad06-tmp-dir\") pod \"3eaf2e23-fb35-43c0-be5a-511981fbad06\" (UID: \"3eaf2e23-fb35-43c0-be5a-511981fbad06\") "
Sep 10 19:44:32.170017 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:44:32.169886    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xjqbd\" (UniqueName: \"kubernetes.io/projected/3eaf2e23-fb35-43c0-be5a-511981fbad06-kube-api-access-xjqbd\") pod \"3eaf2e23-fb35-43c0-be5a-511981fbad06\" (UID: \"3eaf2e23-fb35-43c0-be5a-511981fbad06\") "
Sep 10 19:44:32.170017 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:44:32.169944    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3eaf2e23-fb35-43c0-be5a-511981fbad06-model-cache\") pod \"3eaf2e23-fb35-43c0-be5a-511981fbad06\" (UID: \"3eaf2e23-fb35-43c0-be5a-511981fbad06\") "
Sep 10 19:44:32.170017 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:44:32.169976    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3eaf2e23-fb35-43c0-be5a-511981fbad06-dshm\") pod \"3eaf2e23-fb35-43c0-be5a-511981fbad06\" (UID: \"3eaf2e23-fb35-43c0-be5a-511981fbad06\") "
Sep 10 19:44:32.170204 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:44:32.170109    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eaf2e23-fb35-43c0-be5a-511981fbad06-home" (OuterVolumeSpecName: "home") pod "3eaf2e23-fb35-43c0-be5a-511981fbad06" (UID: "3eaf2e23-fb35-43c0-be5a-511981fbad06"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:44:32.170204 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:44:32.170137    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eaf2e23-fb35-43c0-be5a-511981fbad06-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3eaf2e23-fb35-43c0-be5a-511981fbad06" (UID: "3eaf2e23-fb35-43c0-be5a-511981fbad06"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:44:32.170319 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:44:32.170197    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eaf2e23-fb35-43c0-be5a-511981fbad06-model-cache" (OuterVolumeSpecName: "model-cache") pod "3eaf2e23-fb35-43c0-be5a-511981fbad06" (UID: "3eaf2e23-fb35-43c0-be5a-511981fbad06"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:44:32.170319 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:44:32.170280    2546 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3eaf2e23-fb35-43c0-be5a-511981fbad06-home\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:44:32.170319 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:44:32.170294    2546 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3eaf2e23-fb35-43c0-be5a-511981fbad06-tmp-dir\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:44:32.170319 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:44:32.170303    2546 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3eaf2e23-fb35-43c0-be5a-511981fbad06-model-cache\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:44:32.171963 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:44:32.171938    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eaf2e23-fb35-43c0-be5a-511981fbad06-dshm" (OuterVolumeSpecName: "dshm") pod "3eaf2e23-fb35-43c0-be5a-511981fbad06" (UID: "3eaf2e23-fb35-43c0-be5a-511981fbad06"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 19:44:32.172081 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:44:32.172061    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3eaf2e23-fb35-43c0-be5a-511981fbad06-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3eaf2e23-fb35-43c0-be5a-511981fbad06" (UID: "3eaf2e23-fb35-43c0-be5a-511981fbad06"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 19:44:32.172126 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:44:32.172092    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3eaf2e23-fb35-43c0-be5a-511981fbad06-kube-api-access-xjqbd" (OuterVolumeSpecName: "kube-api-access-xjqbd") pod "3eaf2e23-fb35-43c0-be5a-511981fbad06" (UID: "3eaf2e23-fb35-43c0-be5a-511981fbad06"). InnerVolumeSpecName "kube-api-access-xjqbd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 19:44:32.230247 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:44:32.230211    2546 generic.go:356] "Generic (PLEG): container finished" podID="3eaf2e23-fb35-43c0-be5a-511981fbad06" containerID="f5dc7a1c5ca581fd52a134057b3c9881dd0d75c215507daf7c7de6ac513703f1" exitCode=0
Sep 10 19:44:32.230452 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:44:32.230270    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-d6143394-kserve-6c45955c95-mrzl7"
Sep 10 19:44:32.230452 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:44:32.230306    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-d6143394-kserve-6c45955c95-mrzl7" event={"ID":"3eaf2e23-fb35-43c0-be5a-511981fbad06","Type":"ContainerDied","Data":"f5dc7a1c5ca581fd52a134057b3c9881dd0d75c215507daf7c7de6ac513703f1"}
Sep 10 19:44:32.230452 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:44:32.230370    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-d6143394-kserve-6c45955c95-mrzl7" event={"ID":"3eaf2e23-fb35-43c0-be5a-511981fbad06","Type":"ContainerDied","Data":"dfb722e69fc0132139d8b3ed65e3af59767234ce81dec518fd2640f44e079ea5"}
Sep 10 19:44:32.230452 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:44:32.230394    2546 scope.go:117] "RemoveContainer" containerID="f5dc7a1c5ca581fd52a134057b3c9881dd0d75c215507daf7c7de6ac513703f1"
Sep 10 19:44:32.239285 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:44:32.239266    2546 scope.go:117] "RemoveContainer" containerID="f5dc7a1c5ca581fd52a134057b3c9881dd0d75c215507daf7c7de6ac513703f1"
Sep 10 19:44:32.239575 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:44:32.239555    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5dc7a1c5ca581fd52a134057b3c9881dd0d75c215507daf7c7de6ac513703f1\": container with ID starting with f5dc7a1c5ca581fd52a134057b3c9881dd0d75c215507daf7c7de6ac513703f1 not found: ID does not exist" containerID="f5dc7a1c5ca581fd52a134057b3c9881dd0d75c215507daf7c7de6ac513703f1"
Sep 10 19:44:32.239620 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:44:32.239582    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5dc7a1c5ca581fd52a134057b3c9881dd0d75c215507daf7c7de6ac513703f1"} err="failed to get container status \"f5dc7a1c5ca581fd52a134057b3c9881dd0d75c215507daf7c7de6ac513703f1\": rpc error: code = NotFound desc = could not find container \"f5dc7a1c5ca581fd52a134057b3c9881dd0d75c215507daf7c7de6ac513703f1\": container with ID starting with f5dc7a1c5ca581fd52a134057b3c9881dd0d75c215507daf7c7de6ac513703f1 not found: ID does not exist"
Sep 10 19:44:32.252597 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:44:32.252571    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-d6143394-kserve-6c45955c95-mrzl7"]
Sep 10 19:44:32.255605 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:44:32.255585    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-d6143394-kserve-6c45955c95-mrzl7"]
Sep 10 19:44:32.271368 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:44:32.271326    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjqbd\" (UniqueName: \"kubernetes.io/projected/3eaf2e23-fb35-43c0-be5a-511981fbad06-kube-api-access-xjqbd\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:44:32.271368 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:44:32.271362    2546 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3eaf2e23-fb35-43c0-be5a-511981fbad06-dshm\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:44:32.271519 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:44:32.271375    2546 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3eaf2e23-fb35-43c0-be5a-511981fbad06-tls-certs\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:44:33.280801 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:44:33.280762    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3eaf2e23-fb35-43c0-be5a-511981fbad06" path="/var/lib/kubelet/pods/3eaf2e23-fb35-43c0-be5a-511981fbad06/volumes"
Sep 10 19:44:37.120609 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:44:37.120571    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-88n8f"]
Sep 10 19:44:37.538538 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:44:37.538456    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-777cfbf6dc-mjjzm"]
Sep 10 19:44:37.538819 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:44:37.538806    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3eaf2e23-fb35-43c0-be5a-511981fbad06" containerName="main"
Sep 10 19:44:37.538863 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:44:37.538821    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="3eaf2e23-fb35-43c0-be5a-511981fbad06" containerName="main"
Sep 10 19:44:37.538898 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:44:37.538880    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="3eaf2e23-fb35-43c0-be5a-511981fbad06" containerName="main"
Sep 10 19:44:37.543294 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:44:37.543273    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-777cfbf6dc-mjjzm"
Sep 10 19:44:37.555559 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:44:37.555532    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-777cfbf6dc-mjjzm"]
Sep 10 19:44:37.616312 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:44:37.616268    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/337ef04e-dfeb-4f91-a0ef-8fcda543a72c-maas-api-tls\") pod \"maas-api-777cfbf6dc-mjjzm\" (UID: \"337ef04e-dfeb-4f91-a0ef-8fcda543a72c\") " pod="odh-ai-gateway-infra/maas-api-777cfbf6dc-mjjzm"
Sep 10 19:44:37.616506 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:44:37.616330    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqjnt\" (UniqueName: \"kubernetes.io/projected/337ef04e-dfeb-4f91-a0ef-8fcda543a72c-kube-api-access-jqjnt\") pod \"maas-api-777cfbf6dc-mjjzm\" (UID: \"337ef04e-dfeb-4f91-a0ef-8fcda543a72c\") " pod="odh-ai-gateway-infra/maas-api-777cfbf6dc-mjjzm"
Sep 10 19:44:37.616506 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:44:37.616380    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/337ef04e-dfeb-4f91-a0ef-8fcda543a72c-openshift-service-ca\") pod \"maas-api-777cfbf6dc-mjjzm\" (UID: \"337ef04e-dfeb-4f91-a0ef-8fcda543a72c\") " pod="odh-ai-gateway-infra/maas-api-777cfbf6dc-mjjzm"
Sep 10 19:44:37.616506 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:44:37.616463    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/337ef04e-dfeb-4f91-a0ef-8fcda543a72c-metrics-certs\") pod \"maas-api-777cfbf6dc-mjjzm\" (UID: \"337ef04e-dfeb-4f91-a0ef-8fcda543a72c\") " pod="odh-ai-gateway-infra/maas-api-777cfbf6dc-mjjzm"
Sep 10 19:44:37.716894 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:44:37.716863    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/337ef04e-dfeb-4f91-a0ef-8fcda543a72c-openshift-service-ca\") pod \"maas-api-777cfbf6dc-mjjzm\" (UID: \"337ef04e-dfeb-4f91-a0ef-8fcda543a72c\") " pod="odh-ai-gateway-infra/maas-api-777cfbf6dc-mjjzm"
Sep 10 19:44:37.717095 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:44:37.716942    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/337ef04e-dfeb-4f91-a0ef-8fcda543a72c-metrics-certs\") pod \"maas-api-777cfbf6dc-mjjzm\" (UID: \"337ef04e-dfeb-4f91-a0ef-8fcda543a72c\") " pod="odh-ai-gateway-infra/maas-api-777cfbf6dc-mjjzm"
Sep 10 19:44:37.717095 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:44:37.716983    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/337ef04e-dfeb-4f91-a0ef-8fcda543a72c-maas-api-tls\") pod \"maas-api-777cfbf6dc-mjjzm\" (UID: \"337ef04e-dfeb-4f91-a0ef-8fcda543a72c\") " pod="odh-ai-gateway-infra/maas-api-777cfbf6dc-mjjzm"
Sep 10 19:44:37.717095 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:44:37.717015    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqjnt\" (UniqueName: \"kubernetes.io/projected/337ef04e-dfeb-4f91-a0ef-8fcda543a72c-kube-api-access-jqjnt\") pod \"maas-api-777cfbf6dc-mjjzm\" (UID: \"337ef04e-dfeb-4f91-a0ef-8fcda543a72c\") " pod="odh-ai-gateway-infra/maas-api-777cfbf6dc-mjjzm"
Sep 10 19:44:37.717672 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:44:37.717644    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/337ef04e-dfeb-4f91-a0ef-8fcda543a72c-openshift-service-ca\") pod \"maas-api-777cfbf6dc-mjjzm\" (UID: \"337ef04e-dfeb-4f91-a0ef-8fcda543a72c\") " pod="odh-ai-gateway-infra/maas-api-777cfbf6dc-mjjzm"
Sep 10 19:44:37.719869 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:44:37.719843    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/337ef04e-dfeb-4f91-a0ef-8fcda543a72c-maas-api-tls\") pod \"maas-api-777cfbf6dc-mjjzm\" (UID: \"337ef04e-dfeb-4f91-a0ef-8fcda543a72c\") " pod="odh-ai-gateway-infra/maas-api-777cfbf6dc-mjjzm"
Sep 10 19:44:37.719989 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:44:37.719914    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/337ef04e-dfeb-4f91-a0ef-8fcda543a72c-metrics-certs\") pod \"maas-api-777cfbf6dc-mjjzm\" (UID: \"337ef04e-dfeb-4f91-a0ef-8fcda543a72c\") " pod="odh-ai-gateway-infra/maas-api-777cfbf6dc-mjjzm"
Sep 10 19:44:37.726444 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:44:37.726420    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqjnt\" (UniqueName: \"kubernetes.io/projected/337ef04e-dfeb-4f91-a0ef-8fcda543a72c-kube-api-access-jqjnt\") pod \"maas-api-777cfbf6dc-mjjzm\" (UID: \"337ef04e-dfeb-4f91-a0ef-8fcda543a72c\") " pod="odh-ai-gateway-infra/maas-api-777cfbf6dc-mjjzm"
Sep 10 19:44:37.855257 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:44:37.855165    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-777cfbf6dc-mjjzm"
Sep 10 19:44:38.132208 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:44:38.132182    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-777cfbf6dc-mjjzm"]
Sep 10 19:44:38.134970 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:44:38.134944    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod337ef04e_dfeb_4f91_a0ef_8fcda543a72c.slice/crio-8546c73a83475ce79c5703097a3eecb7709deb2515010d863669cd82dfb52a62 WatchSource:0}: Error finding container 8546c73a83475ce79c5703097a3eecb7709deb2515010d863669cd82dfb52a62: Status 404 returned error can't find the container with id 8546c73a83475ce79c5703097a3eecb7709deb2515010d863669cd82dfb52a62
Sep 10 19:44:38.136669 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:44:38.136648    2546 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 19:44:38.258911 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:44:38.258871    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-777cfbf6dc-mjjzm" event={"ID":"337ef04e-dfeb-4f91-a0ef-8fcda543a72c","Type":"ContainerStarted","Data":"8546c73a83475ce79c5703097a3eecb7709deb2515010d863669cd82dfb52a62"}
Sep 10 19:44:39.266028 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:44:39.265987    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-777cfbf6dc-mjjzm" event={"ID":"337ef04e-dfeb-4f91-a0ef-8fcda543a72c","Type":"ContainerStarted","Data":"06c11c528dea33c32a214b4a8e403c4340e40e59d78c0befd334cc89be8fed85"}
Sep 10 19:44:39.266492 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:44:39.266100    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-777cfbf6dc-mjjzm"
Sep 10 19:44:39.290542 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:44:39.290492    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-777cfbf6dc-mjjzm" podStartSLOduration=1.804189364 podStartE2EDuration="2.290471005s" totalImagesPullingTime="486.281641ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:44:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:44:39.288675295 +0000 UTC m=+2042.517725319" watchObservedRunningTime="2026-09-10 19:44:39.290471005 +0000 UTC m=+2042.519521037"
Sep 10 19:44:44.924958 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:44:44.924912    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-88n8f"]
Sep 10 19:44:45.280976 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:44:45.280898    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-777cfbf6dc-mjjzm"
Sep 10 19:44:45.333784 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:44:45.333744    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-ff9f6d487-dwfz2"]
Sep 10 19:44:45.334052 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:44:45.334023    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-ff9f6d487-dwfz2" podUID="019b7970-6f5b-4637-b611-894853080e70" containerName="maas-api" containerID="cri-o://15d58ad1a0c21007baa46d710559909d21fc79b6e0562a2f0c1f5829ef35fa69" gracePeriod=30
Sep 10 19:44:46.587115 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:44:46.587087    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-ff9f6d487-dwfz2"
Sep 10 19:44:46.702385 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:44:46.702267    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/019b7970-6f5b-4637-b611-894853080e70-metrics-certs\") pod \"019b7970-6f5b-4637-b611-894853080e70\" (UID: \"019b7970-6f5b-4637-b611-894853080e70\") "
Sep 10 19:44:46.702385 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:44:46.702370    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/019b7970-6f5b-4637-b611-894853080e70-maas-api-tls\") pod \"019b7970-6f5b-4637-b611-894853080e70\" (UID: \"019b7970-6f5b-4637-b611-894853080e70\") "
Sep 10 19:44:46.702643 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:44:46.702402    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/019b7970-6f5b-4637-b611-894853080e70-openshift-service-ca\") pod \"019b7970-6f5b-4637-b611-894853080e70\" (UID: \"019b7970-6f5b-4637-b611-894853080e70\") "
Sep 10 19:44:46.702643 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:44:46.702439    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xhtzd\" (UniqueName: \"kubernetes.io/projected/019b7970-6f5b-4637-b611-894853080e70-kube-api-access-xhtzd\") pod \"019b7970-6f5b-4637-b611-894853080e70\" (UID: \"019b7970-6f5b-4637-b611-894853080e70\") "
Sep 10 19:44:46.702829 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:44:46.702804    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/019b7970-6f5b-4637-b611-894853080e70-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "019b7970-6f5b-4637-b611-894853080e70" (UID: "019b7970-6f5b-4637-b611-894853080e70"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 19:44:46.704626 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:44:46.704599    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/019b7970-6f5b-4637-b611-894853080e70-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "019b7970-6f5b-4637-b611-894853080e70" (UID: "019b7970-6f5b-4637-b611-894853080e70"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 19:44:46.704720 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:44:46.704675    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/019b7970-6f5b-4637-b611-894853080e70-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "019b7970-6f5b-4637-b611-894853080e70" (UID: "019b7970-6f5b-4637-b611-894853080e70"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 19:44:46.704720 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:44:46.704681    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/019b7970-6f5b-4637-b611-894853080e70-kube-api-access-xhtzd" (OuterVolumeSpecName: "kube-api-access-xhtzd") pod "019b7970-6f5b-4637-b611-894853080e70" (UID: "019b7970-6f5b-4637-b611-894853080e70"). InnerVolumeSpecName "kube-api-access-xhtzd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 19:44:46.803325 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:44:46.803280    2546 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/019b7970-6f5b-4637-b611-894853080e70-metrics-certs\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:44:46.803325 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:44:46.803319    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/019b7970-6f5b-4637-b611-894853080e70-maas-api-tls\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:44:46.803325 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:44:46.803329    2546 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/019b7970-6f5b-4637-b611-894853080e70-openshift-service-ca\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:44:46.803325 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:44:46.803361    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhtzd\" (UniqueName: \"kubernetes.io/projected/019b7970-6f5b-4637-b611-894853080e70-kube-api-access-xhtzd\") on node \"ip-10-0-132-222.ec2.internal\" DevicePath \"\""
Sep 10 19:44:47.305703 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:44:47.305670    2546 generic.go:356] "Generic (PLEG): container finished" podID="019b7970-6f5b-4637-b611-894853080e70" containerID="15d58ad1a0c21007baa46d710559909d21fc79b6e0562a2f0c1f5829ef35fa69" exitCode=0
Sep 10 19:44:47.305835 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:44:47.305727    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-ff9f6d487-dwfz2" event={"ID":"019b7970-6f5b-4637-b611-894853080e70","Type":"ContainerDied","Data":"15d58ad1a0c21007baa46d710559909d21fc79b6e0562a2f0c1f5829ef35fa69"}
Sep 10 19:44:47.305835 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:44:47.305731    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-ff9f6d487-dwfz2"
Sep 10 19:44:47.305835 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:44:47.305766    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-ff9f6d487-dwfz2" event={"ID":"019b7970-6f5b-4637-b611-894853080e70","Type":"ContainerDied","Data":"0866aa354f526cbadca8c9defdab7138aaa685f4d9db2174337305753ec0930a"}
Sep 10 19:44:47.305835 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:44:47.305787    2546 scope.go:117] "RemoveContainer" containerID="15d58ad1a0c21007baa46d710559909d21fc79b6e0562a2f0c1f5829ef35fa69"
Sep 10 19:44:47.314299 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:44:47.314276    2546 scope.go:117] "RemoveContainer" containerID="15d58ad1a0c21007baa46d710559909d21fc79b6e0562a2f0c1f5829ef35fa69"
Sep 10 19:44:47.314650 ip-10-0-132-222 kubenswrapper[2546]: E0910 19:44:47.314619    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15d58ad1a0c21007baa46d710559909d21fc79b6e0562a2f0c1f5829ef35fa69\": container with ID starting with 15d58ad1a0c21007baa46d710559909d21fc79b6e0562a2f0c1f5829ef35fa69 not found: ID does not exist" containerID="15d58ad1a0c21007baa46d710559909d21fc79b6e0562a2f0c1f5829ef35fa69"
Sep 10 19:44:47.314719 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:44:47.314660    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15d58ad1a0c21007baa46d710559909d21fc79b6e0562a2f0c1f5829ef35fa69"} err="failed to get container status \"15d58ad1a0c21007baa46d710559909d21fc79b6e0562a2f0c1f5829ef35fa69\": rpc error: code = NotFound desc = could not find container \"15d58ad1a0c21007baa46d710559909d21fc79b6e0562a2f0c1f5829ef35fa69\": container with ID starting with 15d58ad1a0c21007baa46d710559909d21fc79b6e0562a2f0c1f5829ef35fa69 not found: ID does not exist"
Sep 10 19:44:47.325547 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:44:47.325522    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-ff9f6d487-dwfz2"]
Sep 10 19:44:47.329598 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:44:47.329574    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-ff9f6d487-dwfz2"]
Sep 10 19:44:49.280634 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:44:49.280602    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="019b7970-6f5b-4637-b611-894853080e70" path="/var/lib/kubelet/pods/019b7970-6f5b-4637-b611-894853080e70/volumes"
Sep 10 19:44:52.235285 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:44:52.235246    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-88n8f"]
Sep 10 19:45:00.334982 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:45:00.334881    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-88n8f"]
Sep 10 19:45:08.121000 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:45:08.120958    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-88n8f"]
Sep 10 19:45:16.617413 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:45:16.617380    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-q6cqb_9977ad8b-b590-4004-93b2-7e93892c0b7f/manager/0.log"
Sep 10 19:45:16.737592 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:45:16.737559    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-6lcpk_58903c8d-079b-45c6-9750-717e1fa44405/manager/0.log"
Sep 10 19:45:16.980908 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:45:16.980810    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-jsgn2_f239b2bf-edcf-4852-adc1-c905d4681e29/server/0.log"
Sep 10 19:45:17.219078 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:45:17.219049    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-d54bdf466-f2gzj_f0396ab1-e6a3-4b4e-8ebc-2f0031109a0f/manager/0.log"
Sep 10 19:45:17.226180 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:45:17.226156    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-d54bdf466-f2gzj_f0396ab1-e6a3-4b4e-8ebc-2f0031109a0f/copy-manifests/0.log"
Sep 10 19:45:17.804815 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:45:17.804786    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-777cfbf6dc-mjjzm_337ef04e-dfeb-4f91-a0ef-8fcda543a72c/maas-api/0.log"
Sep 10 19:45:19.147998 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:45:19.147966    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-66vvb_062a8b66-7428-42b5-81e3-c4f3c8730d74/manager/0.log"
Sep 10 19:45:19.253282 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:45:19.253242    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-6gwdr_80fb63b4-18c7-46f3-aac2-110f0ae1efe9/manager/0.log"
Sep 10 19:45:19.366281 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:45:19.366255    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-hp4b2_290976c4-b4cb-4ca3-91b3-57b236ea7a1f/kuadrant-console-plugin/0.log"
Sep 10 19:45:19.490498 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:45:19.490402    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-dkgb4_b3cfc842-39ae-4984-9d76-98a5215e996c/manager/0.log"
Sep 10 19:45:19.605247 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:45:19.605216    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-88n8f_9127a66c-9536-49d4-ae69-0910b3374c49/limitador/0.log"
Sep 10 19:45:20.394263 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:45:20.394228    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6b5f7cd94b-d6plt_bda5e467-10e3-458d-b507-91b0772d3d32/kube-auth-proxy/0.log"
Sep 10 19:45:20.511921 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:45:20.511890    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-ps4m9_698ce0a1-dce5-40d0-8bd4-c2042400e6f2/istio-proxy/1.log"
Sep 10 19:45:21.636086 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:45:21.636055    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjv585_e3d89b4d-5fa9-4ddd-88e0-6034a43ad6f8/main/0.log"
Sep 10 19:45:21.760867 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:45:21.760832    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-t4zlp_7e58e6e4-8e40-45e2-a24f-8d1f800382be/main/0.log"
Sep 10 19:45:21.871010 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:45:21.870977    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-rp6w8_f0679a40-f013-4921-9453-4beb3f651af7/main/0.log"
Sep 10 19:45:26.983557 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:45:26.983521    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-2vr76/must-gather-z56nr"]
Sep 10 19:45:26.983964 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:45:26.983865    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="019b7970-6f5b-4637-b611-894853080e70" containerName="maas-api"
Sep 10 19:45:26.983964 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:45:26.983876    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="019b7970-6f5b-4637-b611-894853080e70" containerName="maas-api"
Sep 10 19:45:26.983964 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:45:26.983947    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="019b7970-6f5b-4637-b611-894853080e70" containerName="maas-api"
Sep 10 19:45:26.986999 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:45:26.986978    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2vr76/must-gather-z56nr"
Sep 10 19:45:26.990072 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:45:26.990050    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2vr76\"/\"openshift-service-ca.crt\""
Sep 10 19:45:26.990184 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:45:26.990077    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2vr76\"/\"kube-root-ca.crt\""
Sep 10 19:45:26.990909 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:45:26.990891    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-2vr76\"/\"default-dockercfg-j88m6\""
Sep 10 19:45:26.999386 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:45:26.999362    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2vr76/must-gather-z56nr"]
Sep 10 19:45:27.058911 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:45:27.058877    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/1e5b8d22-7ae3-4853-b8e8-8546472dfb56-must-gather-output\") pod \"must-gather-z56nr\" (UID: \"1e5b8d22-7ae3-4853-b8e8-8546472dfb56\") " pod="openshift-must-gather-2vr76/must-gather-z56nr"
Sep 10 19:45:27.059090 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:45:27.058923    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfr74\" (UniqueName: \"kubernetes.io/projected/1e5b8d22-7ae3-4853-b8e8-8546472dfb56-kube-api-access-kfr74\") pod \"must-gather-z56nr\" (UID: \"1e5b8d22-7ae3-4853-b8e8-8546472dfb56\") " pod="openshift-must-gather-2vr76/must-gather-z56nr"
Sep 10 19:45:27.159594 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:45:27.159556    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfr74\" (UniqueName: \"kubernetes.io/projected/1e5b8d22-7ae3-4853-b8e8-8546472dfb56-kube-api-access-kfr74\") pod \"must-gather-z56nr\" (UID: \"1e5b8d22-7ae3-4853-b8e8-8546472dfb56\") " pod="openshift-must-gather-2vr76/must-gather-z56nr"
Sep 10 19:45:27.159798 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:45:27.159667    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/1e5b8d22-7ae3-4853-b8e8-8546472dfb56-must-gather-output\") pod \"must-gather-z56nr\" (UID: \"1e5b8d22-7ae3-4853-b8e8-8546472dfb56\") " pod="openshift-must-gather-2vr76/must-gather-z56nr"
Sep 10 19:45:27.160026 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:45:27.160005    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/1e5b8d22-7ae3-4853-b8e8-8546472dfb56-must-gather-output\") pod \"must-gather-z56nr\" (UID: \"1e5b8d22-7ae3-4853-b8e8-8546472dfb56\") " pod="openshift-must-gather-2vr76/must-gather-z56nr"
Sep 10 19:45:27.168931 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:45:27.168903    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfr74\" (UniqueName: \"kubernetes.io/projected/1e5b8d22-7ae3-4853-b8e8-8546472dfb56-kube-api-access-kfr74\") pod \"must-gather-z56nr\" (UID: \"1e5b8d22-7ae3-4853-b8e8-8546472dfb56\") " pod="openshift-must-gather-2vr76/must-gather-z56nr"
Sep 10 19:45:27.296051 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:45:27.295972    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2vr76/must-gather-z56nr"
Sep 10 19:45:27.569868 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:45:27.569841    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2vr76/must-gather-z56nr"]
Sep 10 19:45:27.570843 ip-10-0-132-222 kubenswrapper[2546]: W0910 19:45:27.570820    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1e5b8d22_7ae3_4853_b8e8_8546472dfb56.slice/crio-c25a79065db7ca9c3f075788b0a765fd82b98b67c987da24378175b72e8af874 WatchSource:0}: Error finding container c25a79065db7ca9c3f075788b0a765fd82b98b67c987da24378175b72e8af874: Status 404 returned error can't find the container with id c25a79065db7ca9c3f075788b0a765fd82b98b67c987da24378175b72e8af874
Sep 10 19:45:28.497089 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:45:28.497045    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2vr76/must-gather-z56nr" event={"ID":"1e5b8d22-7ae3-4853-b8e8-8546472dfb56","Type":"ContainerStarted","Data":"8b6a1264654015aa332db3dc38700f99afb0c0fb848a75710217ee35a25dc46e"}
Sep 10 19:45:28.497633 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:45:28.497106    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2vr76/must-gather-z56nr" event={"ID":"1e5b8d22-7ae3-4853-b8e8-8546472dfb56","Type":"ContainerStarted","Data":"c25a79065db7ca9c3f075788b0a765fd82b98b67c987da24378175b72e8af874"}
Sep 10 19:45:29.504843 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:45:29.504504    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2vr76/must-gather-z56nr" event={"ID":"1e5b8d22-7ae3-4853-b8e8-8546472dfb56","Type":"ContainerStarted","Data":"7b2be6a249edbe1ac0b628206334df57ad59c805fc34b0bf81f367119836f8ad"}
Sep 10 19:45:29.524272 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:45:29.524208    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-2vr76/must-gather-z56nr" podStartSLOduration=2.736959202 podStartE2EDuration="3.524185383s" totalImagesPullingTime="787.226181ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:45:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:45:29.523123974 +0000 UTC m=+2092.752173998" watchObservedRunningTime="2026-09-10 19:45:29.524185383 +0000 UTC m=+2092.753235408"
Sep 10 19:45:30.104100 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:45:30.104063    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-qhq2q_63940132-1151-4ec6-b8fd-a68baa948a23/global-pull-secret-syncer/0.log"
Sep 10 19:45:30.240508 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:45:30.240448    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-l4kqt_639f9409-c20b-4e73-aa74-9859ab69bdaa/konnectivity-agent/0.log"
Sep 10 19:45:30.296792 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:45:30.296755    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-132-222.ec2.internal_658d2ad482679f9690b7cc4c32ebd70f/haproxy/0.log"
Sep 10 19:45:35.046216 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:45:35.046150    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-66vvb_062a8b66-7428-42b5-81e3-c4f3c8730d74/manager/0.log"
Sep 10 19:45:35.081186 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:45:35.081153    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-6gwdr_80fb63b4-18c7-46f3-aac2-110f0ae1efe9/manager/0.log"
Sep 10 19:45:35.113078 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:45:35.112463    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-hp4b2_290976c4-b4cb-4ca3-91b3-57b236ea7a1f/kuadrant-console-plugin/0.log"
Sep 10 19:45:35.198114 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:45:35.198082    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-dkgb4_b3cfc842-39ae-4984-9d76-98a5215e996c/manager/0.log"
Sep 10 19:45:35.233747 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:45:35.233716    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-88n8f_9127a66c-9536-49d4-ae69-0910b3374c49/limitador/0.log"
Sep 10 19:45:37.403887 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:45:37.403854    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wd2gl_81bdadb5-2940-4fbd-ae51-5a04b73f89f4/node-exporter/0.log"
Sep 10 19:45:37.433725 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:45:37.433685    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wd2gl_81bdadb5-2940-4fbd-ae51-5a04b73f89f4/kube-rbac-proxy/0.log"
Sep 10 19:45:37.465456 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:45:37.465414    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wd2gl_81bdadb5-2940-4fbd-ae51-5a04b73f89f4/init-textfile/0.log"
Sep 10 19:45:39.639890 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:45:39.639854    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-79977f9844-97jbx_ac0e549a-a86a-4ac5-8a59-9b0cebc5b7f0/networking-console-plugin/0.log"
Sep 10 19:45:40.565867 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:45:40.565826    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-2vr76/perf-node-gather-daemonset-j9qv4"]
Sep 10 19:45:40.569882 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:45:40.569851    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2vr76/perf-node-gather-daemonset-j9qv4"
Sep 10 19:45:40.579587 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:45:40.579558    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2vr76/perf-node-gather-daemonset-j9qv4"]
Sep 10 19:45:40.702043 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:45:40.701998    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khkxx\" (UniqueName: \"kubernetes.io/projected/28490cc3-78ae-4771-a790-30d8b3d3fe44-kube-api-access-khkxx\") pod \"perf-node-gather-daemonset-j9qv4\" (UID: \"28490cc3-78ae-4771-a790-30d8b3d3fe44\") " pod="openshift-must-gather-2vr76/perf-node-gather-daemonset-j9qv4"
Sep 10 19:45:40.702717 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:45:40.702691    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/28490cc3-78ae-4771-a790-30d8b3d3fe44-sys\") pod \"perf-node-gather-daemonset-j9qv4\" (UID: \"28490cc3-78ae-4771-a790-30d8b3d3fe44\") " pod="openshift-must-gather-2vr76/perf-node-gather-daemonset-j9qv4"
Sep 10 19:45:40.702921 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:45:40.702904    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/28490cc3-78ae-4771-a790-30d8b3d3fe44-proc\") pod \"perf-node-gather-daemonset-j9qv4\" (UID: \"28490cc3-78ae-4771-a790-30d8b3d3fe44\") " pod="openshift-must-gather-2vr76/perf-node-gather-daemonset-j9qv4"
Sep 10 19:45:40.703238 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:45:40.703217    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/28490cc3-78ae-4771-a790-30d8b3d3fe44-lib-modules\") pod \"perf-node-gather-daemonset-j9qv4\" (UID: \"28490cc3-78ae-4771-a790-30d8b3d3fe44\") " pod="openshift-must-gather-2vr76/perf-node-gather-daemonset-j9qv4"
Sep 10 19:45:40.703472 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:45:40.703455    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/28490cc3-78ae-4771-a790-30d8b3d3fe44-podres\") pod \"perf-node-gather-daemonset-j9qv4\" (UID: \"28490cc3-78ae-4771-a790-30d8b3d3fe44\") " pod="openshift-must-gather-2vr76/perf-node-gather-daemonset-j9qv4"
Sep 10 19:45:40.804659 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:45:40.804615    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khkxx\" (UniqueName: \"kubernetes.io/projected/28490cc3-78ae-4771-a790-30d8b3d3fe44-kube-api-access-khkxx\") pod \"perf-node-gather-daemonset-j9qv4\" (UID: \"28490cc3-78ae-4771-a790-30d8b3d3fe44\") " pod="openshift-must-gather-2vr76/perf-node-gather-daemonset-j9qv4"
Sep 10 19:45:40.805012 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:45:40.804989    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/28490cc3-78ae-4771-a790-30d8b3d3fe44-sys\") pod \"perf-node-gather-daemonset-j9qv4\" (UID: \"28490cc3-78ae-4771-a790-30d8b3d3fe44\") " pod="openshift-must-gather-2vr76/perf-node-gather-daemonset-j9qv4"
Sep 10 19:45:40.805148 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:45:40.805032    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/28490cc3-78ae-4771-a790-30d8b3d3fe44-sys\") pod \"perf-node-gather-daemonset-j9qv4\" (UID: \"28490cc3-78ae-4771-a790-30d8b3d3fe44\") " pod="openshift-must-gather-2vr76/perf-node-gather-daemonset-j9qv4"
Sep 10 19:45:40.805311 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:45:40.805295    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/28490cc3-78ae-4771-a790-30d8b3d3fe44-proc\") pod \"perf-node-gather-daemonset-j9qv4\" (UID: \"28490cc3-78ae-4771-a790-30d8b3d3fe44\") " pod="openshift-must-gather-2vr76/perf-node-gather-daemonset-j9qv4"
Sep 10 19:45:40.805535 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:45:40.805520    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/28490cc3-78ae-4771-a790-30d8b3d3fe44-lib-modules\") pod \"perf-node-gather-daemonset-j9qv4\" (UID: \"28490cc3-78ae-4771-a790-30d8b3d3fe44\") " pod="openshift-must-gather-2vr76/perf-node-gather-daemonset-j9qv4"
Sep 10 19:45:40.805799 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:45:40.805785    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/28490cc3-78ae-4771-a790-30d8b3d3fe44-podres\") pod \"perf-node-gather-daemonset-j9qv4\" (UID: \"28490cc3-78ae-4771-a790-30d8b3d3fe44\") " pod="openshift-must-gather-2vr76/perf-node-gather-daemonset-j9qv4"
Sep 10 19:45:40.806027 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:45:40.805731    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/28490cc3-78ae-4771-a790-30d8b3d3fe44-lib-modules\") pod \"perf-node-gather-daemonset-j9qv4\" (UID: \"28490cc3-78ae-4771-a790-30d8b3d3fe44\") " pod="openshift-must-gather-2vr76/perf-node-gather-daemonset-j9qv4"
Sep 10 19:45:40.806148 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:45:40.805463    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/28490cc3-78ae-4771-a790-30d8b3d3fe44-proc\") pod \"perf-node-gather-daemonset-j9qv4\" (UID: \"28490cc3-78ae-4771-a790-30d8b3d3fe44\") " pod="openshift-must-gather-2vr76/perf-node-gather-daemonset-j9qv4"
Sep 10 19:45:40.806253 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:45:40.805950    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/28490cc3-78ae-4771-a790-30d8b3d3fe44-podres\") pod \"perf-node-gather-daemonset-j9qv4\" (UID: \"28490cc3-78ae-4771-a790-30d8b3d3fe44\") " pod="openshift-must-gather-2vr76/perf-node-gather-daemonset-j9qv4"
Sep 10 19:45:40.815234 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:45:40.815213    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-khkxx\" (UniqueName: \"kubernetes.io/projected/28490cc3-78ae-4771-a790-30d8b3d3fe44-kube-api-access-khkxx\") pod \"perf-node-gather-daemonset-j9qv4\" (UID: \"28490cc3-78ae-4771-a790-30d8b3d3fe44\") " pod="openshift-must-gather-2vr76/perf-node-gather-daemonset-j9qv4"
Sep 10 19:45:40.884400 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:45:40.884357    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2vr76/perf-node-gather-daemonset-j9qv4"
Sep 10 19:45:41.180359 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:45:41.180306    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2vr76/perf-node-gather-daemonset-j9qv4"]
Sep 10 19:45:41.620523 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:45:41.620489    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2vr76/perf-node-gather-daemonset-j9qv4" event={"ID":"28490cc3-78ae-4771-a790-30d8b3d3fe44","Type":"ContainerStarted","Data":"c5e5ba96a29bcfb1e28b704c8305fed3f1fc883d3f7675b8bdf7403dc3e90d5c"}
Sep 10 19:45:41.620725 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:45:41.620541    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2vr76/perf-node-gather-daemonset-j9qv4" event={"ID":"28490cc3-78ae-4771-a790-30d8b3d3fe44","Type":"ContainerStarted","Data":"1296b05380adb3008eaa675cc98556cb9b092d8fa195b5e6a8843965582e9bee"}
Sep 10 19:45:41.620725 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:45:41.620691    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-2vr76/perf-node-gather-daemonset-j9qv4"
Sep 10 19:45:41.639474 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:45:41.639390    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-2vr76/perf-node-gather-daemonset-j9qv4" podStartSLOduration=1.639368546 podStartE2EDuration="1.639368546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 19:45:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:45:41.637632559 +0000 UTC m=+2104.866682585" watchObservedRunningTime="2026-09-10 19:45:41.639368546 +0000 UTC m=+2104.868418572"
Sep 10 19:45:42.745021 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:45:42.744990    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-m7z8f_f05a659a-63af-4897-a74b-e8b246a1a59f/dns/0.log"
Sep 10 19:45:42.775094 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:45:42.775063    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-m7z8f_f05a659a-63af-4897-a74b-e8b246a1a59f/kube-rbac-proxy/0.log"
Sep 10 19:45:42.916014 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:45:42.915986    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-x4n86_808e9bad-e9c5-4316-b4b6-e967cd0e3ef9/dns-node-resolver/0.log"
Sep 10 19:45:43.489810 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:45:43.489767    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-b4fdbd8dd-5xwv4_403c4bb0-0592-40f6-9c80-e1a680b6f480/registry/0.log"
Sep 10 19:45:43.579932 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:45:43.579902    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bq2kb_20594470-2f86-4aab-88b9-20cbdae08a97/node-ca/0.log"
Sep 10 19:45:44.889104 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:45:44.889070    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6b5f7cd94b-d6plt_bda5e467-10e3-458d-b507-91b0772d3d32/kube-auth-proxy/0.log"
Sep 10 19:45:44.924003 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:45:44.923970    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-ps4m9_698ce0a1-dce5-40d0-8bd4-c2042400e6f2/istio-proxy/0.log"
Sep 10 19:45:44.937033 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:45:44.936998    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-ps4m9_698ce0a1-dce5-40d0-8bd4-c2042400e6f2/istio-proxy/1.log"
Sep 10 19:45:45.962891 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:45:45.962857    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hjfhn_809134f0-f6fa-4ce5-befc-58bda6b2c1d4/serve-healthcheck-canary/0.log"
Sep 10 19:45:46.581898 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:45:46.581868    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qhr8m_b579572a-1c79-4730-a112-193b3887bbce/kube-rbac-proxy/0.log"
Sep 10 19:45:46.605507 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:45:46.605473    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qhr8m_b579572a-1c79-4730-a112-193b3887bbce/exporter/0.log"
Sep 10 19:45:46.629565 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:45:46.629535    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qhr8m_b579572a-1c79-4730-a112-193b3887bbce/extractor/0.log"
Sep 10 19:45:47.638335 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:45:47.638305    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-2vr76/perf-node-gather-daemonset-j9qv4"
Sep 10 19:45:48.990274 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:45:48.990233    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-q6cqb_9977ad8b-b590-4004-93b2-7e93892c0b7f/manager/0.log"
Sep 10 19:45:49.349817 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:45:49.349721    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-6lcpk_58903c8d-079b-45c6-9750-717e1fa44405/manager/0.log"
Sep 10 19:45:49.446536 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:45:49.446491    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-jsgn2_f239b2bf-edcf-4852-adc1-c905d4681e29/server/0.log"
Sep 10 19:45:49.566182 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:45:49.566137    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-d54bdf466-f2gzj_f0396ab1-e6a3-4b4e-8ebc-2f0031109a0f/manager/0.log"
Sep 10 19:45:49.595114 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:45:49.595084    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-d54bdf466-f2gzj_f0396ab1-e6a3-4b4e-8ebc-2f0031109a0f/copy-manifests/0.log"
Sep 10 19:45:51.031474 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:45:51.031441    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6b5f7cd94b-d6plt_bda5e467-10e3-458d-b507-91b0772d3d32/kube-auth-proxy/0.log"
Sep 10 19:45:51.054824 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:45:51.054782    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-ps4m9_698ce0a1-dce5-40d0-8bd4-c2042400e6f2/istio-proxy/0.log"
Sep 10 19:45:51.077482 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:45:51.077447    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-ps4m9_698ce0a1-dce5-40d0-8bd4-c2042400e6f2/istio-proxy/1.log"
Sep 10 19:45:52.935550 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:45:52.935500    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6b5f7cd94b-d6plt_bda5e467-10e3-458d-b507-91b0772d3d32/kube-auth-proxy/0.log"
Sep 10 19:45:52.961611 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:45:52.961544    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-ps4m9_698ce0a1-dce5-40d0-8bd4-c2042400e6f2/istio-proxy/0.log"
Sep 10 19:45:52.972235 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:45:52.972197    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-ps4m9_698ce0a1-dce5-40d0-8bd4-c2042400e6f2/istio-proxy/1.log"
Sep 10 19:45:56.521695 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:45:56.521658    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-rt6b9_03f3a2ce-0fe6-4460-a42e-279026aba630/migrator/0.log"
Sep 10 19:45:56.548210 ip-10-0-132-222 kubenswrapper[2546]: I0910 19:45:56.548179    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-rt6b9_03f3a2ce-0fe6-4460-a42e-279026aba630/graceful-termination/0.log"